OIL PRICE FORECASTING

Systematic bias in EIA oil price forecasts: Concerns and consequences

Is the forecasting model wrong? Outdated? And can price forecasting be improved?

Frank Clemente and Timothy J. Considine, Penn State University

Over the past decade, EIA projections differ substantially from actual oil prices. Further, Gately (2001; 2007)¹ has suggested that the EIA National Energy Modeling System (NEMS) is “internally inconsistent” and routinely forecasts oil prices which are “far too low.” A review of the data provides strong support for Gately’s criticism. Since 1998, of 45 annual forecasts from the EIA, 42 (93%) have under-predicted the price of oil.

To statistically assess the situation, we conducted an error decomposition analysis of EIA projections of oil prices from 1998-2006. Error decomposition is commonly used to evaluate economic forecasting models by identifying three sources of error: random chance, linear bias and model bias. The statistical analysis revealed:

1. The root mean squared error of the one-to-four-year EIA forecast of oil prices averages over 30% and ranges as high as 46%.

2. These forecast errors are not reflective of random chance, but instead display a pattern of asymmetric bias, arising either from a fixed, linear bias or from systematic error associated with the NEMS model. In fact, in the four-year oil price forecasts, over 70% of the error can be attributed to systematic bias.

3. This bias is directed toward “optimism,” i.e., actual market price of oil is systematically under-predicted in more than 90% of the cases.

CONCERNS/CONSEQUENCES

Despite the biased divergence between their forecasts and actual oil prices, EIA has published virtually nothing on the question of asymmetrical error. In fact, EIA’s model evaluation methodology may itself camouflage the problem. Further, EIA has apparently not backcasted its model, and no retrospective verifications have been made public. Finally, absent any change, there is concern about another spate of under-predictions going forward.

As Winebrake and Sakva (2005)² have noted, policymakers routinely use EIA forecasts for future decades “to justify policy action today.” The real-world impacts of oil price under-predictions have been significant: a $380 billion cost differential between EIA forecasts and what consumers actually paid for oil; a chilling impact on investments in coal-to-liquids and other unconventional fuels; and the risk that climate change policies, based on EIA forecasts, will significantly underestimate economic impacts.

The analysis reported here suggests that policymakers should exercise considerable caution when using these EIA forecasts. Similar caution should be employed when using NEMS to assess the broader socioeconomic consequences of energy policy initiatives, e.g., carbon cap-and-trade programs.

Systematic bias. Oil prices are notoriously difficult to predict. Even the most sophisticated forecasting models cannot account for all the variables, and broad ranges in forecast results are expected. The focus here, however, is not on forecast difficulty. Rather, we ask: Are systematic biases built into certain methodologies, causing them to err repeatedly in the same direction? Systematic bias, and the failure to recognize that bias, can lead to consistent and routine error, which multiplies the problem as 1) policy decisions are based on inherently biased results, and 2) subsequent forecasts by third parties perpetuate the original mistake.

Few oil price forecasts are more visible, or more highly used, than those of the US Energy Information Administration (EIA). The EIA Annual Energy Outlook³ forecasts are routinely used in regulatory proceedings, budget projections, energy facility planning, scientific research, investment decisions, litigation, legislation and energy policy decisions.

Given the widespread use of EIA forecasts, it is clear that a systematic bias in its results can have profound implications not only in the US, but in other nations as well. The National Energy Board in Canada uses the EIA forecast; so does the Jaffee Center for Strategic Studies in Israel.⁴ Even OPEC scholars use EIA projections as a benchmark in their research.⁵

A problem perceived but not analyzed. Over the past decade, it has become increasingly apparent that there is disjuncture between EIA oil price forecasts and actual outcomes. More recently, Laitner, et al.,⁶ and Sweeney⁷ have documented significant divergence between EIA forecasts and reality.

Indeed, these discrepancies appear to have even led some organizations to forego use of the EIA price forecasts altogether. In April 2007, the US Government Accounting Office (GAO) published a review of the Department of Defense (DOD) method of future oil pricing.⁸ The GAO found that DOD conducted an evaluation of EIA oil price forecasts and decided not to use them in fuel purchase decisions. Based on the results of these analyses: “DOD selected its current method as the preferred forecasting approach over the DOE/EIA.”⁸

In addition to these concerns, there is a question as to whether the EIA model has a systematic bias to produce “optimistic” results, i.e., predictions of declining prices. Gately, for example, has questioned the EIA “sanguine view of the future-with oil prices at low-to-moderate levels.”¹ In a review of the EIA model, he contends:

“the model underlying DOE/EIA is internally inconsistent...if it accurately represents price responsiveness of world oil demand...then [it projects] oil prices that are far too low.”

Gately’s concerns have significant face validity:

• In 1998, EIA predicted world oil prices would be $24.34 (2006 dollars) in 2000. Actual prices were $32.21-a 24% underestimate.⁹
• In 2000, EIA predicted prices would be $24.00 in 2002. Actual prices were $26.41-a 9% underestimate.
• In 2002, EIA predicted prices would be $26.41 in 2004. Actual prices were $38.22-a 31% underestimate.
• In 2004, EIA predicted prices would be $25.15 in 2006. Actual prices were $63.64-a 60% underestimate.

Although these data are only a select sample, on the surface, they support the argument that there is an inherently low-price bias to the EIA analyses.

FORECAST EVALUATION BY ERROR DECOMPOSITION

The specific goal of this research is to assess whether EIA oil price forecasts have a systematic bias. Accordingly, three standard economic forecast evaluation metrics were used to examine oil price predictions from the EIA National Energy Modeling System (NEMS):⁹

• Average percentage error-commonly used, but must be supplemented by more sophisticated measures, because large positive and negative values cancel each other out.
• Average absolute error-provides an estimate of the average magnitude of the forecast errors.
• Mean squared error-compares predicted versus actual changes. Squaring the errors has the effect of penalizing large errors, either negative or positive. The square root of the mean squared error, often referred to as the Root Mean Squared Error (RMSE), is more commonly reported, because the square root operator on changes closely approximates percent change.

Each of these measures of error compares actual observations with predicted values. In the ideal case, such errors would be random, with no particular pattern of over- or underestimation.

Economists and statisticians have developed a variety of methods to determine whether forecast errors exhibit randomness or reflect systematic bias. These methods involve decomposing the root mean squared error into various error components. An approach devised by Theil¹⁰ and later recommended by Maddalla,¹¹ and subsequently used in a wide range of subsequent studies,¹² involves computation of the following three components:

where is the mean of in which P_t is the prediction from the model for period t, and A_t is the actual realized value of the variables in that period, is the mean of , S_p is the population standard deviation of p, r is the correlation coefficient between p and a, and S_a is the standard deviation of a, and all three measures sum to one, i.e., B + M + R = 1. Both Maddalla and Theil noted that the bias and the model components measure what can be called “systematic” errors. If B is large, then the average predicted change deviates substantially from the actual average change. This is a serious error because forecasters should be able to reduce such errors in the course of time. In short, if B is close to 1, the forecast is considered biased. The model component of the forecast error reflects the linear association between the actual and predicted values. If M is relatively large, then this would suggest that the model itself is generating systematic errors. In a perfect forecast, both M and B would be zero.

DATA AND FINDINGS

During December of each year, EIA publishes a forecast that forms the basis of the Annual Energy Outlook (AEO) for the subsequent year and for the next 20-plus years.¹³ The present analysis examines EIA oil price forecasts published from 1998 to 2006-yielding 45 annual predictions-nine one-year-ahead forecasts, eight two-year, seven three-year, six four-year, five five-year, four six-year, three seven-year, two eight-year and one nine-year. Thus, there is an extensive array of forecasts available for evaluation.

EIA forecasts oil prices in constant dollars. To establish a consistent basis for comparison, these constant price forecasts were inflated by the corresponding forecasts for the price deflator for Gross Domestic Product (GDP). Once the forecasts were sorted, the prices were converted back to 2006 dollars using the latest GDP price deflator.

TABLE 1. Actual and EIA forecasts of world oil prices in 2006 dollars/bbls and percentage errors

Before presenting the decomposition analysis, it is useful to examine the tabular and graphic representations of how well EIA forecasts have tracked actual oil prices over the past decade. Table 1 presents the actual data, and Fig. 1 provides a visualization of the results. Three key points can be gleaned from these findings:

1. EIA forecasts since 1998 have consistently underestimated future oil prices. Of 45 forecasts, 42 (93%) presented a prediction of lower prices.

2. Since 2000, this phenomenon of optimism has become increasingly pronounced. Of 28 forecasts since 2000, for instance, 27 (96%) have underestimated future oil prices.

3. These optimistic estimates are not merely errors of direction, but more importantly, errors of magnitude. For example, as recently as 2003, EIA underestimated oil prices for 2004, 2005 and 2006 by 29%, 48% and 59% respectively.

Fig. 1. Short-term EIA forecast.

These data strongly support Gately’s contention that the internal inconsistencies of EIA’s model generate oil prices that are “far too low.” To quantify the size and systematic tendencies of these forecasts errors, we turn to the error decomposition methods discussed earlier.

ERROR DECOMPOSITION RESULTS

To keep the analysis manageable and comprehensible, the decomposition analysis is conducted for the one- through four-year-ahead forecasts from 1998 to 2006, which appear in Table 2. On average, the one-year-ahead forecast error for the world oil price is 6.2% with an absolute error of $6.50/bbl. These errors steadily rise and reach more than 31% with the four-year-ahead forecast, and an absolute level of nearly $15/bbl.

TABLE 2. Forecast performance metrics

The RMSE, which penalizes large errors more severely than the average percentage error, averages 26% for the one-year-ahead forecast. Like the average percentage error, it too rises with the forecast horizon, reaching more than 46% with the four-year-ahead forecasts.

The decomposition of the MSE for the one-year-ahead oil price forecast errors indicates that 56% of the errors can be attributed to systematic bias. This bias crests to more than 70% for the four-year-ahead forecasts. While random disturbances are substantial for the one-year-ahead forecast, the large proportion attributed to bias is noteworthy. A plot of the actual time series for average world oil prices and the four different forecasts appears in Fig. 1, and illustrates the tendency of the EIA price forecasts to systematically under-predict actual prices.

The long-term projections of EIA exhibit a tendency to mean reversion. In other words, they tend to drift back down a lower level. As Fig. 2 illustrates, this lower level was in the vicinity of $30/bbl. After 2006, EIA revised this long-term target up, to between $50 and $60/bbl. Like their prior forecasts, EIA continues to project lower oil prices through 2015. Unfortunately, their track record on long-term projections, as illustrated above, does not inspire significant confidence in such a belief.

Fig. 2. Long-term EIA forecast.

CONCERNS DUE TO SYSTEMATIC BIAS

Although our analysis is limited to previous forecasts, the findings raise questions about current and future forecasts for oil prices from EIA.

Continuing optimism is prevalent in current EIA Outlooks. In the 2007 AEO, for example, EIA forecasts oil prices to decline to $49 by 2015-a substantial decrease from $63 in May 2007. Taken by itself, this forecast might be credible. But when viewed in the context of biased optimism since at least 1998, yet another prediction of moderating prices must be viewed skeptically, especially in the context of the fact that oil prices have increased significantly in the last five years, and that the bulk of this increase was not projected by the EIA.

Failure to recognize the problem. Despite the biased divergence between their oil price forecasts and actual outcomes, the EIA has published virtually nothing on the question of systematic error. In fact, EIA’s model evaluation methodology may itself camouflage the problem. Auffhammer, for example, has commented “The EIA conducts its own forecast evaluation...(but) this type of evaluation ignores potentially persistent biases in the forecasting model.”¹⁴

Lack of retrospective analyses. The EIA has presented no “backcast” of its model results. Koomey, et al., have noted:

“One of the most striking things about forecasters is their lack of historical perspective. They rarely do retrospectives, even though looking back at past work can both illuminate the reasons for its success or failure, and improve...future forecasts.”¹⁵

The error decomposition analysis presented here strongly suggests there is a systematic bias in the NEMS model, and the EIA would be well advised to carefully examine the validity of its model through retrospective backcasting.

Systematically low price predictions can negatively affect many decisions. In addition to political policy, the difference between the forecasted price and the actual oil price affects planning across a range of activity: from airlines to trucking and from commuters to plastic manufacturing. The budget of virtually every state has been impacted by unexpectedly rising fuel costs. These unanticipated costs are significant. Using the differences between actual prices and the 1998 EIA forecast, one can calculate that the unanticipated cumulative costs from 2000 to 2006 exceed $380 billion.

Chilling impact on research and development on technology. These include coal to liquids (CTL), gas to liquids (GTL) and other unconventional fuels. In their general review of the EIA NEMS model, Laitner, et al., concluded:

“The record of (EIA) energy forecasting yields evidence that such models provide biased estimates that tend to reinforce the status quo...and serve to constrain the development of innovative polices.”⁵

From 1998-2003, EIA predicted oil prices in 2015 would generally not exceed $30. This continuing stream of low price predictions clearly stultified interest and investment in CTL, GTL, tar sands, oil shale and ethanol. Few companies, or nations for that matter, are willing to invest several billion dollars in unconventional fuel development under the looming specter of EIA’s ever-optimistic oil price forecasts.

Climate change proposals now before Congress (e.g., Bingaman, McCain, Lieberman¹⁶) depend heavily on predictions of moderating oil price increases. In their analysis of the Bingaman proposal, for example, EIA forecasts that oil prices will decline to $45 by 2015.¹³ Underestimating the future price of oil, as EIA has done consistently over the past decade, could lead to inaccurate cost projections of carbon regulations.

CONCLUDING COMMENTS

This analysis suggests that policymakers should exercise considerable caution when using EIA oil price forecasts. Similar caution should be employed when using NEMS to assess the broader economic impacts of energy policy initiatives, e.g., carbon cap and trade programs. The systematic tendency of the EIA model toward under-prediction of oil prices has had significant adverse socioeconomic consequences and continues to threaten the development of sound US energy policy.

LITERATURE CITED

¹ Gately, D., “How Plausible is the Consensus Projection of Oil Below $25 and Persian Gulf Oil Capacity and Output Doubling by 2020?” The Energy Journal. Vol. 22, No. 4, 2001; Gately, D. “What Oil Exports Should we Expect from OPEC?” The Energy Journal, Vol. 28, No. 2, 2007.
² Winebrake, J., Sakva, D., “An Evaluation of Errors in the U.S. Energy Forecasts: 1982-2003,” Energy Policy, Vol. 34, pp. 3475-3483, 2005.
³ Each year the EIA publishes the Annual Energy Outlook AEO. The data used in this analysis are drawn from the AEO publications from 1998 - 2000. Energy Information Administration, Annual Energy Outlook, Washington, D.C. yearly.
⁴ Rivlin, P., “The Oil Market”, Strategic Assessment, Vol. 3, No. 3, 2003.
⁵ Franssen, H., “Long Term Oil Market Outlook,” OPEC Review, pp. 203-213, Sept., 2003:
⁶ Laitner, J., DeCanio, S., Kooney, J., Sanstad, A. “Room for Improvement,” Utilities Policy, Vol. 11, No. 203: pp. 87-94.
⁷ Sweeney, M., “More Attention Needs to be Focused on the CEC Fuel Price Forecasts,” paper presented at California Energy Commission Workshop, October, 2006.
⁸ Pickup, S., “Defense Budget: Review of DOD’s Report on Budget for Fuel Costs,” GAO report to U.S. Congress, April 26, 2007.
⁹ Energy Information Administration, Annual Energy Outlook (AEO), U.S. Department of Energy, Washington, D.C. Data were taken from each AEO from 1998-2006.
¹⁰ Theil, H., Applied Economic Forecasting, Rand McNally and Company, 1966.
¹¹ Maddala, G.S., Econometrics, McGraw Hill, 1977.
¹² For example, O’Neill, B., Desai, M., “Accuracy of past Predictions,” Energy Policy. Volume 33, pp. 979-993, May 2005,
¹³ Energy Information Administration, Annual Energy Outlook (AEO), U.S. Department of Energy, Washington, D.C. Data were taken from each AEO from 1998-2006.
¹⁴ Auffhammer, M. “The Rationality of EIA Forecasts under Symmetric and Asymmetric Loss,” Resource and Energy Economics, Vol. 21; pp. 102-121.
¹⁵ Koomey, J., Craig, P., Gadgil, A. and D. Lorengetti, “Improving Long-Range Energy Modeling: A Plea for Historical Retrospectives,” The Energy Journal, Vol. 24, No. 4, 2003.
¹⁶ Energy Information Agency, “Energy market and Economic Impacts of a Proposal to Reduce Greenhouse Gas Intensity with a Cap and Trade System,” Washington, D.C., January 2007.