• Forecasts and uncertainty

    I need to come back to one thing related to the CBO’s work, and that of other budget forecasters. Long-range predictions are uncertain. Nobody should believe a budget forecast more than a few years out, let alone decades. There are many reasons for this, but they don’t imply that such forecasts are not worthwhile or useful.

    Forecasts are uncertain because the assumptions that underly them won’t remain true. Some of them aren’t even true today. Forecasts are based on all sorts of approximations. There are limits to our current knowledge and resources that cause our models to differ from reality. Those differences compound the further we look ahead. The world will change, as will our understanding of it.

    For example, the CBO’s scoring of health reform is grounded in the current state of research on how individuals and organizations respond to various incentives like the price of health care and to the available options for coverage. Scholars, myself included, are studying those issues all the time. New results emerge that enhance our understanding. What we thought was true yesterday is no longer the state of the art. I have no doubt I’ll be posting findings on this blog that suggest that the assumptions underlying today’s long-range projections of the evolution of our health system are further from the truth than we thought.

    That’s good! That’s progress. That’s not bashing the CBO (which I pledged not to do). The CBO is making what I believe to be the best, unbiased estimates they can. They’re not going to prove to be right, but they’re not biased. They’re as likely to be wrong on the high side as the low. I’ve got no good basis to second guess what CBO analysts have done based on what they knew at the time they had to produce their estimates.

    So why should we pay any attention to long-term forecasts we know to be wrong (without knowing by how much or in which direction)? Because we have to make decisions today. Just because forecasts are wrong doesn’t mean they’re uninformative. I wrote about this almost one year ago.

    [When predictions deviate from reality] many will be tempted to conclude that experts can’t predict the budget accurately so the whole budget prediction exercise is a useless waste of time at best and a political tactic at worst. That type of thinking misses a crucial point of budget predictions.

    The point is not necessarily to predict the future accurately, though that would certainly be nice. The point is to use prediction tools to observe the expected changes to future budgets under different policies put into place today. […]

    Notice that this exercise has less to do with predicting a single future budget accurately than it does with predicting the difference in future budgets under two regimes. And what’s of great importance is the sign of the difference. […]

    So, don’t get worked up about inaccuracy of budget predictions. Don’t dismiss the exercise just because the result misses the mark a few years down the road. The best use of budget modeling is as a tool to policy planning. It can lead to better relative decisions even if it is inaccurate in an absolute sense. (The key, of course, is the extent to which the tool is systematically biased, favoring one policy type over another, say. That’s a different story.)

    Put another way, is there any reason to believe the CBO’s estimates in the difference between forecasts under current law and a deviation from it are biased? That’s the only sense in which they could mislead us from making the best decision we can today given what we know today, uncertainty about the future and all.

    Share
    Comments closed
     
    • I’ve been meaning to post this here. Charles Manski has a paper with a bizarre section on scoring the PPACA (starting on p.8):

      I think you’ll find quite a contrast between what Manski says and with both the actual CBO analysis and, especially, Elmendorf’s Director’s follow-up blog post of March 19, 2010. To preview, here’s Manski:

      “However, the twenty-five page letter from Elmendorf to Pelosi expressed no uncertainty and did not document the methodology generating the prediction.”

      And here’s the title–just the title!–Elmendorf’s follow-up blog post published the day after he sent the letter to Pelosi:

      “Uncertainty in Estimates for Health Care Legislation.”

      You be the judge….

      Links:
      CBO Letter:
      Director’s Blog Post:

      • @Paul – The CBO has produced a huge volume of documentation on their estimates and methodology.

        The Director’s Blog Post says “CBO staff, in consultation with outside experts, has devoted a great deal of care and effort to this analysis, and the agency strives to develop estimates that reflect the middle of the distribution of possible outcomes. As a result, we believe that CBO’s estimates of the net savings that would result from the legislation have a roughly equal chance of turning out to be too high or too low.”

        If anyone thinks that’s not true, I’d like to see the evidence. I say this without having read Manski’s paper. I will take a look at it.

    • You have cited (some of) the reasons why I said Manski’s complaint is bizarre. 🙂