• Pay for performance fail – ctd.

    Long time readers of the blog know of my skepticism for “pay for performance”. That’s not because I take issue with the defining principle; it’s because I think that we are nowhere near a good way of defining “quality” or linking it to actual outcomes.

    That doesn’t stop the politicians. They love to talk about how we will “pay for quality, not quantity”, as if that is something new. Why haven’t we done that before? We haven’t because it’s hard.

    When it comes to this kind of work, we often act like the drunk searching for his car keys under the street light. We pick metrics that we can easily measure, not those that might actually matter. When we do that, actual outcomes don’t improve. Case in point, “The Long-Term Effect of Premier Pay for Performance on Patient Outcome“, over at the NEJM:

    BACKGROUND: Pay for performance has become a central strategy in the drive to improve health care. We assessed the long-term effect of the Medicare Premier Hospital Quality Incentive Demonstration (HQID) on patient outcomes.

    METHODS: We used Medicare data to compare outcomes between the 252 hospitals participating in the Premier HQID and 3363 control hospitals participating in public reporting alone. We examined 30-day mortality among more than 6 million patients who had acute myocardial infarction, congestive heart failure, or pneumonia or who underwent coronary-artery bypass grafting (CABG) between 2003 and 2009.

    What’s the Premier HQID? Well, way back in 2003, CMS invited a bunch of hospitals to participate in a demonstration project for quality, and 252 agreed to participate. Those hospitals agreed to turn in data on 33 measures, for medical conditions like heart attacks, congestive heart failure, and pneumonia, as well as for  procedures like CABGs, knee replacements, and hip replacements. Indicators were assigned to these conditions and procedures, and they were then used to measure “quality”. Those hospitals that did well could get 1-2% Medicare bonuses, and later those that did poorly might suffer a 1-2% Medicare penalty. The real question, though, is whether hospitals that worked to achieve “quality” by these metrics actually made a difference in outcomes that matter. Did they?

    No. Seriously, go look at the chart. It’s the 30-day mortality for these issues in hospitals both in and not in the pay-for-performance program. Can you see a difference? Me neither.

    They even did an analysis of just those hospitals that started out doing “poorly” according to the metrics. You’d think that those hospitals would have the most to gain by focusing on “quality” metrics. The results were pretty much exactly the same.

    The authors’ conclusion:

    We found no evidence that the largest hospital-based pay-for-performance program led to a decrease in 30-day mortality. Expectations of improved outcomes for programs modeled after Premier HQID should therefore remain modest.



    • Dear Aaron,

      I share your skepticism of P4P. Of course everyone is in favor of improving “quality” — who could possible argue otherwise? But like you I suspect that the things measured by the easily calculated quality metrics don’t have much to do with patient outcomes.

      The one thing that puzzles me about this study is the duration of the outcome measure. 30-day mortality seems a bit short for me. The counterargument could be made that “quality” won’t make much of a difference with patients who are death’s door, but instead the payoff is to be found with healthier patients. It would be interesting to know what (for instance) the 2-year or the 5-year mortality rate was. Perhaps the nature of the data (harvested from Medicare) made it difficult to ascertain this.

      Anyway, thanks for directing our attention to this interesting (although disappointing) study, and I will be sure to follow the link and read the article.

    • These amounts seem to small to write off the idea. Would you change your behavior for a 1 to 2% change in pay?

    • I’m not sure I get what quality is in medicine. In a car, you can add stuff like leather seats and ABS. But if you extend the analogy to healthcare, ABS improves handling and reduces crashes. In medicine, you get a huge penalty (malpractice suit) for not doing your all. So, ABS is basically mandatory. The only real options are the stuff like leather seats which have not been proven to save lives.

      Further, if something like an extra night in the hospital is not paid for by insurance, it won’t happen. A 2-3% premium is probably not going to make up the cost, so even with a proven benefit, no car/patient will get ABS under the new scheme.

      When I worked at a hospital in Pittsburgh, their scheme for becoming a world class center of excellence was poaching doctors from a nearby world class center of excellence, by making their hospital more attractive to doctors. That’s the kind of thing that might net a hospital a 2-3% premium, but does nothing to improve overall Medicare outcomes.