• Pay for performance fail

    A colleague alerted me to a study just published in the BMJ that examined how pay for performance measures worked in the UK. Let’s go to the tape:

    We studied a large scale pay for performance policy in the four countries of the United Kingdom (England, Scotland, Wales, and Northern Ireland), which targeted several chronic diseases in primary care, and evaluated its impact on the management and outcomes of hypertension. Based on the proportion of patients achieving certain quality indicators, general practitioners could receive payments as high as 25% of their total income. The programme started in April 2004 and included 136 quality indicators, including five for hypertension (see web extra), one of which was the proportion of patients with blood pressures controlled to 150/90 mm Hg or less.

    This natural experiment was ideal for detecting the effects of pay for performance on the care and outcomes of hypertension. Although the programme was voluntary, 99.6% of general practitioners participated. The financial incentives for doctors to achieve the quality standards were substantial; the UK National Health Service committed £1.8bn (€2.1bn; $2.8bn) in funding. Almost simultaneously (June 2004) the National Institute for Health and Clinical excellence (NICE) released guidelines for the detection and management of hypertension, which were consistent with the pay for performance intervention guidance and may have also reinforced the intervention. Using data obtained from representative primary care practices in the United Kingdom, we evaluated whether the introduction of pay for performance had an impact on quality of care for hypertension and the risk of major adverse clinical outcomes.

    Here’s the gist. If you believe that pay for performance works, then you would expect that once the bonuses kicked in, physicians would try to earn them. In this case, you would expect to see the percentage of patients with their high blood pressure controlled go up. From 2000 to 2007, there were over 470,000 patients with hypertension (high blood pressure) that were examined for this study. The pay for performance incentives kicked in mid-2004.  Let’s see what happened:

    The top chart shows the percentage of patients whose blood pressure was controlled. As you can see, things were improving slightly before the pay for performance kicked in. After it did, there was no change in the trend. The economic incentive program had no effect. The bottom chart shows the percentage of patients who were having their high blood pressure monitored. No change there because of the program either.

    I concede that these are process measures. But process measures are often how pay for performance is operationalized. We measure whether labs are drawn or visits happen, not how sick patients get. But let’s say we did. Was there a difference in outcomes?

    No. The top chart here is the cumulative percentage patients in the study who had a heart attack, heart failure, stroke, or renal failure. There was no difference in the cumulative rate before or after pay for performance kicked in. The bottom chart is the percentage of patients who died.  No effect there either.

    Pay for performance didn’t work. Now I know many of you are going to argue that the UK is not the US.  I concede the point. But there’s no reason to believe that the docs there wouldn’t be interested in economic incentives. And the incentives are large here. Physicians could receive bonuses up to 25% of income? That’s an enormous amount of money. Heck, I think even I would be enticed by that amount.

    So why did this fail? Perhaps the doctors were already improving without the program. If that’s the case, though, then you don’t need economic incentives. It’s possible the incentives were too low. But I don’t think many will propose more than a 25% bonus. It’s also possible that the benchmarks which define success were too low and therefore didn’t improve outcomes. There’s no scientific reason to think that the recommendations weren’t appropriate, however.

    More likely, it’s what I’ve said before. Changing physician behavior is hard. It’s the reason I think tort reform won’t be a huge cost-saver either. But this is bad, because many people have hung their hat on pay for performance being the solution to health care quality and costs in the US. This study adds to the body of evidence that says that’s not the case.

    The same colleague who told me about the study suggested that we need to change the organizational structure of the practice, ie change the system, so quality doesn’t rely on one person or one process. I agree that’s much more likely to work. Others will say that’s the point of ACOs. But as we’ve discussed before, there are lots of reasons to believe ACOs won’t lower costs (or improve quality). Moreover systems level changes would be harder, perhaps require investments up front, and not make for good sound bites. It might be better policy, but it doesn’t make for better politics.

    So for now, we’ll likely continue down this potential dead end.

    • This study made an impression on me when it was released. I went back to review, and i do remember the cynicism of NHS FP’s “success,” even back then.


      I did not read the BMJ paper, but I wonder about gamesmanship, adverse selection, etc. Attainment was so darn high, much higher than USA advances (national strides in CORES–although they are gamed to a degree her as well), that one wonders if it is more than the fish&chips that produced the difference.


    • Brad –

      Go read it. I quote them:

      We found no evidence of gaming of the system to achieve quality targets. The level and trend for newly treated patients did not change after the implementation of pay for performance. We hypothesised that this measure would change because of the ease of implementation for general practitioners. (Doctors could have stepped up the treatment of patients with blood pressure close to control, resulting in an increase in the proportion of newly treated patients.) This lack of evidence for gaming is consistent with other studies.

    • Yup. Got it. Thought BMJ was pay access only.

      Still curious about secular trends of other measures not reported on–that is another area ripe for examination. Would not want to throw the baby out with the bathwater if other signals in NHS were promoting BP reduction, regardless of P4P availability.

      As authors state, if HTN levels of achievement were set too low or bonuses were too permissive with this metric, does not necessarily mean all is lost.

      HOwever, your point is well taken. As with EMRs, the P4P blush is off the rose. I agree that population level measures are far more enticing, but as one in the trenches, I understand these are the needed baby steps to get us there…


    • “Changing physician behavior is hard.”

      As someone with hypertension, let me assure you that changing patient behaviour is probably even harder.

    • GENERAL POINT: The Devil is in the Details

      “Our study sample consisted of 470 725 patients with a diagnosis of hypertension during the observation period (January 2000 to July 2007)”

      This is why it’s important to have a control group – panel data is tricky. A patient diagnosed with hypertension in Jan2000 will have had hypertension for six years by Jan2007. Their P(complication) is likely increasing over this time; so, we would expect complications
      to increase in the population in general as the patients disease gets more serious.

      It’s called an omitted variable! Control for it – you have the data! Sheesh!

      “general practitioners could receive payments as high as 25% of their total income.”

      With what probability?
      Rational agents will care about the expected value, not the strict payoff conditional upon winning.

      Also, what are the conditions for winning? I can’t seem to find that anywhere. It may be that this incentive program was inapplicable to a wide swath of practicing physicians, e.g. “Bonus for the top 1%” is going to discourage anyone that knows they’re never going to be in the top quartile.

      “Although the programme was voluntary, 99.6% of general practitioners participated.”

      That smacks of a default rule…have you ever heard of people opting-in to any social program with that kind of success rate? You can’t even get most people to make matching contributions to their 401k’s – that’s sure money, and it’s still left on the table! Sorry, I’ve worked in social policy…that take up rate raises serious questions as to whether the participants even knew that they were participating.

      In sum:
      I know that medical journal entries differ in length from economics journal entries, but these details are the difference between valid/invalid inference in their “Policy Implications” section. As such, they ought to be included

    • Those that argue for pay for performance believe it is consistent with medical ethics and that it will not only improve the safety and quality of patient care, but perhaps more important keep down the cost of care. Suggesting that the quality of medical care will improve fails to consider that physicians are obliged to do their best regardless of a reward and this obligation is due regardless of any fiduciary responsibility. As with all pay for performance reviews it implies that the fiduciary relationship is insufficient and the motivation for physicians and all other employees to do their best is a function if an extrinsic reward and this reward is the only method to get quality effort. It reduces the physician or employee to the role of an American waiter or servant. In most countries tipping or offering gratuities is considered an insult. I recall being in a fish market in Bibione, Italy, a customer after making a major purchase handed the employee money, a tip, and she threw it against the wall in disgust. Why do restaurateurs want tipping? It allows them to keep wages low. In many restaurants all tips are pooled and everyone from manager to dishwasher gets a share, “profit sharing off the books.” Tipping is permitted in all low paying jobs from mowing lawns to being a pool boy at the local swim club. In America tipping is considered insulting when it does not match the recipients expectations. What happens when this is applied to the corporation of physicians, and they do not get the tip that they have grown accustomed to? Would you want to be that physician’s next patient, or be that employee’s next customer?

    • It also depends on how the Doctors were “paid out”. Based on the results how many achieved the bonus?