• Does practice make perfect?

    Hospitals that perform a greater volume of a specific procedure do it better, right? Well, actually, that’s believable and supported by evidence. Now for the harder question: it’s the greater volume that causes the better outcomes, right? You know, “learning by doing,” “practice makes perfect,” etc. That’s believable too. But it’s not as empirically verifiable.

    In fact, there’s good reason to believe causality runs the other way too. Hospitals that yield better outcomes have higher volume, a referral effect. Ask 100 physicians in your area where to have a CABG and the results won’t be random. They’ll point you to the well-known facility or two that do the best job, perhaps with the lowest mortality. So, more patients will go to those, increasing their volume. Low mortality causes higher volume. That’s not “practice makes perfect.” That’s a referral effect.

    So, outcomes (like mortality) and procedure volume are likely to be simultaneously determined (causality runs both ways). How much is “practice makes perfect,” and how much is “selective referral”? We don’t have to guess. Let’s hit the literature.

    A recent paper in Health Economics by David Barker, Gary Rosenthal, and Peter Cram (ungated working paper available) provides one route to an answer and includes a literature review that provides others. They examine the relationship between volume and mortality for cardiac revascularization in specialty hospitals versus general hospitals. After controlling for the simultaneity of volume and mortality, they conclude that “specialty hospitals do not have an advantage over general hospitals in mortality rates after cardiac revascularization.” Moreover, they find that mortality rates do influence volume. Therefore, efforts to increase volume may not themselves increase the quality of outcomes. Volume isn’t exogenous.

    Here’s their literature review:

    The pioneering study of the relationship between volume and outcome (Luft et al., 1979), inspired by the economics literature on industrial learning curves, reported a strong correlation between volume and outcome. It mentioned, however, the possibility that the causal relationship might be from outcome to volume, rather than the other way around. In a second study, Luft (1980) explored the directionality of this relationship using the econometric technique of two-stage least squares and confirmed this concern. He found evidence suggesting that the effect of outcome on volume (the ‘selective referral’ hypothesis) was stronger than the effect of volume on outcome (the ‘practice makes perfect’ hypothesis).

    A later study (Hughes et al., 1988) found evidence that the relationship was strong in both directions. Another study (Luft et al., 1990) found that hospital quality affected patient’s hospital choice even before explicit data on quality were available, supporting the relevance of the ‘selective referral’ hypothesis.

    Although failure to take account of both hypotheses has the potential to bias the results of studies of hospital volume, very few studies in mainstream medical research journals demonstrate awareness of the bi-directional volume–outcome relationship. Some have addressed the issue using longitudinal data. Hannan et al. (1992, 1995) found evidence supporting both directions of causality. Hamilton and Ho (1998) and Hamilton and Hamilton (1997) found no evidence supporting the ‘practice makes perfect’ hypothesis after controlling for fixed quality effects, supporting the ‘selective referral’ hypothesis.

    Studies using cross-sectional data and correcting for possible simultaneity are even less common than longitudinal studies. Other than the work of Harold Luft, we are aware of only three studies that explicitly test or correct for simultaneity. Farley and Ozminkowski (1992), using longitudinal data and two-stage least squares, found that the ‘practice makes perfect’ effect disappeared for coronary artery bypass grafting (CABG) once the ‘selective referral’ effect was controlled for, although the effect remained for some other procedures. Norton et al. (1998) found a t-statistic of only 1.80 in a secondstage regression and concluded that they were unable to reject exogeneity, and hence a correction for simultaneity was not required. This conclusion was criticized as premature by Luft (1998a,b), who argued that a t-value that approached significance should have led the authors to explore additional model specifications to rule out endogeneity. Tsai et al. (2006) used patient distance from hospitals to construct an instrument for volume. After adjusting for simultaneity, the ‘practice makes perfect’ effect became statistically insignificant.

    Although ‘selective referral’ is ignored in the vast majority of studies of volume and outcome, there is considerable evidence that it should be taken into account. Failure to take simultaneity into account does not, of course, call into question the correlation between volume and outcome. As Luft (2003) states, if he were injured in an unfamiliar city, he ‘would ask the ambulance driver to avoid taking me to the low-volume facility.’ However, correlation does not imply causation; hence policymakers deciding whether to regionalize care require more sophisticated statistical analysis than do patients deciding where to obtain care.

    That final paragraph makes two very good points. First, volume is informative about quality. Second, volume does not necessarily produce quality. Keeping those in mind, one should be very careful about whether and how to influence volume. The authors note elsewhere,

    [I]f ‘selective referral’ explains the relationship between volume and outcome, then regionalization could actually worsen outcomes if the wrong hospitals are chosen for expansion. The benefits of regionalization must also be weighed against the cost of transporting patients over long distances (Birkmeyer et al., 2003), and hence, knowing the magnitude of the ‘practice makes perfect’ effect is important. Failure to take account of the ‘selective referral’ effect will tend to exaggeratethe estimated ‘practice makes perfect’ effect, biasing policy recommendations toward regionalization.


    Omitted because of the existence of an ungated working paper. Most or all references should be found there. Of course, they’re all in the gated version too.

    Comments closed
    • Austin
      This is always interesting stuff, did not know subject was on your radar.

      Speaks to policy, and implicit in specialty hospitals is funding and changes in the law–and current battles, particularly in TX. Also, Leapfrog’s (? misguided) emphasis on volume preferred centers.

      But other practice and measurement issues:
      1) CABG functions both on a surgeon AND hospital volume axis (crummy CT surgeon in a good hospital vs converse). Unlike a hospital’s treatment, of lets say CHF, COPD, etc., this kind of diagnosis easier to measure if not because of narrow nature of primary DRG, but isolating reasons for success. A chronically ill medical patient much harder to disentangle (i am not implying former easier to disentangle, but less challenging).
      2) The endogeneity problem: would be interesting to see how outcomes effected in small cities where hospital system consolidation through mergers/closings effect outcomes. Puts the volume or practice makes perfect in a better place to assess. Unifies practice playing field with potentially less market dynamics, more quality dynamics at play.
      3) This is a narrow surgical diagnosis like mentioned above. Outcomes are likely more dependent on hospital approach to care overall–although Dartmouth variations may indicate different service lines perform differently under same hospital roof. Regardless, medicine much broader than bypass surgery, and cant make sweeping conclusions (think Joslin for DM, MSK for cancer).


      • @Brad F – Not really on my radar. Just read a paper. 🙂 Your points about surgeons was on my mind. I nearly edited the piece to say so, but decided to keep it simple, even at the risk of incompleteness.

        I’m a sucker for a good “things aren’t causal the way you think they are” story. 🙂

    • Couple of thoughts. First, I think that networks triumph referral effects. Along the same lines, I am always amazed at how unwilling most patients are to travel to a new place, even if it supposedly has better care. Talk with the people in marketing and they complain about this all of the time. We have better outcomes than our competitors in many areas. It does not seem to matter.

      On surgeons, those numbers can be manipulated. You need to look at surgeon and facility. To get the best picture, you really should be looking at even broader numbers. If your surgeons get patients only after they have already had seven stents, they may have different outcomes than those with fewer interventions prior to CABG. Since cardiologists are the gatekeepers, and they have their own incentives, this data is always murky and hard to sort out IMHO.

      Still, a very nice thought piece. Besides quality, I would expect that we should see opportunity for controlling costs, but it is not clear that happens either.