What now seems like ages ago (but was only in early April), Joe Doyle and colleagues published an NBER paper finding, in their words, that “higher-cost hospitals have significantly lower one-year mortality rates compared to lower-cost hospitals.” The paper has already been discussed in the blogosphere. See Sarah Kliff’s first and second posts, Thom Walsh, and David Dranove, for example.
Given all that commentary, which you are perfectly capable of reading along with the abstract, there’s little point in me describing or critiquing the results. Instead I’ll focus on some aspects of the methods.
With acknowledgement of their imperfections, randomized trials are considered the gold standard for causal inference. One of the fundamental problems with studying the spending-outcomes relationship, however, is that we can’t randomize individuals to spending levels or even to hospitals. Instead, we must rely on the data we observe. If we’re clever, we can find something that is almost like randomizing patients to hospitals (or EDs), though. In this regard, Doyle and colleagues were extremely clever.
We consider two complementary identifcation strategies to exploit variation in ambulance transports. The first uses the fact that in areas served by multiple ambulance companies, the company dispatched to the patient is effectively random due to rotational assignment or even direct competition between simultaneously dispatched competitors. Moreover, we demonstrate that ambulance companies serving the same small geographic area have preferences in the hospital to which they take patients. These facts suggest that the ambulance company dispatched to emergency patients may serve as a random assignment mechanism across local hospitals.
Our second strategy localizes the “natural randomization” approach adopted by the Dartmouth researchers by exploiting contiguous areas on opposite sides of ambulance service area boundaries in the state of New York. In New York, each state-certified Emergency Medical Service (EMS) provider is assigned to a territory via a certificate of need process where they are allowed to be “first due” for response. Other areas may be entered when that area’s local provider is busy. We obtained the territories for each EMS provider from the New York State Department of Emergency Medical Services, and we couple these data with unique hospital discharge data that identifies each patient’s exact residential address. This combination allows us to compare those living on either side of an ambulance service area boundary. To the extent that these neighbors are similar to one another, the boundary can generate exogenous variation in the hospitals to which these patients are transported.
If this doesn’t fill you with admiration you’re probably not an economist. In that case, trust me, they have found an exceptionally good source of quasi-randomness in patient assignment.
Not long after I read this, I noticed this bit in a post by Jordan Rao about a recent paper by Emily Carrier, Marisa Dowling, and Robert Berenson:
The paper, published in Health Affairs, found hospitals “wooing” EMS workers that service well-off neighborhoods, even sprucing up the rooms where the workers rest and fill out paperwork.
This is a new phenomenon and, therefore, doesn’t detract from my admiration for Doyle et al.’s work, which focused on the early-to-mid 2000s. I raise the issue of hospitals trying to attract EMS workers from more affluent areas to suggest that in the future, an approach like Doyle et al.’s may have to address this type of thing. To the extent certain hospitals preferentially choose patients (e.g., more affluent ones) by influencing EMS workers, it is possible ambulance transports do not serve as a random assignment of patients to hospitals. What if higher spending hospitals are also the ones that play this game, attracting a more wealthy set of patients? If that were the case, it is likely that there are other unobservable characteristics of those patients that are correlated with outcomes. That would be a source of bias.
This raises a more general point about the ambulance transport approach. It only addresses demand-side selection. The patients are (quasi-) randomly assigned to hospitals, in a way (potentially) not correlated with hospital spending. But that does not mean there aren’t unobservable (non-random) aspects of hospitals that are correlated with spending and outcomes. The quasi-randomness of ambulance transport does not address this supply-side selection.
Good quasi-randomness is hard to find. Doyle et al. found some. Still, it doesn’t address every source of bias, nor should anyone expect as much from any study, even randomized experiments.
