It isn’t hard to find selection bias once you know how to spot it. Yet many people, including reporters and bloggers, don’t see or willfully overlook it, even in its simplest form. It is so much fun (even profitable) to report results that suffer from selection bias because they “show” such “interesting” things. I think the media actually loves selection bias and doesn’t know it.
Quick review: What is selection bias? Suppose a study finds that a group of individuals with larger feet have higher on IQs than a group with smaller feet. Does having big feet cause people to be smarter? No. Big footed people are older and have more education. The selection criterion (foot size) for the groups across which IQ is compared leads to a bias in the estimate of the causal relationship between foot size and IQ because the two groups differ in systematic ways that are important (age, education). Clearly taking growth hormone to increase one’s foot size is unlikely to make one smarter.
Recent posts by Kevin Drum (Mother Jones), Mark Blumenthal (National Journal), Matt Yglesias, and Ezra Klein (Washington Post) all draw conclusions from comparisons that potentially suffer from selection bias. The first three report the same comparison of survey results on the satisfaction of individuals with their health plan. Medicare is associated with greater satisfaction among its beneficiaries than is private insurance among its policyholders. Drum, Blumenthal, and Yglesias claim this indicates that people really like (or would like) a government run health plan better than those provided by private insurers. They suggest that the public nature of Medicare causes greater satisfaction. (By the way, Krugman has also implicitly made this argument too.)
That may be so, but these comparisons don’t illustrate that. The samples for which satisfaction is compiled for the different plan types are systematically different in ways that could bias the findings. The Medicare population is very different from the population that has private coverage. The most obvious way is in age, but also in health, income, and others. The documentation that accompanies the results upon which these statistics are extracted illustrates other ways in which the samples differ.
Ezra Klein’s selection bias problem arises in comparing survey results from populations of different nations on the proportion that thinks their national health system should be completely rebuilt. Do differences in the characteristics of the health systems of nations cause the differences in population level of dissatisfaction with those systems? Perhaps, but one can’t conclude it from the simple comparison that Klein reports. Not only do the health systems differ but so do the populations. Which is the cause of the differences in satisfaction levels? One can’t tell. It is the same problem as above: comparison across samples with systematic differences can produce biased results.
There is no way to know the extent of bias in studies based on non-randomized samples (such as those above) without using more advanced statistical methods (which, by the way, is what some health economists (and others) do–ahem.) Therefore, by themselves the results shown by Drum, Blumenthal, Yglesias, and Klein referenced above are useless in supporting their arguments. Sadly, reporting of results that suffer from selection bias is very common. They’re in the news every day and they’re meaningless.
