• Oh come on! This just won’t do.

    Some readers helped locate the source for the figures in that infographic I blogged about yesterday. (Go read that post if you don’t know what I’m talking about.) See the chart on page 28 of this report (PDF). The source says “Institute for the Future, University of California-San Francisco, CDC.” What’s that? Well, back on page 14, those questionable stats are described in words:

    Access to health care accounts for a relatively small percentage of health status (as little as 10 percent, for the entire population), while behaviors that promote or threaten health (“health risk factors”) account for as much as 50 percent of health status. [15]

    Aha! Finally, a pointer to a full reference, to be found under item 15 in the end notes. Page forward and, lo and behold,

    15 Original unpublished research: University of California at San Francisco, the Institute for the Future (Menlo Park, California), and the US Centers for Disease Control and Prevention.

    The bold is mine. I’m sorry, this just won’t do. It’s another citation to nowhere. How about a URL? Anything? It doesn’t even have a date of (un)publication on it?

    It seems like folks want to say that 10% of our health is due to the health system (or access to it) without actually being able to point to anything substantial to back it up. Maybe my readers with super Google skills can track this down. I’m not saying there’s nothing to this 10% figure. I’m saying we shouldn’t have much confidence in it unless and until we can see the research that supports it. Where is it, and why didn’t the Bipartisan Policy Center track this down and cite a source that has some actual work behind it?

    It might turn out that there is some good work backing up the 10% figure, but I want to write for a moment as if there isn’t (because right now we don’t know if it exists). This type of thing — vague citations of august sounding institutions with no published work — is terrible scholarship. It has the appearance of evidence-informed policy analysis, but it isn’t. On many questions, there is plenty of good research to cite. On some questions, there isn’t. In the former case, we should stick to the quality evidence. In the latter, we should just admit we don’t know.


    • Thanks for calling out this lack of actual primary evidence. Like many public health people I am naturally disposed to believe that kind of “statistic” but just because we want to believe it doesn’t make an unsupported assertion acceptable.

      Also I look with suspicion on any data presentation that uses 3-D charts for effect (and obfuscation) without actually presenting three dimensions of data that require those three axes (see pages 25, 27, 28). The fact that Thomas Hubbard and his coauthors unnecessarily make 3-D graphics to dress up their data doesn’t increase my confidence in their assertions.

      Here are staff bios for the New England Healthcare Institute.
      I’ll send them a link to these two posts and ask them to comment.