• Rating media coverage of the Oregon Health Study

    I’m back from vacation and just trying to keep everyone honest. Or pick a fight. To avoid conflicts of interest, I won’t rate Aaron and Austin’s coverage. But you can in the comments.

    UPDATED to add Leonhardt’s NYT article, h/t to Dennis in the comments


      Publication Author Correctly identifies the important findings of the study? Notes the methodological strengths or weaknesses of the study? Does not improperly extrapolate or interpret the study?
    ★★★★ WSJ Anna Wilde Mathews Yes – in the lede and the opening paragraphs Substantial explanation in the middle of the article Carefully noted the authors’ caution about extrapolation
    ★★★★ Slate Ray Fisman Yes – in various paragraphs across the story; story led with policy context Substantial explanation in the middle of the article Substantial explanation of limitations in the penultimate paragraph
    ★★★½ Wash Po Ezra Klein Yes – in detail Leads with the methodology; substantial discussion Mentions some limitations, but not fully
    ★★★½ The Atlantic Megan Mcardle Yes, in the middle, after critiquing improper extrapolation by others Describes problems with prior studies; great description of “Rorschach effect;” fails to describe the unique methodology of this study Very careful description of problems extrapolating the study to mortality
    ★★★ NYT David Leonhardt Yes, with good email quotes from several of the study authors Yes, but did not explain why the lottery methodology was superior; discussed the political backgrounds of the study authors to suggest bipartisanship Focused on  “health” outcomes in the article without a careful explanation of the authors’ cautionary statements
    ★★★ NYT Gina Kolata Yes – in the lede and the opening paragraphs Substantial explanation in the middle of the article + brief mention in the lede Did not note the authors’ caution about extrapolation
    ★★ Think Progress Matt Yglesias Yes, but includes “saves lives” which was not in the study Mentions study design and calls the study “rigorous” but does not tell us why Fails to mention the authors’ caution and improperly extrapolates to mortality
    ★★ Cato Michael F. Cannon No – goal of article is to critique Yglesias’ coverage of the study No – does not mention the methodology Critiques Yglesias’ coverage of the study and tries to prevent (improper) extrapolation
    Huffington Post No byline Yes, briefly No mention at all No mention at all

    Comments closed
    • My favorite coverage item came from the Weekly Standard:


      Is there such a thing as negative stars?

      • That Weekly Standard article failed to mention the methodology at all, attacked the findings based on funding from CMS and RWJF, presented some of the results while ignoring many others, and then cited studies with an inferior design.

        It would rate zero stars.

    • I don’t get your giving most of the blog authors good marks for explaining the methodology. Unlike Leonhardt of NYT (not on your list), I don’t think any of the blog authors you chose explained the distinction between a surveyed person winning the lottery and not winning the lottery and a surveyed person having or not having Medicaid or some other type of insurance. A few of the blog authors alluded to the difference but in general the strong impression is left everywhere in the press (not just in these articles you mention) that this research represents a straightforward survey of 10,000 Medicaid recipients vs. some number (varies by blog author) of uninusred people.

      As I understand it, winning the lottery (and therefore being in the “treatment” survey group) and losing the lottery (and therefore being in the control survey group) did not mean a person answering the survey a year later had insurance. The research authors say in their abstract that the treatment group is “25 percentage points” more likely to have insurance than the control group but they never indicate that I could find what those absolute numbers are. Given other information in the research, it looks like about 30% of the control group was insured at the time of the survey and 55% of the treatment group was insured at the time of the survey.

      The research authors then model this information into the differences they see between insurance recipients and the non-insured. Then the research authors assume (which would seem to make sense) that the insurance that the treatment group has is Medicaid.

      (Also note that this does not appear to be vanilla Medicaid, but more like Massachusetts’ Commonwealth Care. At least it looks like some recipients pay at least a nominal premium.)

      If I am describing this correctly, I haven’t seen it explained anywhere. Hopefully one of you can do that better than I. As I said, Leonhardt comes closest. (see http://economix.blogs.nytimes.com/2011/07/07/how-health-insurance-affects-health/ — paragraph 4). If Leonhardt and I are wrong, delete this comment.

      • The treatment group was LOTTERY, but they also modeled INSURANCE. You are right that some of the results reported in the study are just for giving people the opportunity to apply for deeply discounted Medicaid.

        Our blog is research-based health policy. The methodology for this study is novel, and deserves prominent discussion.