• Rating media coverage of the Oregon Health Study

    I’m back from vacation and just trying to keep everyone honest. Or pick a fight. To avoid conflicts of interest, I won’t rate Aaron and Austin’s coverage. But you can in the comments.

    UPDATED to add Leonhardt’s NYT article, h/t to Dennis in the comments


     PublicationAuthorCorrectly identifies the important findings of the study?Notes the methodological strengths or weaknesses of the study?Does not improperly extrapolate or interpret the study?
    ★★★★WSJAnna Wilde MathewsYes – in the lede and the opening paragraphsSubstantial explanation in the middle of the articleCarefully noted the authors’ caution about extrapolation
    ★★★★SlateRay FismanYes – in various paragraphs across the story; story led with policy contextSubstantial explanation in the middle of the articleSubstantial explanation of limitations in the penultimate paragraph
    ★★★½Wash PoEzra KleinYes – in detailLeads with the methodology; substantial discussionMentions some limitations, but not fully
    ★★★½The AtlanticMegan McardleYes, in the middle, after critiquing improper extrapolation by othersDescribes problems with prior studies; great description of “Rorschach effect;” fails to describe the unique methodology of this studyVery careful description of problems extrapolating the study to mortality
    ★★★NYTDavid LeonhardtYes, with good email quotes from several of the study authorsYes, but did not explain why the lottery methodology was superior; discussed the political backgrounds of the study authors to suggest bipartisanshipFocused on  “health” outcomes in the article without a careful explanation of the authors’ cautionary statements
    ★★★NYTGina KolataYes – in the lede and the opening paragraphsSubstantial explanation in the middle of the article + brief mention in the ledeDid not note the authors’ caution about extrapolation
    ★★Think ProgressMatt YglesiasYes, but includes “saves lives” which was not in the studyMentions study design and calls the study “rigorous” but does not tell us whyFails to mention the authors’ caution and improperly extrapolates to mortality
    ★★CatoMichael F. CannonNo – goal of article is to critique Yglesias’ coverage of the studyNo – does not mention the methodologyCritiques Yglesias’ coverage of the study and tries to prevent (improper) extrapolation
    Huffington PostNo bylineYes, brieflyNo mention at allNo mention at all

    • My favorite coverage item came from the Weekly Standard:


      Is there such a thing as negative stars?

      • That Weekly Standard article failed to mention the methodology at all, attacked the findings based on funding from CMS and RWJF, presented some of the results while ignoring many others, and then cited studies with an inferior design.

        It would rate zero stars.

    • I don’t get your giving most of the blog authors good marks for explaining the methodology. Unlike Leonhardt of NYT (not on your list), I don’t think any of the blog authors you chose explained the distinction between a surveyed person winning the lottery and not winning the lottery and a surveyed person having or not having Medicaid or some other type of insurance. A few of the blog authors alluded to the difference but in general the strong impression is left everywhere in the press (not just in these articles you mention) that this research represents a straightforward survey of 10,000 Medicaid recipients vs. some number (varies by blog author) of uninusred people.

      As I understand it, winning the lottery (and therefore being in the “treatment” survey group) and losing the lottery (and therefore being in the control survey group) did not mean a person answering the survey a year later had insurance. The research authors say in their abstract that the treatment group is “25 percentage points” more likely to have insurance than the control group but they never indicate that I could find what those absolute numbers are. Given other information in the research, it looks like about 30% of the control group was insured at the time of the survey and 55% of the treatment group was insured at the time of the survey.

      The research authors then model this information into the differences they see between insurance recipients and the non-insured. Then the research authors assume (which would seem to make sense) that the insurance that the treatment group has is Medicaid.

      (Also note that this does not appear to be vanilla Medicaid, but more like Massachusetts’ Commonwealth Care. At least it looks like some recipients pay at least a nominal premium.)

      If I am describing this correctly, I haven’t seen it explained anywhere. Hopefully one of you can do that better than I. As I said, Leonhardt comes closest. (see http://economix.blogs.nytimes.com/2011/07/07/how-health-insurance-affects-health/ — paragraph 4). If Leonhardt and I are wrong, delete this comment.

      • The treatment group was LOTTERY, but they also modeled INSURANCE. You are right that some of the results reported in the study are just for giving people the opportunity to apply for deeply discounted Medicaid.

        Our blog is research-based health policy. The methodology for this study is novel, and deserves prominent discussion.