• Numbers stuff every reporter should know

    I was emailing with some journalist-friends about what reporters on policy or social science beats should really know. It’s unreasonable to expect reporters to have the full skill-set of academic empirical research. I may not be a representative voice on this issue, but I think the below set is pretty reasonable. Roughly speaking, these are skills I would expect of the median undergraduate public policy major across the country, especially an undergraduate studying what’s on the reporter’s own beat:

    1. Knowing how to interpret the size of a linear regression coefficient.
    2. Understanding what statistical significance means–and doesn’t mean.
    3. Understanding what r-squared is.
    4. Having some sense of why regressions go wrong—for example, the possibility of selection bias and reverse causality.
    5. Understanding the inherent limitations of cross-sectional analyses such comparisons of mortality across states with different levels of inequality.
    6. Innate suspicion of any complex statistical analysis that lacks a compelling underlying causal story. This would include the fact that most elaborate non-experimental analyses that reach dramatic or counter-intuitive conclusions are simply wrong.

    I don’t think it’s reasonable to expect reporters to know anything that requires a derivative, the words “heteroskedastic,” or anything involving the term “instrumental variables.”

    If covering health policy or public health, I would add a few more items:

    1. Knowing what odds ratios and relative risks are, and how these concepts can be misleading when base rates vary.
    2. Understanding the basic properties of screening tests–e.g. the index card of formulas that link the sensitivity and specificity of screening tests and underlying prevalence of a condition to positive and negative predictive value.
    3. Familiarity with milestones such as the RAND Health Insurance Experiment, the 5-10 leading articles in the field and the classics such as Paul Starr’s and Ted Marmor’s books that provide the context of current health policy.
    4. Familiarity with basic vocabulary such as moral hazard and adverse selection.
    5. Understanding the basic mechanics of a clinical trial, including terms such as “intent to treat” and “effect of treatment on the treated group.”
    6. Understanding the rudiments of population genetics and basic facts about genes and chromosomes.
    7. Knowing basic numbers on U.S. health expenditures, overall and within the key categories subject to debate.
    8. Knowing basic numbers regarding leading causes of mortality.

     

    Unfortunately, we live in a stupidly innumerate society and popular culture. This puts reporters at a tremendous disadvantage, since they come out of this culture. They need to fight the stupid and tool-up.

    None of the above items is very hard. You can’t be a good film critic if you’ve never seen Citizen Kane, Star Wars, or the Godfather. You can’t cover a foreign country properly if you don’t know the language and culture. You can’t cover public policy properly if you don’t speak the language and lack bare-bones familiarity with the tools of that trade.

    That’s my list, anyway. Is it reasonable?

    @haroldpollack

    Share
    Comments closed
     
    • Very reasonable. Perhaps you could do another list on what practicing physicians should be familiar with in healthcare policy. Would it be the same? Do you think that most physicians are familiar with the list that you generated for this post. My gut tells me no.

    • Harold,
      Good concept and good thoughts. I like the first half better than the second half, and be inclined to translate that directly into health science (not much use for R^2, lots more worry about confounding). I have to say I am a bit doubtful about any list like this; checklists for literacy kind of remind me of No Child Left Behind. But no so doubtful that I will not blog about it and also suggest a few amendments:

      When dealing with epidemiology (public health science), there is a crucial step in between “what statistical significance is not” and “need to tell us the baseline”, which is that magnitude matters. Epidemiology (along with most microeconomics) is a science of measurement, but it gets reported as if it were physics or engineering, it terms of the mere existence of a signal. Yes, someone might report OR=3 without the baseline, but it is worse than that because they would report OR=1.2 exactly the same.

      In health research, much more so than in economics (though not exactly innocent there), research methods for all but simple medical experiments are ad hoc and inadequately reported. You might have selection bias, but you would have a hard time knowing. What is most important is researchers fishing through possible functional forms to find a result they like — that is more the rule than the exception. (Then again, given journalists did not question WMDs, can we really hope they will question semi-dishonest researchers?)

      Basic mental fact checking: If this claim were true, it would mean…? It does not get you very far for most research, but it immediately casts doubts on some of the most absurd public health claims.

      My favorite general principle about empiricism: In a paper, the phrase “ours is the first study to show…” should always be followed by “…and therefore it is probably wrong.”

      Ok, I’ll quit there to avoid commenting longer than the post.

    • I would add something about Type 1 and Type 2 error – and that the the magnitude of the effect is important – and needs to be understood in the context of statistical significance [we spend way too much time on the later and not enough on the former IMHO]

    • I’ll only focus on the first list, since I’m not a health economist.

      I mostly like the first list. One potential deletion: I’m not sure why R-squared is so important. So it is not clear to me why a reporter should know anything about it except that it’s not that important.

      And I think I might want a reporter to understand a bit about: (1) why experiments can help solve the selection bias and causation problem; (2) some common ways in which experiments can go wrong; (3) some common types of quasi-experiments such as regression discontinuity.

      • R^2 is certainly important to understand; as are its limitations.

        But knowing the predictive power of your regression is certainly useful.

        • Why is R squared important? If I have a study that estimates that treatment or policy Z has effect of size B, and I have some estimate of the standard error of that effect, and some ability to interpret whether that size B is large or not, I don’t think it really makes a huge amount of difference whether the R-squared is 0.10 or 0.80. All that this means is how much else is affecting the dependent variable that is not in my model. But if my main interest is the policy or treatment effect, and I have some criteria for determining what a large effect is, then I don’t see what additional information R squared gives me.

          Now, holding all else equal, “larger” effects that are more statistically significant will tend to be associated with estimates with a higher R squared. But whether an effect is large should be judged on substantive criteria, not simply by the percentage of variance explained.

          • I will add that maybe you’re interested in R-squared if you’re interested in forecasting, although forecasting is trying to minimize the out-of-sample prediction error. So maybe then you need to use other criterion to judge goodness of fit such as the Akaike Information Criterion, not R squared.

            R-squared may also give you some idea of whether there’s a lot of room for adding to the model, although I don’t really think it tells you much about whether the variables you do have in your model are more or less likely to be good estimates.

    • From my brief foray into health journalism, the Health News Review criteria (http://www.healthnewsreview.org/about-us/review-criteria/) were very useful (and were prominently displayed on every mousepad in the office). Most of them seem obvious, but many reporters fail to follow them.

      1. Does the story adequately discuss the costs of the intervention?
      2. Does the story adequately quantify the benefits of the treatment/test/product/procedure?
      3. Does the story adequately explain/quantify the harms of the intervention?
      4. Does the story seem to grasp the quality of the evidence?
      5. Does the story commit disease-mongering?
      6. Does the story use independent sources and identify conflicts of interest?
      7. Does the story compare the new approach with existing alternatives?
      8. Does the story establish the availability of the treatment/test/product/procedure?
      9. Does the story establish the true novelty of the approach?
      10. Does the story appear to rely solely or largely on a news release?

      In general, understanding the limitations of various study designs is probably one of (if not the most) crucial skills for healthcare journalists.

    • The problem is that knowing that stuff and acting on that knowledge will be a negative for attracting more readers.

    • That’s the point of the NIH Medicine and Media conference… http://prevention.nih.gov/medmediacourse/

    • I would increase the technical requirements a little. A reporter–really anyone who graduates highschool–should know what a derivative is and how to interpret it. Also, reporters who follow academic papers closely in fields like health economics that rely heavily on reduced form empirical work should be at least aware of some of the common identification strategies like instrumental variables, difference-in-differences and regression discontinuity, Even if they don’t know much about those techniques, they should at least know to look for them, and to be critical of papers that make causal claims without a clean identification strategy,

    • Could we also require that our politicians voting on health care policy also learn these numbers and studies?

      Steve

    • I want to make sure that I have a sound foundation in Health Policy. I already have a sound understanding of statistics.

      I am wondering what are the 5-10 leading articles in the field?

      Which of Ted Marmor’s books is considered classic?

      Thanks

    • ” stupidly innumerate society”

      uncommon knowledge comes with the burden of needing uncommon communication skills