Hilda Bastian pushed back against recent promotion of the concept of the number needed to treat (NNT), including the post authored by me and Aaron at The Upshot. Bastian’s main point is that the NNT is too complex for patients, and therefore not helpful in communicating risk reduction of treatments. In her post, titled “The NNT: An Overhyped and Confusing Statistic,” she wrote,
[An NNT] takes data about patients, inverts it to a “treater” perspective, and then requires them to go through cognitive gymnastics to get back to their point of view.
Well before my Upshot post with Aaron published, I was aware that this was a common feeling about the NNT, which is why I dug into the literature on how well the metric is understood. I wrote up my findings last November on the AcademyHealth blog. You can read the details at the link, but here’s my conclusion: As to whether patients have difficulty understanding NNT, relative to other metrics of risk reduction,
the evidence base is both thin and inconclusive. There are two relevant studies about NNT comprehension included in two systematic reviews. They point in opposite directions.
My conclusion is that NNTs may, in fact, perform quite well relative to other measures. Or maybe they don’t. We don’t know. They may perform even better for practitioners than for patients, but we don’t know that either. What we can’t say from the evidence, however, is that NNTs are harder for people to understand than other metrics of risk. From two studies with conflicting findings, we just don’t know that.
Bastian, on the other hand, came to a strong conclusion about the comprehension challenges posed by the NNT. After considering natural frequencies, relative risk reduction, absolute risk reduction, and NNT, she wrote, “Which of these is the hardest for patients and clinicians to understand? The evidence points to the NNT being the hardest.”
Because our evidence-based conclusions differ, I tweeted at her. You can read her response below:
@afrakt @pash22 @stephensenn I did: I strongly disagree with your interpretation, & that post considered too little of the evidence.
— Hilda Bastian, PhD (@hildabast) March 3, 2015
I always take charges that I missed some evidence seriously. That’s not only because of a desire to be right, but also because I’m always eager to learn when I might have been wrong. What evidence did I miss? Looking for it, I went through every link of her post. Here’s what I found:
- As background, she cites a Wired magazine piece on NNT. Aaron and I also cite it, and I did in my AcademyHealth post as well.
- About NNT’s development, she cites Laupacis et al. (1988). Aaron and I do as well.
- Her evidence that the NNT is harder to understand than other metrics of risk reduction are the same 2011 Cochrane review and 2014 review by Zipkin et al. that I also examined. In my post, I dug into what they cited as evidence. It’s not very convincing, as I explained.
- Bastian cited a survey of urologists by Dahm et al. that I “overlooked.” (My focus was on patients, not physicians, but never mind that. I wish I had seen this study because I would have put it to use.) The study found that “[t]he percentage of respondents who indicated that they ‘understand and could explain to others’ the term[] number needed to treat […] was 42%.” That sounds, well, not so good, until you look at the results from that same study for relative/absolute risk (the only other risk reduction metrics considered): about 20% of responding urologists said they could understand and explain those ideas (see Figure 2, which I’ve included at the end of this post). So, this study hardly supports the idea that the NNT is more difficult to understand and explain than other common metrics of risk reduction.
- She cited qualitative work on how patients engage with evidence-based health information. I confess to not have considered qualitative work because one cannot use it to draw quantitative conclusions. Nevertheless, that work concluded, “Further research is needed to examine strategies to explain the specific character of EBHI [evidence-based health information] so as to minimize unintended or adverse reaction patterns.” More research needed! That’s not the same thing as “NNTs are confusing!”
- She cited work by Trevena et al. that I had not read. It argues that we need to communicate in a way that does not require people to do mental gymnastics they find challenging. OK, I fully agree with that. The goal is to find that way, which may differ by population and situation. So far, I see no basis for rejecting NNT from consideration.
Ultimately, I just can’t agree with Bastian—based on the evidence she cited, all of which I have now considered and the most relevant of which I had already—when she says,
[S]o many supporters of evidence-based medicine [] remain in favor of a form of communication [NNT] that itself is, let’s face it, pretty contrary to the evidence.
Well, it seems we support NNT as possibly useful because we have not seen a convincing body of evidence to the contrary. And, I should add, Aaron and I have not claimed that the NNT is perfect or superior for all settings. We find it valuable, as do others, but your mileage may vary. That’s fine!
For all that, I agree where Bastian ends up.
Statistics aren’t value-free, even though they are more objective than many other ways of distilling information. They’re difficult to translate for people who don’t already “get” them very well in their original form. You need lots of context and several statistics to get a handle on the results of most clinical studies anyway.
Quite right. But is it clear that the NNT can’t play a role? Dr. David Newman, a champion of the NNT and one of the creators of theNNT.com, put it to me this way:
The purpose of using the NNT is to refine communication so that the negative space (all those not affected) are never left out of the conversation, as they so commonly are. To talk about relative numbers as if they’re as good as or better is deeply misleading. They are much, much worse, and all studies validate that. They may leave people feeling like they understand, but they misunderstand, and in dangerous ways.
I would never advocate using the NNT in low literacy populations as a sole method of communication, or in any population where it is not understood and commonly used. But first, this is a tool that takes some getting used to. Once everyone’s on the same page it can be an absolute goldmine to have this device. And second, for those not acclimated, it should be used alongside other devices to ensure that someone understands, in the moment.
There is no one device that works for that—patient communication is an art (think: painting), and devices are the medium (think: paintbrush). The brush is much less important than the final product, and if a device helps you get there, then we’re all on board.
(You can read more defence of the NNT by Newman here.)
Still, maybe I have missed or misunderstood some evidence that strongly points to the NNT being too confusing to consider a worthy arrow in the quiver. If so, what is it? By email, Hilda Bastian generously offered to pull together some that had been left out of her post. I look forward to seeing it.
***
From a Survey of Urologists (Dahm et al.)
