I get questions all the time from people who ask how much we should trust medical advice when it seems like recommendations keep changing in ways that contradict themselves. How often does research wind up changing what we previously thought was true?
I’m pleased to point you towards a great study out in Archives of Internal Medicine, entitled The Frequency of Medical Reversal:
Background:
We use the term reversal to signify the phenomenon of a new trial—superior to predecessors because of better design, increased power, or more appropriate controls—contradicting current clinical practice. In recent years, a number of such reversals have occurred. Use of hormone therapy, the class 1C antiarrhythmic agents, and the pulmonary artery catheter have decreased when trials demonstrated that they are either less effective than previously thought or harmful. Reversal not only affects medications and diagnostic tests. Previously accepted indications for surgical and medical procedures have also been contradicted. In 2007, the Clinical Outcomes Utilizing Revascularization and Aggressive Drug Evaluation (COURAGE) trial found no benefit to support percutaneous coronary intervention (vs optimal medical therapy) in many patients with stable coronary artery disease, an indication that was previously accepted. The implications of reversal are notable. Reversal implies error or harm to patients who underwent the practice in question, during the years it was considered effective. Reversal also undermines trust in the medical system. We sought to estimate the frequency of reversal by examining 1 year of original publications in the New England Journal of Medicine.
Other researchers have studied the rate of reversal in medical research. Studies of medical interventions are often followed by studies that either reach the opposite result or suggest the magnitude of effect was initially overestimated. Among high-citation count publications, Ioannidis found that 16% were contradicted by future studies, and another 16% were found to have smaller effects than initially thought. Herein, we focused on existing practices that were contradicted in a given period in high-impact literature. Knowing the rate of, and predisposing factors for, reversal may have implications for the approval of medical therapies.
Methods:
We reviewed all Original Articles in the New England Journal of Medicine in 2009 (the last complete year of the publication at the time of our investigation). Articles were classified on the basis of whether they addressed a medical practice, whether that practice that was new or already in place, and whether the studies’ results were positive or negative. Two reviewers independently classified these articles (V.P. and V.G.). This yielded a highly similar profile (weighted Cohen = 0.94). Where there was disagreement, a third reviewer (A.C.) adjudicated those discrepancies. Next, we studied the precondition(s) that permitted reversal in each case. Two reviewers independently articulated the precondition (V.P. and A.C.), and these results were combined. This again yielded a highly similar profile (weighted Cohen = 0.85).
Let’s break this down. Reversals happen when newer and better research shows that what we are doing in clinical practice is wrong. Previous work has found that reversals have happened, but people argued that some of the original studies were bad or published in lesser journals. So these researchers decided to look at the NEJM (arguably the source for high quality medical research) for a specific period of time. In 2009, they found 212 “original articles” published in that journal, 124 (58%) of which were about medical practice. The rest were more basic sciencey or descriptive. Here is the breakdown of those articles that focused on medical practice:
- 72% looked at a new medical practice
- 73% were randomized controlled trials
- 66% reported a positive finding
- 49% reported a new practice that was better than current practice
- 10% reported on a new practice that was not better than current practice
- 13% confirmed that a current practice was good
- 15% were inconclusive
- 13% reversed a current practice
It’s that last bullet point we’re interested in. I can’t tell if it’s behind a paywall, but here’s a link to a table that describes the 16 practices that were reversed in the NEJM in 2009, including reasons as to why the initial practices failed. Interestingly, 13 of the 16 reversals included the reason, “confidence that the pathophysiologic concepts underlying the practice were rational.” In other words, we thought we understood what was going on, but we were wrong.
I make light of my book sometimes (now on sale for one week), but the reason we started doing work on medical myths was because we felt that physicians (and people) spend too much time worrying about new things and not enough time thinking enough about what they already assume to be true. The problem is, we often think we know what’s going on, but are wrong. This study is a flashing red warning light that this mistake occurs too often. These researchers were only looking at one journal for one year, and more than 10% of articles on medical practice showed that what we were doing was wrong, mostly because our assumptions about what we “knew” were flawed.
I know the NIH is fixated these days on innovation and the new, but there are times I wish we’d spend more time investigating what we “know” to be true already. I bet more of it is wrong than people think.
