Fascinating new paper in the International Journal of Epidemiology. “Why do we think we know what we know? A metaknowledge analysis of the salt controversy“:
Background: Although several public health organizations have recommended population-wide reduction in salt intake, the evidence on the population benefits remains unclear. We conducted a metaknowledge analysis of the literature on salt intake and health outcomes.
Methods: We identified reports—primary studies, systematic reviews, guidelines and comments, letters or reviews—addressing the effect of sodium intake on cerebro- cardiovascular disease or mortality. We classified reports as supportive or contradictory of the hypothesis that salt reduction leads to population benefits, and constructed a network of citations connecting these reports. We tested for citation bias using an exponential random graph model. We also assessed the inclusion of primary studies in systematic reviews on the topic.
So, so, so many papers on salt and health. I should know, I reviewed many of them when I wrote my Upshot column on it. These researchers found 269 reports, including 68 primary studies, 11 guidelines, and 14 systematic reviews. About half supported the idea that salt reduction leads to population benefits, about a third contradicted it, and 14% were inconclusive.
But that’s not what interested me. I’d already done the review myself and didn’t expect to learn a new “truth”. What blew me away was how badly these studies took sides, even in the other research they cited. The reviewed publications were 1.51 times more likely to cite other papers with a similar conclusion than those that disagreed with them. Cherry picking, proven with data.
There were 48 primary studies selected for inclusion in the 10 systematic reviews. You would hope that there would be some consistency there, at least. But they found that if a paper was selected for inclusion in a systematic review, it only had a 27% chance of being included in a later review. Studies were more likely to be included if they supported the salt reduction hypothesis than if they contradicted it. Cherry picking, even in the “systematic” reviews.
This was odd, too: About a quarter of authors produced 75% of the supportive works and another quarter produced 75% of the inconclusive works. In other words, “the literature in the field was dominated by a few reports and by a few prolific authors, who each hold and repeat particular positions.”
I try very hard to make sure I’m not cherry picking, often by relying on meta-analyses and systematic reviews. This is a good reminder that even there, biases abound.
