• Underuse of trial registries

    Publication bias is a serious issue, threatening the credibility of medical and social science. Yet, it’s an issue about which something can be done. One thing would be to require all trials to be prospectively registered.* If that were done systematically, at least one could be aware of trials not leading to publication. If trial data were disclosed, one could also have some hope of assessing why.

    A natural enforcement mechanism is journals’ uniform refusal to publish studies that were not prospectively registered. In the absence of such steps, we only have the (biased) published record to go on. With so many trials sponsored by parties with a clear financial conflict (e.g., drug and device manufacturers), this is not an insubstantial problem.

    In a new study, published in BMJElizabeth Wager and Peter Williams conducted a systematic analysis of a random sample of 200 medical journals’ policy with respect to prospective registration of trials, as well as a qualitative analysis of reasons so few journals require it.

    Of the 200 journals sampled, 142 (71%) did not require registration (or at least did not mention this on their website), 55 (28%) required registration, and 3 (2%) encouraged registration but did not make it a requirement for publication. […]

    Our study shows that most journals that publish clinical trials do not make prospective registration a requirement for publication or even encourage it in their web based instructions. The editors and publishers we interviewed proposed several reasons why journals might not require trials to be registered, or might not enforce a trial registration policy strictly. These included fear of losing good submissions to other journals, concern about preventing publication of studies from developing countries, and scepticism about the value of insisting on registration for small or exploratory studies. Other reasons why journals may not have a policy on trial registration include not publishing many primary trial reports. […]

    Although prospective trial registration is predicted to reduce publication bias and selective reporting of trials and outcomes, only a minority of journals that publish clinical trials make it a requirement for publication and many editors do not seem convinced that they should adopt such a policy. This suggests that journal editors may not believe that the benefits of trial registration are sufficiently important to be a mandatory requirement, or have concerns that requiring registration as a condition of publication would harm their journal, or both. As one editor (referring to small trials) said “it seems hardly worth the effort.” [Emphasis added.]

    Among the other results, a collective action problem is clearly identified. From a scientific point of view, we’d all be better off if all journals required trial registration. Yet, any given journal can offer an easier route to publication by not doing so. No doubt some journals can distinguish themselves as more rigorous and prestigious by adopting a registration requirement policy. However, there is heterogeneity in authors’ (and funders’) preferences. No doubt, some just want to publish. They’re willing to pay for it, and don’t care as much about the science or consequences of bias. Indeed, they may not be consciously aware of it. It’s a race to the bottom and, apart from the sponsoring stakeholders and, perhaps, the authors and journals that benefit, we all suffer for it.

    * An extension to observational studies is to prospectively disclose analysis plans. This would provide the information needed to assess whether published results are the outcome of “fishing expeditions,” i.e., running regression after regression for outcome after outcome until something pops up as significant, as eventually something will just by chance alone.

    @afrakt

    Share
    Comments closed
     
    • I understand Frakt’s position as a researcher, but pharmaceutical companies must consider how a prospective registration system may affect their stock. A prospective registration could have a material effect on the company’s stock price, both positive and negative. Indeed, a company may be reluctant to register a trial unless the likely results are positive for fear of accusations of manipulating its stock – the very existence of the trial suggests positive news for the company that would have a positive effect on its stock price which would be more than offset if the results of the trial aren’t positive. This is a dilemma

      • If a company adopts a policy of registering all their research, they can’t be accused to manipulating stock prices. They might have more volatile stock prices as registering promising studies, or failing to publish previously registered studies occurs. They won’t be able to control when the news gets out that a promising new drug has failed to live up to its promise.

    • Don’t both i) the uncertainty that a particular manuscript will be accepted by the first choice journal and ii) data ownership cause problems with both the trial registration and the prospectively disclosed analysis plans? W.r.t. the observational studies, could journals not just ask the authors to disclose the range of the number of regressions run? Ten to 100 would require a different significance level than 1,000 to 2,000.

    • This is a great idea, I bet that over time it would show that most nutritional study results are wrong.