• A few suggestions on peer review–ctd.

    Continuing the conversation on how to revamp peer review…yesterday’s issue of Inside Higher Ed has a piece by Scott Jaschik making the case that peer review is “dead, but just doesn’t know it” at least in the Humanities. I don’t think that is true, at least not in health policy/medicine/economics, nor should it be.

    In the comments and in a few emails, folks were asking for more on the plusses and minuses of moving away from authors and reviewers being blinded to the identity of the other. In thinking this through I ran across  Bradley Voytek’s interesting post in Scientific American blog on peer review back in November, 2011. The post is titled “What is Peer Review For?” and this answer is given:

    The scientific peer-review process increases the probability that the scientific literature converges–at long time scales–upon scientific truth via distributed fact-checking, replication, and validation by other scientists. Peer review publication gives the scientific process “memory”.

    This post has links to many interesting pieces on the purpose of peer review and its shortcomings, and I can’t do the rich dialogue justice, so just go read through the links if you are interested in this topic. I don’t have a neatly wrapped up bottom line, but the below points may amplify a few of my thoughts:

    • At its best peer review can improve the quality of published research. This should improve our quest for scientific truth. That should be the goal
    • I think (unnecessary) delay harms the process of identifying scientific truth (too fast also harms it if it causes error)
    • I believe the primary positive effects of making identities of reviewers and authors known is to speed the review process, out of the belief that reviewers and authors will be embarrassed to delay too long in doing what they have said they will do (review the manuscript/complete revisions)
    • In seeking scientific truth, I think that replicability is particularly important, and more openness of the negotiation between author/editor/reviewer could aid by providing more information
    • The biggest positive of maintaining blind review is to prevent bias of reviewers either for or against publication based on reputation of authors or institutions
    • Interestingly, in review of grants by NIH and many other funders, track record/ability of the team/institutional capability is an explicit review criteria. Is this at odds with goal of blind review in journals? There are many issues here including degree of risk taken with public research money….those decisions of course effect the pipeline of publishable research
    • When I review papers I often think that I know who wrote the paper. Is that better or worse than actually knowing? How about being wrong?

    In collecting more information, I found this interesting paper by Richard Smith in the Journal of the Royal Medical Society that reviews these issues (and studies) more elegantly than have I. The most important studies he discusses are related to randomized trials done that show that impact of blinded review on review quality; unfortunately, they provide mixed results with the initial trial showing an improvement in review quality due to blinding with follow up trials not doing so. Further studies that assessed open review also found no impact on review quality.* They raise many questions about reviewer training that I won’t review.

    This is an important topic. The further into it I go, the number of questions I have increases, while answers decrease.

    DT

    *Key studies reviewed in the Smith paper (I have not reviewed all of the studies independently)

    McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990 Mar 9;263(10):1371-6.
    Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. JAMA. 1998 Jul 15;280(3):240-2.
    van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial.JAMA. 1998 Jul 15;280(3):234-7.
    van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ. 1999 Jan 2;318(7175):23-7.
    Share
    Comments closed
     
    • I don’t think that peer review is a particularly bad thing, and I’m certainly hard-pressed to come up with a viable alternative. But I’m also not convinced that it’s enormously helpful in improving papers. Everything depends on the quality of the reviewers, and it’s a rare quality to be good at doing this — reviewers are selected because they presumably have considerable experience in the field, but this is very different from being able to read a manuscript and make constructive suggestions about how to make it better. My experience has been that most reviewers’ comments tend to be uninsightful, largely on the order of “This isn’t the way that I would have done it”, and it’s unusual when I read a comment and think, “Yes, that would really improve the paper”. Most of all, I am constantly frustrated by reading papers in prestigious, high-profile journals, and being unable to think about basic questions because the paper was written in an awkward manner. So, sure, let’s keep the process, but let’s also not get our hopes up about what can be expected of it.