Continuing the conversation on how to revamp peer review…yesterday’s issue of Inside Higher Ed has a piece by Scott Jaschik making the case that peer review is “dead, but just doesn’t know it” at least in the Humanities. I don’t think that is true, at least not in health policy/medicine/economics, nor should it be.
In the comments and in a few emails, folks were asking for more on the plusses and minuses of moving away from authors and reviewers being blinded to the identity of the other. In thinking this through I ran across Bradley Voytek’s interesting post in Scientific American blog on peer review back in November, 2011. The post is titled “What is Peer Review For?” and this answer is given:
The scientific peer-review process increases the probability that the scientific literature converges–at long time scales–upon scientific truth via distributed fact-checking, replication, and validation by other scientists. Peer review publication gives the scientific process “memory”.
This post has links to many interesting pieces on the purpose of peer review and its shortcomings, and I can’t do the rich dialogue justice, so just go read through the links if you are interested in this topic. I don’t have a neatly wrapped up bottom line, but the below points may amplify a few of my thoughts:
- At its best peer review can improve the quality of published research. This should improve our quest for scientific truth. That should be the goal
- I think (unnecessary) delay harms the process of identifying scientific truth (too fast also harms it if it causes error)
- I believe the primary positive effects of making identities of reviewers and authors known is to speed the review process, out of the belief that reviewers and authors will be embarrassed to delay too long in doing what they have said they will do (review the manuscript/complete revisions)
- In seeking scientific truth, I think that replicability is particularly important, and more openness of the negotiation between author/editor/reviewer could aid by providing more information
- The biggest positive of maintaining blind review is to prevent bias of reviewers either for or against publication based on reputation of authors or institutions
- Interestingly, in review of grants by NIH and many other funders, track record/ability of the team/institutional capability is an explicit review criteria. Is this at odds with goal of blind review in journals? There are many issues here including degree of risk taken with public research money….those decisions of course effect the pipeline of publishable research
- When I review papers I often think that I know who wrote the paper. Is that better or worse than actually knowing? How about being wrong?
In collecting more information, I found this interesting paper by Richard Smith in the Journal of the Royal Medical Society that reviews these issues (and studies) more elegantly than have I. The most important studies he discusses are related to randomized trials done that show that impact of blinded review on review quality; unfortunately, they provide mixed results with the initial trial showing an improvement in review quality due to blinding with follow up trials not doing so. Further studies that assessed open review also found no impact on review quality.* They raise many questions about reviewer training that I won’t review.
This is an important topic. The further into it I go, the number of questions I have increases, while answers decrease.
DT
*Key studies reviewed in the Smith paper (I have not reviewed all of the studies independently)
