• A few suggestions on peer review

    This post was also published in the LSE impact blog today.

    A few weeks back, the LSE Impact blog had a piece by Jason Priem on the use of twitter by academics that suggested peer review journals might become a thing of the past. Austin Frakt and I wrote a brief post noting that as much as we love twitter, the role of peer review journals cannot be replaced by twitter, blogs or anything else (and we really believe in blogs!). We need the slow deliberative process that emphasizes trying to get it right, as opposed to doing it quickly. We concluded:

    We absolutely need the slow, peer review system as the foundation of thoughtful, careful scholarship. Twitter and other social media are important additions that can give scholarly content “reach” and “relevancy”. However, it’s a both/and, not an either/or proposition. Traditional peer review journals should remain the bedrock of the research evidence that can be brought to bear on health policy.

    However, I think the peer review process often is too slow and could be sped up without losing precision. In addition, I think there is too much secrecy in the process and a bit more disclosure would likely be good (though there are likely pluses and minuses). Following are a few personal thoughts about changes I would like to see in the peer review process used by journals that are based on my personal experience and preferences (I have published ~70 peer review papers and reviewed dozens of manuscripts for journals). Others will likely have different thoughts, and I would be interested to know them. This is not meant to be a definitive word, just my personal thoughts.

    • The identity of reviewer and reviewee should be known to one another
    • The title of manuscripts under review should be public, along with the authors of the manuscript and the identity of the reviewers
    • How long the reviewers have been reviewing the manuscript should be public
    • How long authors have had a request for revision should be public
    • Upon publication, the correspondence between reviewers/editors/authors should be public (this is important because often people say “why didn’t you do this sub-analysis”; often it was done, but cut from a published paper due to length restrictions)
    • The use of online early publication is a good thing; I wonder if it will eventually become the only modality? (I only take one journal in hard copy now, Health Affairs, and otherwise utilize Duke’s global subscription service)
    • Gated papers hinder academic investigation and discourse, but I am unsure of how to fund journals without subscriptions

    Making the identify of all parties public and how long they have had to review a manuscript or complete revisions should provide some “speed” to the process. More information about the give and take leading up to the publication would provide a fuller context for the paper. And a big issue going forward is the financial model by which journals survive. Who should pay for them and how much?


    update: a few tweaks for clarity

    • I would like to hear more about the plusses and minuses raised by having the reviewer and reviewee’s identities not known to each other. As Mark van Roojen says here (http://bit.ly/yIq0hC) in a discussion among philosophers on peer review:

      “Blindness is really important. There is all sorts of evidence that factors irrelevant to scholarly quality have more influence when things aren’t blind. (To be clear what I mean by blind here is what many call double or triple blind where the reviewer and even the editor do not know the identity of the author.) I cannot see that any process that makes things less blind makes things better in at least that respect.”

      We are unduly influenced by all sorts of irrelevant factors. Should not manuscripts stand or fall as much as possible on their merits? Or does this bring unacceptable costs in terms of secrecy and unaccountability?

    • I also published papers and reviewed them. As a paper submitter, I occasionally thought a reviewer completely misunderstood/ was dead wrong with respect to a negative review; as a reviewer I tried VERY hard to understand and honestly review the paper but no doubt some times failed to fully understand — and may never have known about my mistake nor have known that I reduced the chance of that paper to be published. I’d welcome your “more open” suggestions both as a reviewer and a submitter.
      Supporting peer reviewed publications and the peer review process? Perhaps something akin to the proposed financial transactions mini-tax but on money flowing through research that may lead to opportunities to publish?
      And thanks for your work and your blog.

    • @Paul and Austin
      The biggest plus is improving speed via shame from being identified as being delayed. Openness seems to have a general benefit, all else equal. The biggest negative is the degree to which it makes peer review less fair. This was in Inside higher ed today about peer review and talks at least partly about the idea that reviewers may be able to tell who wrote it in any event http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycode=418635&c=1

      The true calculation is how fair it is in practice v. more openness. Also important is the goal of pee review: I think it is to increase the chance we obtain scientific truth. I have found a variety of sources that I will pull together in a follow up post.

      • I’m receptive to being sold on this, I just haven’t seen the killer argument yet. I think you’re getting closer with the idea that reviewers can figure out authors in many cases. Speeding up the process is also of value. Openness isn’t the only way to achieve that, though.

    • My understanding (and this is not based on direct personal experience or expertise!) is that the high-energy physics community effectively operates nowadays without a conventional hardcopy-publication peer review process. Initially, a server was established with the intent of providing easy dissemination of pre-publication results, and even though this was a glorified community bulletin board it rapidly evolved into a de facto journal. Of course, there is a very real form of peer review in this approach, because you post your paper and then everybody in the community can discuss it. However, the whole system depends critically on having a very well-informed (and presumably small) community, and it can easily descend into clique-ism (of course the high-energy physics community has been accused of that long before the server was set up). In fact, I suspect that if you examined the history of science closely, essentially every fast-moving field operated this way, with the real debate being conducted through seminars, oral discussions, and correspondence, with the paper journal publication just a formality.

      Incidentally, a process like this depends critically on being unblinded — if I’m going to read a manuscript reporting the results of an experiment, I’m certainly going to think about the reputation of the experimental team.