• The power (and weakness) of peer review

    There’s a tension in academia between work that has and has not been peer-reviewed. Readers of the blog will know that – in general – I heavily favor peer-reviewed work. I’m not alone in this; most people in research feel the same way.

    Because of that, I’m continually faced with questions about the value of what I do here. Many of my colleagues wonder why I don’t write things and send them in for publication in medical journals instead of writing them for a blog.

    I’m sensitive to these arguments, but I think they are missing the point.

    First of all, let’s own the fact that peer review isn’t perfect. As a friend of mine said repeatedly this last weekend, “look how well peer review worked for Andrew Wakefield.” We can’t pretend that just because it’s had the “stamp” of peer review, that it’s “truth”.

    Unfortunately, that’s the perception of many in the general public, though. If it’s peer-reviewed, then it’s “correct”. That’s not the value of the process, however. It’s that in order to be peer reviewed, the methods must have been examined closely and made public. That’s the value of peer-review. If I know a study is peer-reviewed, then I can go look it up, read it for myself, and judge it. I know enough is there for me to make up my own mind as to the validity of the results.

    Thus, when one of you sends me a study and tells me it says one thing or another, I can not only tell you that I agree or diagree with it, but I can tell you why.

    Using the example above, I could go read the original Andrew Wakefield study in Lancet, come back, and tell you that it didn’t prove that the MMR vaccine caused autism. His “study” was a case series of children with neurological disorders where a number of parents thought that maybe the MMR vaccine was associated with the onset of the disease (assuming that what was written was not a massive fraud).

    Peer review doesn’t guarantee quality. It makes it more likely, and additionally guarantees that the means for you to assess quality for yourself are available.

    Moreover, I just don’t feel like I could do as much good with this work if I tried to go through medical journals. I could send in a piece, they could take months to review it, and then after having it watered down to please everyone, it might see the light of day in 6 months, long after anyone would care. Traditional peer-review just doesn’t work in this domain.

    The down side of this is that although I’ve written more on this blog than the sum total of all my other writings, what I say here doesn’t “count” as part of my real job. The truth is that – to my employers and to many of my colleagues – the least of my manuscripts is more significant than this blog. I’m not saying this out of spite or with any negative emotions at all; it just happens to be true.

    I (and I’m sure Austin and Don) would disagree. I think this has tremendous value, not only I hope for all of you, but for me as well. Moreover, I’d argue that my writing is peer-reviewed every day. Other wonks read it, comment on it, and judge it. Health policy experts and journalists pick it up or don’t. And readers come back or go elsewhere. It’s peer-reviewed – just by a much broader group of peers.

    • Review seems vitally important as a way to judge accuracy. One question is the quality of review. Are anonymous referees likely to do a better job than those who comment on blog posts? That seems an empirical rather than a priori question.

      Most comparisons I’ve seen of Wikipedia to published encyclopedias show that wikipedia does at least as well. That may be a good analogy for publication v. blogging. Of course, there is the issue of how to judge quality, which has a certain circularity to it (how to judge the judges).

      Blogging does have the advantage of speed compared to formal publication.

    • Hey Aaron,

      I, a random individual who reads your blog daily via rss, disagree too!

      Layfolk don’t read journals — subscriptions are expensive! — and, being unfamiliar with the conventions of academic writing, find the content there opaque and hard to read.

      And here you guys come along throwing a voice that’s informed by your scholarly expertise and precision into the semi-insane jumble of soundbite, polemic, and gossip that constitutes mainstream political discourse. You are doing the world a service! For free! You should be proud! The internet should be grateful!

    • I can attest that your efforts on the blog are incredibly valuable — to me. I have learned so much, my understanding of the issues has become so much deeper. I am truly grateful for you efforts.

    • Peer review continues the traditional model of paper journals. This model, a point on which I think we agree, no longer functions at the pace of science nor allows for the conversations that are now at the core of scientific discovery. A journal “paper” was once just that – a single, authoritative printed replica of the original found in a bound volume in a library, monolithic and unchanging. Peer review stopped “junk” from getting enshrined in the Tomes of Science.

      Now, we have the ability to make an open, transparent conversation the real core of the discovery process. Peer review is no longer a gatekeeper, but a conversation starter. Blogs, Wikipedia, etc… are moving this information sharing process forward. We share more freely, but we trade off the authority and veracity of the previous system. We could have faith that at least two good sets of eyes had been on the paper. Today, we can get a million eyes on a paper, but we don’t have the assurance that the same quality of review is, in each piece, produced from that review.

      Unfortunately, I think the crutch holding up the old paper system is the publish-or-perish mentality. How would a tenure committee consider the scientific contribution of a blog? I know our tenure committees all want to see thick CVs with publications in top-tier journals. You seem to be able to do both. Congrats! I hope by the time I can get tenure, there is more value placed on scientific blogs, wikipedia curation, etc…

    • I think that you guys also function as aggregators. We need people who have extensive knowledge of past literature, who are also familiar with current issues. This has always been true of medicine in general, where fads come and go. It seems just as important in health care policy, where people cite ideas which have already been studied and found wanting or supported, as the case may be. Just knowing where to look for prior studies is very helpful in policy debate. It is a shame in some ways that only original research counts when it comes to academics.