• Is open science feasible with human subjects?

    The short answer is no. However, I haven’t been able to get this question out of my mind since attending a session on open notebook laboratory science at the ScienceOnline 2012 conference last week at N.C. State University. Is there something of use from open science techniques that could be brought to the health policy/health economics work that we do?

    A bit on definitions. Open science is the practice of immediately making all results of the research enterprise publicly available, while open notebook is the practice of making a lesser amount of data publicly available, with other information withheld for a variety of reasons (volume, usability, legal restrictions, worries about being scooped, etc.). The idea is that immediate disclosure of results increases transparency and allows for the possibility of feedback along the way in a research project (Open Science Federation, @openscience are advocates of the general approach). It is unclear to me if true open science (immediate sharing of everything) actually exists, and I think this movement is really scientists with differing degrees of open notebook approaches (I may be wrong).

    My initial reaction upon hearing an example at the conference (research seeking the boiling point of a metal while modifying the context and environment) was to say this doesn’t apply to what I do, since most of my work is based on human subject data that we collect, or on secondary data (like Medicare claims); in both cases, access to the data is expressly limited by legally binding data use agreements and IRB restrictions.

    However, the fascinating session, another on peer review, and some extended discussion the next day over coffee with Anthony Salvagno, a graduate student in Physics and Astronomy at the University of New Mexico who works in an open notebook lab (here is his notebook) and Andy Maloney, a post-doc at the University of Texas at Austin who trained in an open notebook lab, has kept me stewing on ways in which the essence or spirit of open notebook science could be brought to good effect to the kind of research that I do, and that most readers of this blog would find of interest.

    I am thinking of trying an experiment of sorts and taking a specific paper from a grant, and blogging (on a new blog, not here) a few times a week about the conduct of the research process for that paper. I will tell the IRB ahead of time, and ensure I don’t violate any restrictions. This will really be blogging about how we are thinking about answering a question. Here are the papers we have reviewed, have we missed some? Here is how we are dealing with missing value issues with the explanatory variables. These are the regression models being considered, anyone got a better idea, etc. Does anyone have experience with this type of approach in health policy research, or other work using human subjects data?


    • This is a great idea, but I hope we can focus on more proximal, possible goals first. Here are 3.

      Open analysis: In the quantitative social sciences, it is standard to place a paper’s statistical coding for the data analysis on the corresponding author’s web-site. A methods section is always clear if you can read the code. I have never seen this in a medical journal. This doesn’t require every wrong tree and change of plans, just the polished, final product to assess basic replicability and accuracy.

      Open protocols: Study protocols and IRB forms should be standard appendices in journals. Changes between the early IRB and the final analysis should be explicitly explained. This happens, but very rarely.

      Open data: Most importantly, de-identified original patient data should be made available for all studies that are published in medical journals or are the bases of marketing or guidelines. Publication bias is common (Ross BMJ 2012), clinically relevant (Turner NEJM 2008), and undermines the credibility of all clinical research (Lehrer, New Yorker 2010). Without public data we will never be sure that we’re seeing the full picture.

      None of these changes require Open Notebooks, but they would be a mammoth step forward from what we have now.

    • @j
      you make excellent suggestions/goals, 2 and 3 would seem to be easy to pull off. Points 2 and 3 also dovetail with some of my earlier thoughts on changes to pee review http://theincidentaleconomist.com/wordpress/a-few-suggestions-on-peer-review/ and http://theincidentaleconomist.com/wordpress/a-few-suggestions-on-peer-review-ctd/ especially in making public after the fact the negotiation between editor and author. I think unblinded review is better, though the more I read about it the more muddled I become. How to improve quality of peer review, while making sure any fix is not worse than the problems is a key issue for us.

    • Don,
      From the perspective of a student, it would have been profoundly useful to have a series of posts like what you’re describing while I was writing my undergrad honors thesis. Just seeing the kinds of research issues that other people encounter, and the kinds of thinking that help them get over hurdles, would have made the process loads easier.

      Also, seconding the “put your code online” comment. Even in economics, that’s less well-followed that would be optimal.