• The rest of our real jobs

    Aaron did a nice job of describing the trials and frustrations of the grant application and award process. In brief, it can take years for a good idea to finally get funded. His salary depends on grant funding, and so does mine.

    But the challenges don’t end with getting funding. You then have to do the work. That, of course, is the fun part. And, when the work is done, you’ve got to publish. Moving a paper from regression output to print can take years too. Just getting the first two or three reviewers to read your paper itself can take the better part of a year or more.

    Meanwhile, what the world needs is timely, policy relevant research results. How on earth does one predict what the policy relevant questions will be several years ahead so one’s grant is funded and the paper is published just as the world needs your answer? Well, most people can’t do it most of the time. It’s too hard. The time scales are too long. And policy is too variable.

    Fortunately, I work with Steve Pizer, who happens to be amazingly good at reading the policy landscape and seeing a few years ahead. Still, we get tripped up by the grant funding process or reviewers all the time. Sometimes our results are out after the debate has settled. Sometimes we’re too early because the debate shifted and then circles back (that’s OK). I’d have a lot more stuff coming out around now about health reform if only I’d received funding for it a couple of years ago. I tried. It’s hard!

    One other challenge to doing policy-relevant research is that it isn’t enough to have a good question or to get funding, one has to have data. It’s a pretty fine needle to thread. If it were only sufficient to have data, or only enough to have a good question, it’d be far easier. But you have to have both and get funding and then get published and try to do it when the policy question is relevant.

    I’m not complaining. But all this explains why we researchers defend our work pretty strongly. Once it reaches print, it’s gone through a lot of vetting. The idea and research plan has been worked over by a peer-reviewed merit process to secure funding. Then we’ve toiled for years trying to get the right answer and anticipate all the ways we could have deceived ourselves. Then the paper reviewers get to critique it. Sometimes the work is presented at conferences. We get a lot of feedback and pushback. When it finally gets published, we’ve heard most of the criticisms and dealt with them as best we can given the limitations of data and resources.

    So yeah, we’re pretty proud of what we publish. And we think it is right. All that scrutiny by others actually serves a good purpose. It doesn’t guarantee that published results are correct. But it sure does increase the odds.

    Comments closed
    • I’m curious to hear your and Aaron’s thoughts on the differences in publishing in the economics literature versus the health services research or medical literature. The two realms seems so different even though the topics and methods are often very close. At first blush, the idea that you could wait 3 years to get an economics article published after submitting it seems crazy in the medical/policy context, where a premium is placed on recent information, Medical journals have a much quicker turnaround.

      What do you see the pros and cons of publishing in each realm? Can the time horizon for economics publishing be justified?

      • @Aaron – My colleagues and I struggle with this all the time. We do policy, and sometimes clinically, relevant work but use econometrics that health services researchers and physicians are generally not familiar with. That makes our work topically suitable for a policy or medical journal but methodologically appropriate for an economics one. Sometimes we just cannot go to NEJM or Health Affairs because they won’t get it. We’re forced to JHE, IHFE, or Health Econ. There are others that split the difference in other ways.

        I’ve never waited 3 years for a paper to appear, but 1-2 is common in economics. If it is policy relevant, that’s a shame. It really cannot be justified. 20 of those 24 months are waiting for reviewers to do their job.

        When I receive a paper to review I do it in a few days. Sometimes the work of the review actually takes longer, requiring some background reading. My feeling is if one can’t do it in a month, one should say so and let someone else do the job.