In a comment to my review of Angrist’s and Pischke’s The Credibility Revolution in Empirical Economics squidy wrote “You may also want to check out the response to this paper by Liran Einav and Jon Levin. That paper is titled Empirical Industrial Organization: A Progress Report.
In fact a paper by the same authors with the same title is one I noted on this blog that I would print and read. I did so, though it was the NBER version, which seems have disappeared from the web. I can’t be certain that the two versions of the paper are identical. However both include the following passage, which actually seems to support Angrist’s and Pischke’s view of new empirical IO.
In part because it is hard to find independent movement of many product prices, some of the most popular identification strategies rely on restrictions across equations in the demand system. One such approach is to use a product’s price in other markets as an instrumental variable, under the theory that cross-market correlation in the price of a given product, conditional on observed demand characteristics, will be due to common cost factors rather than unobserved features of demand. An alternative is to instrument for prices using the non-price characteristics of competing products, which proxy for the degree of competition.
Neither is a perfect solution, so the source of price variation and its power has to be evaluated in each application.
… Our own view is that many applications of these methods — while they are often very careful in clarifying the statistical conditions under which their identification strategy is valid — tend to be rather thin in explaining the precise source of identifying variation, in arguing why the required statistical condition is likely to hold, or in providing first stage regressions and other diagnostics.
So, there seems to be rather broad agreement that IO may have gone a bit too far out on the limb of creativity in its acceptance of arguments of identification.
Having said that, Einav and Levin do offer what I think is a worthwhile critique of Angrist and Pischke, and it gets to the heart of the real debate while abstracting from IO. There is a fundamental disagreement about the role of theoretical versus empirical evidence in applied economics. Where data are thin, one can make progress by leveraging theory. Where data are abundant, theory is still important but one requires somewhat fewer assumptions to obtain a result. How much credibility does work relying on a greater number of assumptions deserve? Well, this is of course debatable, and it matters a great deal what those assumptions are and the extent to which they comport with prior beliefs and methodological conventions.
The bottom line, however, is that the use of economic theory and the search for compelling sources of identifying variation are not enemies. Indeed, we hope to have conveyed that the applied work we often find most exciting relies on careful measurement based on data with good underlying variation, but then continues by framing the empirical exercise in terms of a coherent economic model. The model can then provide a way to think about the operation of the industry and potentially to draw conclusions about policy or general principles.
To the extent that we have a concern about the current state of industrial organization research, it is that there is not sufficient emphasis on this kind of applications, relative to, say, expanding the set of econometric methods. Of course, better methods are valuable, provided they eventually get used in compelling ways and do not become an end in themselves. If we return again to the demand estimation literature, it is possible that one reason researchers have been willing to tolerate less than ideal price variation is that in some cases the main contribution is not the estimated price elasticities per se but the econometric method, which can be applied more broadly. While this is not terribly objectionable, it is important that the field at large strikes a balance between building tools and using them convincingly. Whether the field has tipped too far is debatable, but the fact that one might engage in a serious debate suggests some grounds for concern.
With that last sentence, one senses that Einav and Levin really do not wish to argue too strongly against Angrist and Pischke. Both sets of authors have valid perspectives. Research design (a focus on sound reasoning for claims of identification of causal effects) and the exploitation of theoretical models are not mutually exclusive. In fact, both are necessary and reinforcing. In many cases they amount to the same thing. While there’s room for debate over whether the reasoning and modeling is convincing in one application or another (or even in a whole body of work), I think few applied economists would reject the importance of design or theory.
What is most important for the practitioner and student is the understanding of the role of both. By raising this debate and writing clearly and accessibly about it, Angrist, Pischke, Einav, and Levin all deserve credit. And their papers ought to be widely read and discussed.
Note: Now both papers seem to have been removed from the NBER website. I suspect that site itself is having difficulties. Perhaps they’ll be back up later.
