• Reading research: The first question you should ask

    In response to a question about a paper I will not identify, I sent something like the following to Brad Flansbaum. He said it was so helpful he has pinned it to his bulletin board. Perhaps it will be of use to others.

    If you (or any researcher or consumer of research) could learn only one thing from economists, I’d want it to be this: when you see a study that purports to show that X causes Y, think this: “Fine, your paper says X causes Y, but what causes X?”

    If anything that can plausibly cause X also causes Y (Y itself causing X counts) then that’s a big red flag. It’s called endogeneity in general (simultaneity or reverse causality in the case of Y causing X; omitted variable bias if it is something else that isn’t controlled for–in fact these are all manifestations of the same thing, but I won’t belabor it).

    If you can think of something that causes X and Y, the authors had better have controlled for it or dealt with it head on. Don’t just look at the list of independent variables they used. Just use your brain. Lots of things could cause X (and Y) that are unobservable.

    There are techniques to deal with this. Randomzied, controlled trials are the gold standard. But if the study must be (or just is) observational, all is not lost. There are other sources of random assignment or random variation–things that cause X to vary but not Y directly. Natural experiments and instrumental variables approaches exploit these.

    Far too few people recognize this fact and don’t ask the “what causes X” question. That’s why many observational studies that are not credible fool lots of people and get in the news. Fancy statistics can’t address these issues. It requires a very careful analytic design and a lot of explanation to back it up. Many people just get fooled by statistics. It’s a shame.

    Share
    Comments closed
     
    • I agree with this sticky and it is very useful, esp for folks that are not economists. However, economists need to learn to think of other things as well….meaning a renaissance is needed in how we think about endogeneity. Almost everything is endogenous, sometimes devastatingly so. But, sometimes the claim ‘but that is endogenous’ when carried to the absurd degree means that nothing can be studied. I mostly say this from watching poor souls be destroyed in job talks…..when all the asst profs try and impress the full proofs by being the first to say ‘it is endogenous.’

      • @Don Taylor – Good points.

        I think there are two issues. One is what happens at job talks. That’s part of the selection process, both the faculty selecting the candidate (testing him) and the candidate testing the faculty (is this department full of pompous jerks?).

        I take more seriously the reflexive charge of “endogenous!” That’s the second issue. It’s not enough to just say it. One has to make a good argument for why. By the same token, one should be prepared to defend the exogeneity of one’s variables, to a point. At the very lest, one should be aware of the issues and be on guard for them. Too few are.

    • Far too many people confuse correlation with causation.

      Even the Wealth effect, the assumption that a rising stock market causes consumers to spend more fails to account for what is making the market rise.

      Might it be that a good economy, strong job market, pay increases and positive psychology cause all of the above?

    • On Peter’s link

      You will recall a recent IV study on stroke death rates and variable was distance patient lived from hospital.

      You mentioned that this was a commonly used metric.

      Question. Is that would the MR post is referencing, ie, it worked for one, dont assume you can use it in another?

      Brad

      • @Brad F – I haven’t read the paper yet. But based on what Tabarrok said there is no concern about using the same instrument for the same (or similar) variable. It’s using it for a different variable that raises an issue, but only under certain conditions. It’s about the assumptions that are required for a good instrument. However, also judging from the post only, it doesn’t seem irreparable since it is a matter of controlling for the right things, all of which are observable. Thus, it strikes me as a concern only for someone really needing a very deep understanding of the empirics in a specific area. When I read the paper I’ll try to explain, but only if I think I can make it comprehensible.