If you’ve been reading my blog for a while, you know I have a special love for Nate Silver’s blog. Especially when he pulls out the statistical analyses. He did just that today:
A study purporting to find a connection between stimulus spending and the partisanship of a district suffers from an obvious flaw. But in so doing, it provides an example of why it’s important to retain some common sense — and some sense of context — when conducting a statistical analysis.
The study, by Veronique de Rugy of George Mason University and the National Review, claims that congressional districts which elected a Democrat to the Congress received a larger amount of stimulus finds by a margin which is statistically significant even after controlling for certain other effects like the unemployment rate. However, the study does not control for at least one other variable that is overwhelmingly important in determining the dispensation of stimulus funds.
The variable in question is in fact pretty obvious if you simply look at the districts that have received the largest amount of stimulus money, according to de Rugy’s dataset…
In fact, the differences are pretty overwhelming. There are 78 congressional districts that contain all, or part, of a state’s capital city. Collectively, they received $118 billion in the fourth quarter. The 357 districts that are not home to a state capital received only $48 billion, however. On a per-district basis, the districts with state capitals received 11 times more funding. The ratio would be higher still if we excluded districts that included only outlying areas of state capital cities that do not host any state governmental institutions.
The other piece of the puzzle, of course, is that state capitals are much more likely to elect Democrats to Congress for a variety of reasons. They are, by definition, urban (although some smaller state capitals like Montpelier stretch the definition). They are, by definition, home to large numbers of governmental employees, who may be more sympathetic to bigger government. They tend to be highly educated and often are home to large state universities.
That de Rugy has testified before Congress on the basis of her evidence, and never paused to consider why the top five congressional districts on her list overlap with Sacramento, Albany, Austin, Tallahassee and Harrisburg, is mind-boggling. The presence of a state capital is the overwhelmingly dominant factor it predicting the dispensation of stimulus funds. This could have been discerned in literally five minutes if she had bothered to look at the apparent outliers in her dataset and considered whether they had anything in common — a practice that should be among the first things that any researcher does when evaluating any dataset.
You really should go read the whole thing.
I want to bring this up not because I wish to discuss the stimulus, or even politics, but to point out why peer-review is so important. Before our research goes public, it’s important for independent people to review it. The reason is so that fresh eyes (like Nate’s) can look at what we’ve done and make sure we didn’t forget something (like state capitals).
When I disparage work from think tanks, it’s not because I believe they can’t do unbiased work. It’s because in those environments, everyone things along similar lines. And unless they allow independent people to review their work, they may miss something critical. That’s one of the reasons we need peer-review. It’s one of the reasons I keep harping on the subject.
