In the early days of the HIV epidemic, I was invited by the US House of Representatives Select Committee on Children, Youth, and Families* to give expert testimony on AIDS risks to adolescents. I was young and thrilled that I might have something to contribute on a critical topic.
But the event was a parody of a truth-finding inquiry. I had been invited by the Democrats, then in the majority, and served as the first witness. I read a statement about the epidemiology of HIV infection among adolescents and about behavioral factors that would put teens at risk. Frank Wolf (R-VA) interrupted and asked me how many children I had. “Two”, I answered, nonplussed. He then told a lengthy story about how he was raising his children to be sexually abstinent. A Democrat from California took exception — I kid you not — to my suggestion that health education classes should stress to teens how alcohol can impair sexual decision making. This Representative’s district included the Napa Valley; he was literally a chardonnay Democrat. The Republican witness who followed me argued that condoms were deadly health risks for adolescents. Pat Schroeder (D-CO) was drinking water, laughed, and nearly choked. She proceeded to mock the witness with a series of contemptuous questions. I left thinking that if this is how the Congress collects scientific evidence, there wasn’t much hope for the Republic.
Almost no one thinks that US government policies are well-grounded in science. According to John Bridgeland (former director of the White House Domestic Policy Council) and Peter Orszag (former director of the Office of Management and Budget and the Congressional Budget Office),
Based on our rough calculations, less than $1 out of every $100 of government spending is backed by even the most basic evidence that the money is being spent wisely.
So why is it hard to get evidence-based social policy? It’s not that people don’t try to bring science to the legislature’s attention (although here too, we can do better). On any contentious issue, there will be white papers and credentialed experts on both sides. However, for many reasons the legislative process fails to work through those data so as to arrive at an evidence-based result.
- There may not be enough data to decide the issue. A health policy or a social program may only have been studied in a limited experiment that does not map well to the new setting and larger scale represented by the proposed legislation.
- The process of passing a law is long, complex, and there is no point where evidence is reviewed in a disciplined and independent way.
- Members of Congress are — of course — partisans. They have convictions that lead them to process evidence through motivated reasoning. Their prior beliefs are impervious to evidence because they find reasons to discount inconsistent data.
This seems hopeless. But a new book by Ron Haskins expresses cautious optimism that we can do better. Haskins — a Republican who was a domestic policy advisor in the G. W. Bush administration — has been impressed by the Obama administration’s attempts to change the process so as to generate more evidence-based policies.
The book presents case studies on six programs (e.g., a bill to fund teenage pregnancy prevention). The cases demonstrate how challenging it is to make policy empirically-driven. At the end, we have a picture of how legislation and policy can become more empirically-informed.
- Legislation should do less to specify the social intervention and more to specify a) the outcomes that the intervention is supposed to achieve and b) the procedures that will be used to select and evaluate candidate interventions.
- The outcomes that should be considered in selecting and evaluating programs should be substantive social and individual benefits (e.g., reduced teen pregnancy). They should not be program or process outcomes (e.g., the number of teens who received sex education).
- Programs should be selected based on good quality evidence: the results of randomized clinical trials (RCTs) or strong quasi-experimental designs. Moreover, there should be procedures for gathering and synthesizing evidence to score solutions. These procedures need to be specified before the evidence is gathered, so that program operators cannot pick the scoring procedure to rationalize the choice of their desired solutions.
- Programs should be implemented so that they collect data on real outcomes. Even more importantly, the implementations should include studies with strong designs (e.g., RCTs) that generate good evidence about program effectiveness.
These ideas may seem commonsensical, but Haskins’ case histories demonstrate how hard it is to pass policy legislation involving these features. Getting legislation passed requires strong advocates, but most advocates already know what works, thank you very much!
Moreover, getting the legislation passed is only step one. Evidence-based policy will only have succeeded if the government can both carry out the planned evaluation studies and act on the feedback from those evaluations. Experience suggests that many — most? — of those evaluations will show that the social program has failed. Will program administrators and legislators have the courage and political power to modify or cancel those programs?
So the movement for evidence-based policy could fail. But if not, we could be seeing the beginning of a much deeper evidence-based culture in legislation and policy. That could be the beginning of a much better world.
*The Select Committee no longer exists. This says something about the importance of children in US government policy.
