Evaluation that matters

We like evidence-based policy. Programme evaluation should be embedded into policy design, and policies that don’t work should be abandoned.

But what happens when people like programs that don’t work? Here’s Real Clear Politics on Head Start.

The perfect case study is Head Start, the oldest federal preschool program. The Head Start Impact Study — a state-of-the-art, multi-site experimental evaluation set in motion by a law Bill Clinton signed in 1998 — came with a price tag of $28 million. Rationally, lawmakers should not have paid for that study unless they expected the results to affect their support for the program. If the Impact Study shows Head Start is effective, they should want to increase funding and look for ways to expand the program’s reach. If Head Start is not proven effective, lawmakers should presumably want to eliminate the program, or at least decrease support and redirect some of the funding toward back-to-the-drawing-board research.

Rationality did not prevail. The Impact Study failed to show lasting effects, yet Head Start is still alive and well. In fact, a couple of months after the study’s final results were released, the Obama administration proposedincreasing funding for Head Start, touting the “success” of the program and the “historic investments” the administration had already made in it. The White House did not say what it meant by “success,” but clearly it must have been judging Head Start on some criteria that the Impact Study did not cover. So why pay for the study in the first place?

Head Start’s defenders argue that the Impact Study is not capturing “sleeper effects” that will emerge later in the participants’ lives. So if the Impact Study had shown positive effects, they would have said, “We should support Head Start because of these positive effects.” Instead, they say, “We should support Head Start because of sleeper effects suggested by other research.” Since the decision is the same either way, the Impact Study was a waste of taxpayer money.

They conclude that there was bipartisan consensus both for programme evaluation and for ignoring its results. A solution?

Dropping the studies altogether would be preferable to paying for them and then ignoring the results. The better solution, however, would be to legally tie program funding to the evaluations. Make the existence of Head Start and other programs contingent on showing impacts on pre-specified outcome measures. That would require lawmakers to be clear about the reasons they support or oppose particular programs. If they protest that the benefits of their favorite program are not necessarily captured by a formal study, the natural question would be, “Since the study has no chance of changing your mind, why do you want taxpayers to fund it?”

There would be logistical difficulties, of course. One can imagine the special pleading that would follow a poor evaluation: “My favorite program almost achieved its required impact, so we shouldn’t penalize it.” A stubborn Congress might pass new legislation that simply restores funding to pre-evaluation levels. But the purpose of tying dollars to results is not so much to force an immediate policy change as it is to generate a more serious discussion about what we expect from social programs. It’s a discussion that is long overdue.

Endorsed, or at least mostly. There’s some chance that some future, better regime will look back through the assembled cost-benefit assessments showing failures and kill all the failed programmes. It’s a slim hope, but there’s better chance of it where data at least exists that the programme was not cost-effective.