Are “randomised evaluations” a better way of doing aid and development policy?

DOCTORS study diseases from several vantage points. Laboratory scientists peer into microscopes to observe the behaviour of bugs. Epidemiologists track sickness in populations. Drug-company researchers run clinical trials. Economists have traditionally had a smaller toolkit. When studying growth, they put individual countries under the microscope or conduct cross-country macroeconomic studies (a bit like epidemiology). But they had nothing like drug trials. Economic data were based on observation and modelling, not controlled experiment.

That is changing. A tribe of economists, most from Harvard University and the Massachusetts Institute of Technology (MIT), have begun to champion the latest thing in development economics: “randomised evaluations” in which different policies—to boost school attendance, say—are tested by randomly assigning them to different groups. In one celebrated example, researchers looked at what happened in 20 antenatal clinics in western Kenya when some gave away insecticide-treated bednets, an anti-malaria therapy, and others sold them for different prices. Their conclusion was that free distribution is far more effective in getting people to use bednets than charging even a nominal sum would be.

Such trials are not unprecedented in economics. America's welfare reform of 1996 was based partly on controlled experiments. But they have been rare enough for today's upsurge to count as a revolution in thinking about development. Last year the Spanish government gave the World Bank €10m ($16m)—the institution's largest trust fund—to spend on evaluating projects. The fund's first criterion calls for randomised trials. This will spread their influence further.

But are such trials all they are cracked up to be? Randomistas recently gathered at the Brookings Institution, a think-tank in Washington, DC, to discuss that.*

Randomised evaluations are a good way to answer microeconomic questions such as how to get girls to go to school, and teachers to turn up for work. They cannot tell you much about macro questions like the right exchange-rate or budget policy. But often, they provide information that could be got in no other way. To take bednets: supporters of distributing free benefits say that only this approach can spread the use of nets quickly enough to eradicate malaria. Supporters of charging retort that cost-sharing is necessary to establish a reliable system of supply and because people value what they pay for. Both ideas sound plausible and there was no way of telling in advance who was right. But the trial clearly showed how people behave.

So evidence from randomised trials is good. But is it better than other economic data? That is what many randomistas believe. Abhijit Banerjee, the co-founder of the Abdul Latif Jameel Poverty Action Laboratory (J-PAL), argues that “the quality of the evidence that informs much of the macro-growth debates is significantly worse than the quality of the data that bears on many of the micro-policy questions”. He adds: “The beauty of randomised evaluations is that the results are what they are.” In other words, they provide hard evidence, resting on a solid empirical base. Aid and development policy, concludes Mr Banerjee, should take more account of that evidence.

But is the evidence really incontrovertible? On its own terms, yes. As Mr Banerjee says, the evidence is what it is. But policymakers do not want to know whether something works in a few villages. They want to know whether it will work nationwide. Here, randomised trials may not be quite so helpful.

Go back to the bednets once more. You might conclude that the trial showed that they should always be given away. Yet it turns out that millions of nets were already in use in the part of Kenya where the field trial took place, so their value was known. The experiment guaranteed supplies, so it did not test the assertion that you need to charge something to encourage reliable suppliers. And the recipients were pregnant women, whereas the point of giving bednets away is to provide anti-malaria treatment universally. The evidence from western Kenya was clear. But it hardly settled the question of whether the government should give bednets away across the country. Questions like that may still have to be made on the basis of the soft evidence that randomistas turn up their noses at.

Randomistas rule?

Mr Banerjee doubts whether randomistas and other development economists will ever get along. The differences over research methods and what counts as evidence are too great. Economists do not know enough about growth, he says, to justify their obsession with it, however important it may be. Following the law of comparative advantage, they should do much more of what can be done best—randomised testing.

But given doubts about how widely applicable such tests are, it may be better to think of them not as a new, superior form of development economics but as one more technique—admittedly a useful one—for finding out what works, filling in gaps in knowledge, testing policy ideas, and puncturing conventional wisdom. Dani Rodrik of Harvard University argues that differences in research methods between randomistas and other economists are in danger of re-opening a split between macro- and micro-economists that is starting to heal. Over the past few years, he claims, both groups have converged on a more experimental approach to development, eschewing lists of standard prescriptions and stressing the importance of context. That approach may be bearing fruit. It would be a shame if triumphalist claims by randomistas were to limit their contribution to it.