randomization

A nice new paper by Abhijit Banerjee, Sylvain Chassang, and Erik Snowberg brings theory to how we choose to do evaluations – with some interesting insights into those of us who do them. It’s elegantly written, and full of interesting examples and thought experiments – well worth a read beyond the injustice I will do it here.

This is a curated list of our technical postings, to serve as a one-stop shop for your technical reading. I’ve focused here on our posts on methodological issues in impact evaluation – we also have a whole lot of posts on how to conduct surveys and measure certain concepts that I’ll leave for another time. Updated August 20, 2015.Random Assignment

We are often in a world where we are allowed to randomly assign a treatment to assess its efficacy, but the number of subjects available for the study is small. This could be because the treatment (and its study) is very expensive – often the case in medical experiments – or because the condition we’re trying to treat is rare leaving us with two few subjects or because the units we’re trying to treat are like districts or hospitals, of which there are only so many in the country/region of interest.

Martin’s post on Monday gave me some food for thought. Berk and David’s posts have added to this. For what it’s worth I will throw in my two cents in terms of an addition rather than a direct response.

In his post this week on ethical validity in research, Martin Ravallion writes:“Scaled-up programs almost never use randomized assignment so the RCT has a different assignment mechanism, and this may be contested ethically even when the full program is fine.”

More thought has been given to the validity of the conclusions drawn from development impact evaluations than to the ethical validity of how the evaluations were done. This is not an issue for all evaluations. Sometimes an impact evaluation is built into an existing program such that nothing changes about how the program works. The evaluation takes as given the way the program assigns its benefits. So if the program is deemed to be ethically acceptable then this can be presumed to also hold for the method of evaluation.

So recently one of the government agencies I am working with was telling me that they were getting a lot of pressure from communities who had been randomized out of the first phase of a program. The second phase is indeed coming (when they will get the funding for their phase of the project) but the second round of the survey has been delayed – as was implementation of the first round of the program. But that doesn’t make the pressure any less understandable.