This post is coauthored with Aletheia Donald
Four years ago, Markus looked at 20 impact evaluations and wrote a post concluding that most of them didn’t have much to say about reducing poverty (where was poverty was defined as expenditure, income, and/or wealth). This summer Shanta Devarajan asked for an update on twitter, so here it is.

We broadened the scope this time, picking 10 impact evaluations each from respected impact evaluation outfits with poverty in their title or in their mission – for a total of 40 impact evaluations. We picked the most recent complete papers on their website in the order they appeared (and none of them come close to 2014, so they’re all new relative to the earlier post).

Let’s start with the paper introductions. Here poverty shows up: 27 of the 40 mention poverty in the introductions. So the motivation does seem to be there.

OK, then, how many of the impact evaluations take place in overwhelmingly poor countries? In 2018, 5 of the 39 evaluations take place in countries with poverty rates above 50% (one country has no recent internationally comparable poverty data). In 2014, this was 3 of 19. So, no better on this metric. It’s a perennial problem – the poorest countries in the world are ones in which it is more difficult to work, particularly since conflict correlates highly with this level of poverty.

Maybe then, these impact evaluations are working with poor populations? Not so clear. Of the 40 impact evaluations we looked at, 9 of them mention the poverty level of the area in which the program takes place and 12 give us poverty statistics for the actual impact evaluation sample and mention them in the text. Accounting for evaluations that do both, we are left with 16 of the 40 where we have some idea of the poverty level of the program participants. Of the impact evaluations in overwhelmingly poor countries, 3 of the 5 tell us that they’re working with the poor.

So, if we add all of this together, in less of half of the evaluations do we know if the results are for poor people or not (keep in mind, that when authors tell us about the poverty status of the beneficiaries, sometimes they are not working with the poor). A potentially interesting tangent: when we took a gander at the affiliation of authors, we found that having a discussion of the poverty level of participants was less common in papers where academics made up a over a third of the authors.

What about impacts on measures of poverty? If we confine ourselves to a money metric notion of poverty (income, expenditures, consumption, assets), only 13 of our 40 include this in the outcomes for which they measure impacts. In 2014, this was 7 of 20. So not great news. There is somewhat better news on the examination of heterogenous effects by poverty levels. Here we find that 8 of the 40 impact evaluations contain this analysis – in 2014, this was only 1.

The sector of intervention most likely to have a discussion of the poverty of the program area or some measure of poverty as an outcome was social protection. Out of the 8 papers that had both, an entire 6 were from social protection. Of course, these interventions, which tilt heavily towards cash transfers (often with accompanying interventions) lend themselves to this. And maybe the push towards direct comparisons (as discussed in last Friday’s post) will help other types of interventions take a harder look at their poverty outcomes, as well as deepen our discussion of which outcomes mean the most for poverty reduction.

This post is the first of two posts on how much impact evaluations tell us about improving life outcomes for the most vulnerable. Post 2 – by David Evans and Fei Yuan – will go up next Monday.

Comments

Thanks Marcus for this relevant post. It would be interesting to see the list of studies that you have reviewed. We have found similar results when we conducted a literature review of impact evaluations of combined agricultural and social protection interventions (http://www.fao.org/3/a-i6589e.pdf table 6 pag 47). Out of 37 studies, in only 5 of them (14%) authors analyzed some measures of monetary poverty. We completed the lit rev by the end of 2016, so it would be interesting to compare how many impact evaluations appear in our and your review. Why authors did not have a closer look at poverty? Did they look at it and did not report the impact estimate because it was not statistically significant or it did not have the expected sign/magnitude? Sometimes we might expect reductions in the poverty gap instead of the head count ratio, but in any case we should try to be as much transparent as possible and at least justify why we did not look at any of these indicators.

Thanks for this Silvio. The overlap will be minimal -- most of ours are from 2017-8, with just a few from 2016.

Often I've heard people shy away from consumption/expenditure because it's noisy -- but I agree, it would be good to know if they considered it. And at the very least, are they working with poor people.