Evaluating after the barn-door has been left open: Evaluating Heifer’s Give-a-Goat or Give-a-Cow Programs

The gift of a dairy goat represents a lasting, meaningful way for you to help a little boy or girl on the other side of the world (US$120); or how about a flock of ducks (US$20); or go all out and donate a water buffalo ($250). People need "a cow, not a cup"—cows that could produce milk so families would not have to depend on temporary aid.

International animal donation programs of the type mentioned here are one of the most well-known types of charitable requests, and are used by a number of major charities worldwide. The best known exponent of this program is Heifer International, from which the examples above are taken. The purported goals of such programs are to improve the nutritional outcomes of participating households and provide a pathway out of poverty.

To my knowledge, none of these programs have been subject to a randomized evaluation, nor to a prospective non-experimental evaluation. So one question is what we can learn from the programs that have already occurred? A new paper by Svetlana Pimkina, Rosemary Rawlins, Chris Barrett, Sarah Pedersen and Bruce Wydick tries to do this, by conducting an ex-post non-experimental evaluation of two of Heifer’s animal donation programs in Rwanda, one that donates dairy cows, and one that donates meat goats. The details are as follows:

Interventions: In both cases recipients presumably get training in livestock care, although this is not described in the paper. Then there are two interventions, which take place in separate regions.

Cow Treatment: beneficiaries are given an imported, pregnant cow which produces a lot more milk than local cow breeds. Recipients are required to give away the first female offspring, but can sell male offspring. Ongoing insemination services are provided so the dairy cows can continue to produce milk and offspring. Approximate value is US$3000. This is given in northern Rwanda in Ruli.

Goat Treatment – beneficiaries are given two female goats. A few beneficiaries in each area are also given a male goat to breed with the area’s female goats. Heifer expects households to breed the goats for sale and/or consumption. The paper doesn’t give the cost. This is given in eastern Rwanda, in Kirehe.

How are recipients selected? Households apply to receive an animal. To get a cow, households must not already have a high producing dairy cow, must have at least one hectare of land, and must commit to constructing a shed with a cement floor and metal roof (through a Heifer loan) if selected. But beyond this “Heifer International was unable to provide a clear set of secondary selection criteria… beneficiaries are selected based on need as determined by a team of assessors…. in some cases the first round of donations were distributed to highly prioritized families, while in other cases the decision was made based on logistics.”

Evaluation methodology: The researchers conducted a cross-sectional survey in the summer of 2011. They surveyed 406 households, consisting of 224 that had applied for the dairy cow program, and 182 for the meat goat program. Sample sizes for anthropometrics are even smaller: a total of 99 children in the cows region and 129 in the goats region. The paper is silent on what the attrition levels were relative to the target list of all such households. Surveys collected information on household dietary diversity and on child anthropometrics. They surveyed three groups of households: those who had received the program one year earlier, those who had been approved to receive an animal sometime in the near future (“prospective households”), and those that had applied, but were not selected by Heifer. The paper contains no table comparing means across these three groups, so it is hard to see how similar they appear on time-invariant observables.

Estimation is then by simple linear regression and tobit estimation, controlling for region, acceptance into the program, household size, and a few household asset holdings (e.g. phones, machetes, rabbits), and the age and education of the respondent. Some estimation is also by propensity-score matching, where the propensity score is based by matching on only 5 variables given the small sample size.

Estimated Impacts:• Households receiving a cow consumed 9.3 more liters of milk per month than members of prospective households (p<0.01); children aged 5 and under in these households had a 0.57 standard deviation increase in height-for-age (p<0.10); and no significant change in weight-for-age. • Households receiving a goat had no increase in dietary diversity; had a marginally significant increase in meat consumption (of 0.20 kg/person/month) in some specifications; a 0.47 reduction in wasting measures for children under 5; and no change in stunting measures.• The authors use Rosenbaum bounds to see how severe the self-selection on unobservables would have to be to explain the results: for the impact of a dairy cow on milk consumption, there would need to be unobserved characteristics perfectly correlated with dairy consumption that make the odds of self-selection 5.5 times higher for those who were treated, which the authors consider unlikely – whereas even mild self-selection would kill the results on goats.Thoughts on the evaluationThe paper is a valiant effort to provide some first evidence on these types of programs. But it also highlights the difficulties of having to come in after the fact and do an ex-post evaluation – it is unclear how people were selected, there is no pre-program available on them to enable difference-in-differences or better matching, etc. The authors themselves conclude “The promise evident in animal donation programs is apparent, even if the impacts remain difficult to estimate precisely in small, observational studies such as ours.

A follow-up study using a randomized control trial design with long-term tracking would enrich our understanding…”. So my hope is that such an evaluation does not serve to enable organizations like Heifer to claim they now have rigorous evidence to support their programs and hence that they see no need for prospective evaluations. Moreover, the paper focuses on short-run outcomes in a very narrow domain. To my mind the three most interesting evaluation questions in an evaluation of these types of animal donation programs would be:

Do these programs provide a lasting pathway out of poverty, or do most of the animals end up getting eaten or dying, and households converge back to their former status?

Do the programs provide better long-term impacts than simply giving cash? A horse-race, or perhaps a cow or goat race between an animal grant and pure cash grant would be welcome.

My sense is that the main advantage of a program like Heifer’s is probably not that it delivers much better outcomes for the recipients than would cash, but rather that it raises a lot more money from people who might not otherwise donate to the poor. So even if Heifer has only one half the impact of a pure cash grant per dollar granted, it may have a much larger impact on the poor if people are many times more likely to give when they have the gimmick of an animal they think they are buying. Testing this empirically would be very interesting.

Comments

Dear David,
Thank you for an interesting summary of the Heifer’s Give-a-Goat or Give-a-Cow Program study.
Regarding your "thoughts on evaluation" above, I wanted to mention that the CGAP-Ford Foundation Graduation Program is currently pilot-testing an interdisciplinary approach for the poorest, which includes an asset transfer alongside other interventions such as consumption support, savings and coaching, adapting a methodology developed by BRAC in Bangladesh.
We are conducting randomized control trial impact assessments in eight out of ten pilot sites, and qualitative research in nine sites. In Ghana, a study by Innovation for Poverty Action (IPA) is designed specifically to “tease out” the relative impact of the asset transfer (and the savings component) versus the wholesale intervention.
An overview of the research design in Ghana is available on IPA’s website (http://www.poverty-action.org/project/0494), and we will be sharing the results from this work on the Graduation Community of Practice’s Website here: http://graduation.cgap.org/research/.

Very interesting post. For those exploring these issues you might include in your hypotheses and theory of change the effect of these gifts on social obligations of the recipients to relatives and visitors.
For those interested in the topic for gifts of chicks and chicken, I conducted a participatory epidemiological assessment of such transfers to women.
The first issue is the potential introduction of disease.
My second observation is the hypothesis we had in this project that the eggs would be used to feed children. This seemed particularly silly to women beneficiaries who could not believe anyone would waste an egg that would become a high priced chicken to feed a child when you could buy the egg for little money.
Finally, women had a strategy of chicken leading to guinea fowls to goats to cows. So if you are evaluating this type of livestock support you should check the upgrading strategy of beneficiaries. In our project, the beneficiaries had in mind an optimal stock of chicken (also managing disease issues), then above that they would upgrade.
Best,
Charles Lor
Regional Results Measurement Specialist
IFC

David, thank you for the very apt comments about the Heifer study. In doing this research, propensity score matching was literally our third or fourth choice for impact evaluation. Our original plan was to work with the organization to carry out an RCT, but logistics in the field prevented any kind of controlled approach. We then searched for some kind of IV or a rule where we could use a regression discontinuity, but obviously one needs (reasonably) firm allocation rules, and even after considerable effort, there just wasn't a clear rule we could find to get identification.
This is one type of program that simply yearns to be evaluated with a nice, moderately sized RCT, and someone should take up the task because a lot of money is given to animal donation by everyday people who would be very interested in its impact.
The main issue, however, is that the program isn't about just giving people animals; it's about the support staff that households need to manage a new animal, ag extension, insemination services, etc. So one can't just randomly allocate animals to households alone to evaluate animal donation programs. It's necessary to have the whole array of services that come with them, and for this you need to work closely with an NGO that is in this business. But we found it difficult to carry out randomization working with Heifer simply because it is hard in this particular case to build randomization into something resembling their normal operations, and too far a departure from their normal approach would render the study invalid as an evaluation of Heifer's program per se.