Final external Evaluation of the emergency cash/voucher program in Southern Somalia

In response to the famine in Southern and Central Somalia, several agencies have been implementing an unconditional cash transfer and food/water voucher intervention from August 2011 until end of March 2012, a second round from April to October 2012. This initiative represents a comprehensive effort to address the alarming food insecurity of the most vulnerable households, for whom high prices form the main and overriding barrier to access food.

The overall figure of USD XXXX million will be disbursed over a twelvemonths timeframe. The average grant amount or voucher for each household has been of USD 100 (from $60 to $120). The payments, when the cash grant modality was used, have been delivered through the local Hawala (traditional money merchants).

Cash and food/water vouchers distributions have been implemented in all regions of central and south Somalia by the following INGOs who will also be part of this evaluation: the Cash Consortium composed of Save the Children ( xxxxxxx of HH), ACF (xxxxxxx of HH), DRC (xxxxxxx of HH) and Horn Relief (xxxxxxx of HH); Oxfam Novib (xxxxxxx of HH), Oxfam GB (xxxxxxx of HH), NRC (xxxxxxx of HH), Concern (xxxxxxx of HH), Coopi (xxxxxxx of HH). Among these NGOs, 8, implementing unconditional and food/water vouchers programs, have adopted a common monitoring approach. In this regard, a team of independent monitors have been contracted through the Overseas Development Institute (ODI) to develop the overall M&E plan and its specific tools, as well as to provide the technical leadership for all the project-related M&E activities. The on-going monitoring has covered both the process aspects and the impact on local market prices. As part of this, some independent field monitors have been recruited for the collection of qualitative data, the quantitative data are collected directly by the NGOs in the field.

2. Purpose of the evaluation

The objective of the evaluation is to assess the efficiency, effectiveness, accountability and impact of the emergency cash transfer/vouchers intervention in Southern Somalia, including its monitoring system.

3. Methodology

Design of the evaluation

Only qualitative data will be collected by the evaluation team during focus group and interviews. The, already collected, ODI quantitative data will be available to the evaluation team.

All regions, where activities have been implemented will be targeted by this evaluation, as well as all partners. Due to security reason one or more regions might not be accessible at the time of the evaluation. The sampling per region will be developed by the evaluation team in collaboration with UNICEF.

The evaluation subjects will be vulnerable households, with a specific attention to under-five children.

Each implementing partner will be evaluated individually; in addition a collective impact analysis will be done by the evaluation team.

The evaluation team is expected to go the field to meet with field teams, hawalas and if feasible conduct focus group with beneficiaries, local elders… if and where security conditions will allow it. In this regard, the evaluation team is expected to take the households sample from the overall set of targeted households and not only those that are easily accessible.

2. Learning objectives of the evaluation

1. Assessing the efficiency of the cash/vouchers distribution system

Assessing to which extent the delivery mode (with a specific focus on the hawala arrangements) and timeliness of implementation of the cash/food voucher transfer project responded to the beneficiaries’ needs;

Assessing the quality of the distribution process (including security/protection issues for the beneficiaries);

Assessing the cost-effectiveness of the cash/vouchers transfer as opposed to in-kind distribution, including direct and indirect costs, and the cost per household

Identifying aspects of the program that would make it more efficient.

Assessing the human right-sensitive aspect of the project in its design and implementation (looking at the way distribution were made, waiting time, travel time to reach the distribution/hawala office site…)

Assessing to which extend the project ensured ethnic minorities/minority clans were included?

Assessing whether the program was gender-sensitive in its design and implementation approach? Was the implementation of the project conducive to women empowerment?

2. Assessing the effectiveness of the cash/vouchers assistance as an emergency intervention at scale

Assessing the effectiveness of beneficiary targeting and the degree of inclusion of vulnerable under-five children; and establishing how well cash/vouchers distribution correlated with vulnerability assessments in a qualitative manner

Comparatively assessing the quality of the diet for both children and household members in households receiving cash and households receiving food vouchers; and impact on reduction of morbidity and mortality of children under five.

Analyzing the experience of the beneficiaries, including their preferences on modes of assistance delivery (i.e. would they prefer to receive cash, vouchers or food, in kind commodities and why?);

Assessing real coverage of the programme versus planned coverage

Analyzing the difference between the cash grants and the vouchers methodologies based on livelihood areas (rural versus urban, agriculture versus pastoralist).

Assessing the appropriateness of the size of the cash grant – purchasing power and whether more could have been done to alter the size of the transfer to match the cost of the food basket.

Drawing a comparison in terms of impact between the cash transfers covering the full MEB and the vouchers /cash grants covering only partially the MEB for either food or water

Assessing market behaviors during project implementation, and the impact the cash/vouchers distribution has had on prices and market supply

Assessing also how the innovative aspect of this program has challenged the regular mode of operation of the implementing partners and involved agencies in emergencies (what were the administrative, managerial, logistical challenges or implementing a large, coordinated voucher/cash program for each of the various agencies and partners, and what can be learned for the future)

Identifying aspects of the program that would make it more efficient

Assessing the resilience of the programme to changes in the versatile and complex environment such as Somalia, and how it affected different response modalities

Assessing to which extend the lack of access and insecurity has influenced the decision to implement cash/vouchers distribution in South of Somalia

3. Assessing the accountability (to beneficiaries and donors) of the emergency unconditional cash/vouchers program

Assessing the effectiveness of the community feedback mechanisms in place and the capacity of implementing partners to respond to comments received (i.e. whether the project is accountable to beneficiaries);

Assessing the level of beneficiary participation in the targeting process (i.e. whether implementing partners are accountable to themselves and to the beneficiaries).

Assessing the upwards accountability to donors and the impact of, and capacity to delivery on, donor reporting requirements

4. Assessing the impact of the cash/vouchers distribution

Assessing the impact of the cash transfer/vouchers distribution on households ability to access food

Assessing the impact of the cash transfer/vouchers distribution on the diet of the household

Assessing the impact of the cash transfer/vouchers distribution on the households decision-making when it comes to return

Assessing the impact of the cash transfer/vouchers distribution on local economy (local traders and markets)

Assessing any other impacts identified during the evaluation

5. Assessing the efficiency of the coordination mechanisms of the emergency unconditional and cash/vouchers program

Analyzing the effectiveness of the coordination of the cash/voucher stakeholders in the planning, implementation and monitoring components of the program.

Recommending circumstances how and when cash interventions needs coordination and recommending potential future options for coordinating access to food as an objective

Analyzing the appropriateness of the staffing structures put in place given the program needs

Analyzing the challenges created by this new program to the cluster system. ( This, from a programmatic angle, would need to take into account the merge of the food security and livelihood clusters ).

Considering the project’s coherence with the humanitarian policy and the effectiveness of coordination done with other humanitarian players; i.e. success or otherwise of interaction between the food cluster, livelihood cluster and CBRWG given their but overlapping mandates?

Assessing the role played by the CBRWG (Cash Based Response Working Group) and CALP

6. Assessing the joint monitoring system

Assessing the set-up of the joint monitoring system, including the mechanisms put in place for giving support on Digital Pen Technology, and the challenges encountered in terms of institutional and contractual arrangements, timeline etc…

Assessing the sampling process and the methodology used

Assessing the quality of the tools developed and their relevance and evolution over the course of the program

Assessing the mechanisms of coordination within the monitoring system

Identifying aspects of the monitoring system that could be improved and providing recommendations for the future

3. Existing information sources for the evaluation

The evaluation team will make full use of the following documents:

All monitoring reports produced by ODI, including market monitoring, monthly and quarterly post-distribution monitoring (PDM) individual and joint reports and the consolidated final M&E report, incorporating the feedback from each INGO partner and published by ODI Humanitarian Policy Group. Data collected are both of quantitative and qualitative nature.

Additional M&E individual final reports prepared by each INGO partner, containing the M&E data analysis and the main lessons learned.

Database of Market Prices collected weekly (the first two months) FSNAU, FEWSNET surveys, and information from FAO.

Project proposals developed by each INGO and organizations’ concept notes

Focus groups with key stakeholders on the perceived impact of the cash transfer project on the local economy (functioning of markets, comparison with situations when food is delivered instead of cash, FSNAU, FEWSNET reports etc.);

Focus group with beneficiaries of the efficiency, effectiveness and accountability aspects

Lessons learnt from other countries

4. Deliverables

Evaluation Inception report outlining a more detailed approach to the evaluation and timeline – following an initial phase of desk reviews and preliminary consultations

Interim/progress report on objectives 5 and 6, and preliminary observations on the others objectives

Final evaluation report (i) answering the main questions of the evaluation, (ii) outlining main lessons learned during the planning, implementation and monitoring phases of the project and (iii) proposing recommendations and future directions.

5. Timeframe of the evaluation

A two-phase’ evaluation:

1. First phase in May 2012 to look at objectives 5 and 6 -Some preliminary observations and recommendations on other objectives are expected to be mentioned in the interim report.

2. Second phase in November 2012 to look at objectives 1 to 4, and 6

Activity

Deadline

Deadline for application

2th of March 2012

Start of the evaluation

1st of May 2012

Final evaluation paper finalized

December 2012

6. Qualifications of the evaluators

An emergency cash/voucher expert (team leader)

At least a master’s degree in planning, monitoring and evaluation, economics or social sciences

At least 8 years of documented experience in evaluation of emergency programmes, including cash transfer projects, preferably with a link to nutrition;

Extensive experience as evaluation team leader of emergency projects

Knowledge of cash transfer and nutrition programmes in crisis context;

Knowledge of cost benefits analysis for cash transfer programmes;

Extensive experience in working with, INGOs and local NGOs, UN, local authorities, beneficiaries

Demonstrated analytical, writing skills;

Excellent knowledge of English (knowledge of Somali an asset).

A livelihood expert:

At least a master’s degree in economics with a good knowledge in nutritional and food security aspects; or in social science/anthropology

At least 8 years of documented experience in evaluation of emergency programmes, including cash transfer projects, preferably with a link to nutrition;

Knowledge of cash transfer and nutrition programmes in crisis context;

Extensive experience in working with, INGOs and local NGOs, UN, local authorities, beneficiaries;

Demonstrated analytical, writing skills;

Excellent knowledge of English (knowledge of Somali an asset).

A humanitarian reform specialist

At least a master’s degree in planning, monitoring and evaluation, economics or social sciences

At least 8 years of documented experience in evaluation of emergency programmes;

Excellent knowledge of the humanitarian system and recent reforms, including the cluster system

Extensive experience in working with governments, INGOs and local NGOs, UN, local authorities, beneficiaries