Bowling in the dark: Monitoring and evaluation during COVID-19 (Coronavirus)

As the implications of the Covid-19 pandemic are felt around the world, few will escape its economic repercussions. Competition for financial resources will be fierce, not only between developing economies but between developing and industrial economies. Bolstering the ability of governments to maintain domestic support for global efforts to respond to the needs of the most vulnerable in the midst of this crisis will be critical.

Institutions like the World Bank Group (WBG), as they seek to respond to COVID-19 impacts, as well as development needs more broadly, will need to assure taxpayers and voters in donor countries that increasingly scarce resources are being well spent. This will require creativity in adapting monitoring and evaluation (M&E) systems to match both the speed and forms of the World Bank Group’s crisis response and its continuing pursuit of its development mandate.

Applying Lessons from M&E in Insecure Areas to COVID-19

Click to see a larger version.

World Bank teams with experience working in “no-go” areas with high risks to personal security have much to teach about M&E in the face of COVID-19.

In situations of conflict, one of the ways in which M&E (and data collection in particular) has been carried out has been by drawing on Third-Party Monitoring. However as discussed by Lauren Kelly and Marie Gaarder in a previous IEG blog, there are ethical considerations associated with sending people into harm’s way. These have been exacerbated by COVID, and must be taken into account in M&E efforts. This is because there is significant risk of third-party monitors both spreading and contracting the disease, including between densely populated capitals and more remote, yet unaffected areas. These risks are further compounded by inadequate supplies of protective equipment and a lack of access to COVID-19 testing in many of the countries in which the World Bank operates.

To address these risks, Elias Sagmeister of the Global Public Policy Institute provides a useful toolkit of tech-enabled handheld devices for digital data collection, including mobile phone-based feedback mechanisms, remote sensing with satellites or delivery tracking, communication with online platforms and broadcasting with radios and other forms of (social) media. But technology has its limitations.

First, technology tools need to be designed in line with data collection requirements. Many tech enabled tools are best used to verify assets; they are less equipped to measure quality and implementation fidelity.

Second, they work best as “a bundle”: earth observation technologies paired with phone-based mechanisms can help verify that assets were delivered and to where, as well as to whom, they were intended to be delivered.

Third, one needs to be mindful of exclusionary factors surrounding the use of technology. In insecure areas, women often have less access to cell phones than men and this can bias reported outcomes, as was shown in IEG’s radio and mobile-based outreach in the Afghanistan Country Program Evaluation.

Fourth, data collected by tech enabled tools are unlikely to be able to capture the granularity needed to ascertain and address critical environmental, social and conflict related risks, that are more adequately identified through consultations, rigorous supervision, and implementation of citizen engagement processes.

Lessons Also Can be Gleaned from the Ebola Experience

If there is one thing we have learned from Ebola, it is that communication matters. In the times of COVID-19, like with Ebola, there will be heightened fear, and confusion, and distrust of governments and donors who are “trying to do the right thing” by conveying hard messages about health and behavior to isolated, disbelieving communities.

M&E has a critical role to play during a pandemic in assessing the continued appropriateness of World Bank activities given the enhanced need to protect personal safety. Do current projects require citizens to participate in-person, in unsafe ways to achieve intended benefits? Are projects helping clients to adapt these systems – education, health services, labor, payments - to maintain essential services and income - while protecting citizen safety? And how will we know?

Countries providing the essential resources for the pandemic response will want to know how funds released through a project’s “Contingency Emergency Response Components” are being spent, but project M&E frameworks are often not equipped to do this. Restructured, adaptive M&E and safe data gathering methods need to be developed alongside the release of such funds, in real time.

This will be particularly important as the global community commits vast sums to support developing countries in responding to COVID-19, including the recently approved $14 billion package of fast-track financing initiative from the World Bank Group to assist companies and countries in their efforts to prevent, detect and respond to the rapid spread of COVID-19. The challenge is to ensure that M&E is not forgotten in the well intentioned rush to respond to urgent needs.

Do you have ideas, lessons, or insights to contribute to this conversation about evaluation during COVID-19? Please share them in the comments below.

Multimedia

Comments

Thanks Lauren and Jeff, I think a key message is to be proactive about just collecting any data when responding to emergencies. In the rush to deliver something, we forget to monitor and reflect as we go. Answering questions at the end becomes impossible without knowing why decisions were made at the time. Using and capturing data at various decision points along the way helps us to know why we did what we did at the time. The United States military have become good at this through their emergent learning approaches (including before-action-reviews and after-action-reviews). Formalizing these simplistic steps and capturing the interactions occurring in emergency situations could be useful to a better response to future pandemics.

Thank you so much for great insights about carrying out M&E work at the time of crisis, when human intervention may be risk. I have some experience on participatory performance monitoring, where we identify and train some selected community based men, women, adolescents or any change agents. These individuals provide data and information on the a range of issues and technical experts sitting in office, use it as per the programme indicators.

Yes, this seems difficult but lot of energy we require to invest to make the selected individual in community trained and the questions are designed in such a way that they feel comfortable responding. As far as platform is concerned, Mobile phone calls, whatsApp messages, audio / video call works well in this situation.

If you compare it with the in person visit to community and collect data and information, it is not comparable however, both model has their own Pros and cons.
Considering the time like COVID-19, if you have already prepared your community on this model, it will work and for sustainability point of view also, it has huge scope.

I would like more to learn on this aspects if somebody has worked on this.

Well mention: "The challenge is to ensure that M&E is not forgotten in the well intentioned rush to respond to urgent needs." Personally, I have been watching the data generation, data analysis, data presentation and data open in regard of COVID19 response of US, India and Nepal (My country). I found US is working excellently in all aspect of data. But both country India and Nepal are just making the number counting also lack access of detail data for public or for interested M&E practitioner. I have started to compile Nepal data that generates from official channel as well as from different public media such as online news portal and shared to M&E professional network . So independent M&E professional/researcher would not stuck with no insightful data for now and future. Here's you can have a look https://drive.google.com/open?id=19rBi4mxvm9hN6uOnSQxC37hYvsm8V_q7

Thank you for the text!
It is very accurate!
One key point is to provide public agents with precise data what is imposible without an accurate and standartizes number of tests per milliion of inhabitants. So I believe this is a way World Bank should support.

Connect With IEG

The Independent Evaluation Group evaluates the work of the World Bank Group to find what works, what doesn't, and why. IEG evaluations provide an objective assessment of World Bank Group results, and identify lessons learned from experience. Through independent evaluation, IEG is helping the World Bank Group achieve its twin goals of eradicating extreme poverty and boosting shared prosperity.