Translate

Tuesday, August 15, 2017

Evaluability Assessments answer the question: Is there enough available information to justify the time and cost of a full-scale evaluation? If they are done early enough during implementation, they can identify basic problems in design which can guide remedial action.

One old study set the stage for 7 recent detailed Guides which synthesize earlier work and provide useful advice on whether, when and how to undertake an evaluability assessment – or to use other approaches to assessing project or programme design integrity.

Evaluability Assessment Guides

Level of difficulty: ModerateLength: 3-72 pagesPrimarily useful for: Evaluation managers, RBM specialists doing evaluation assessmentsMost useful: DFID Guide to Planning Evaluability AssessmentsMost interesting: Program Management and the Federal Evaluator (1974)Limitations: These guides tell us what needs to be done, but they require people with the process skills to do it all.

The History of Evaluability Assessments

Evaluability assessments, pre-assessments or exploratory evaluations, as they were known in some cases 40 years ago, have been used for many years in public health research where the term “evaluable” refers to ‘Patients whose response to a treatment can be measured because enough information has been collected”, education, and justice (PDF) and the term has academic antecedents in testing mathematical propositions, far back into the 19th Century.

The Foundation Evaluability Assessment Document

The foundation evaluability concepts for social programmes were initially presented, from what I can see in a 1974 Urban Institute study, one of many emerging during implementation studies of the Johnson administration's Great Society Program, published as "Program Management and the Federal Evaluator" in the Public Administration Review, included in a 1977 volume of Readings in Evaluation Research, available through Google Books.

The Foundation Evaluability Assessment article

It advocated looking at issues which are at the heart of current guides to evaluability assessment - and for that matter, at the heart of solid results-based design:

Whether there is a clearly defined problem addressed in design

If the intervention is clearly defined

If the short and longer term results are defined clearly enough to be measurable

if “the logic of assumptions inking expenditure of resources, the implementation of a program intervention , the immediate outcome to be caused by that intervention, and the resulting impact” are “specified or understood clearly enough to permit testing them”.

Whether managers are capable of and motivated to using performance data for concrete management decisions.

While there are many other guides to evaluability assessment which are much more detailed of more practical utility today, the most important core concepts are here in this 1974 study, and this short, 43-year old article is still worth reading for its insights on clarity on language, assumptions and results.

Changes in Evaluability Assessment Utilization

Evaluability Assessments started out as ways of improving the design of evaluations, to increase the chances they will be useful to the people who are funding and implementing activities, and in recent years, with a substantial increase in the number of evaluations of international development projects, many agencies, including UNICEF, the World Bank, UNDP and numerous bilateral donors have incorporated evaluability assessments as part of the project cycle, after design, during implementation and before a decision is made to pay for a full-scale evaluation.

Many Evaluability Assessment Guides have been produced in recent years, but here are just a few worth noting.

The DFID Guide to Planning Evaluability Assessments (PDF) (2013-56 pages) produced by Rick Davies, who has a host of useful evaluation documents and websites to his credit, did a detailed job of synthesizing the literature, to provide suggestions on when and how to proceed with an assessment, and what the possible consequence of doing this could be for project or programme design changes.

"In an ideal world projects would be well designed. One aspects of their good design would be their evaluability. Evaluability Assessments would not be needed, other thanas an aspect of a quality assurance process closely associated with project approval (e.g. as used by IADB). In reality there are many reasons why approved project designs are incomplete and flawed, including:

Political needs may drive the advocacy of particular projects and override technical concerns about coherence and quality.

Project design processes can take much longer than expected, and then come under pressure to be completed.

In projects with multiple partners and decentralised decision making a de facto blueprint planning process may not be appropriate. Project objectives and strategiesmay have to be “discovered” through on-going discussions.

Expectations about how projects should be evaluated are expanding, along with the knowledge required to address those expectations.

.... In these contexts Evaluability Assessments are always likely to be needed in a postproject design period, and will be needed to inform good evaluation planning...

…Many problems of evaluability have their origins in weak project design. Some of these can be addressed by engagement of evaluators at the design stage, through evaluability checks or otherwise. However project design problems are also likely to emerge during implementation, for multiple reasons. An Evaluability Assessment during implementation should include attention to project design and it should be recognised that this may lead to a necessary re-working of the intervention logic."[p. 9]

The DFID guide goes through every step of planning and executing an evaluability assessment:

When evaluability assessments are appropriate

Who should conduct the assessment – the types of expertise required for evaluability assessments

How to contract evaluability assessments

How long the assessments could take

How much it will cost to do an evaluability assessment

Different processes required for an evaluability assessment

Detailed lists of questions to be dealt with when assessing project design, assessing availability of indicator data, and stakeholder participation in the evaluability assessment

The types of reports which should come out of an evaluability assessment

The risks of undertaking evaluability assessments.

Project Design Checklist
Click to enlarge

This report also includes annexes summarizing different suggested models of stages suggested by different researchers and agencies, in conducting an evaluability assessment, and even a draft Terms of Reference for an evaluability assessment – something which seems to have been used almost verbatim subsequently by many aid agencies.

2. The UN Office on Drugs and Crime Evaluability Assessment Template

Increasingly, these assessments are now being used early in the implementation period, as an evaluability assessment template produced by the United Nations Office on Drugs and Crime puts it, as a means of cross-checking the validity of the original design, with the possible purpose of enabling mid-course design changes.

The overall purpose of an evaluability assessment is to decide whether an evaluation is worthwhile in terms of its likely benefits, consequences and costs. Also, the purpose is to decide whether a programme needs to be modified, whether it should go ahead or be stopped.

Timing:

The evaluability assessment is appropriate early in the programme cycle - when the programme is being designed but has not yet become operational. A second opinion on a programme and the strength of its design and logic is only worthwhile at this early stage - when something can be done to remedy any weaknesses

This brief 3-page checklist may prove a bit too constrictive in its decision-making path to be applied literally in all evaluability assessments in other fields, but the questions it asks about the credibility of the design, data collection systems, the utility of an evaluation for management are worth reviewing, as is the advice to use theory of change workshops to test logic models and programme theories.

They are, in this context, quality control activities, in effect very early mid-term reassessments of the Results-Based Management process.

(2015 – 24 pages) produced by Greet Peersman Irene Guijt and Tiina Pasanen for the Overseas Development Institute.

This guide provides “guidance, checklists and decision support” in a simple and easy to read format, expanding on the checklists provided by Rick Davies in the 2013 DFID guide, but aimed primarily to be useful for those who want to use the assessment not for project or programme design, but prior to deciding whether to undertake a full scale impact evaluation. It provides a series of easy to read checklists on questions on designing and conducting an assessment. with some practical examples of the resources required for an evaluability assessment.

Level of Effort required for an Evaluability Assessment
Click to enlarge

Guides for Assessing Evaluability in Conflict Situations

Four guides deal with a very practical set of questions about whether to undertake evaluations in conflict situations.

This is a short, easy to read checklist on key questions to consider before deciding on whether and how to do an evaluation in conflict situations. It takes the checklist produced in 2013 by Rick Davies, for DFID, and expanded the content of some of those checklist issues in language intended to be easier for programme managers and others who are not evaluation specialists, to understand raising some practical questions to consider when assessing the costs and benefits of evaluations.

In a similar vein, the CDA Collaborative has a number of interesting guides on whether to use evaluability assessments in peacebuilding and conflict situations, or alternatively, to use what is called programme quality assessment. Many programmes which are not focused specifically on peacebuilding or conflict resolution could benefit from considering the alternative approaches:

6. An Alternative to Formal Evaluation of Peacekeeping (2012 – 34 p) by Cordula Reimann, Diana Chigas & Peter Woodrow compares the utility of and resources required for three approaches – formal evaluation, Program Quality Assessment, and internal Reflection Exercises, for organizations in different situations, and with different needs. People working in areas not related to conflict could find the Program Quality Assessment model a useful alternative to a full evaluability assessment, also.

Comparing Reflection, Program Quality Assessment and Evaluation
Click to enlarge

Conclusion: Using Results Based Management in Evaluability Assessments

Evaluability Assessments apply basic concepts of Results-Based Management to determining, as the original 1974 study proposed, if the original project or programme design:

Addresses a clear problem,

Has an intervention strategy which is logical, and shared by stakeholders,

Has access to information for results indicators which will describe or measure progress against the original problem, and which are likely to be useful to funding and decision-making authorities.

Some evaluability assessments have been limited to document review, but in my experience these can be sterile, almost academic undertakings, not likely to provide solid information on stakeholder agreement to the logic of the intervention.

But, as part of the evaluability assessment process, if field work is recognized as a necessary component, the basic components of Results – Based design will be essential in conducting or engaging in evaluability assessments.

Getting to the stage where we understand if there is enough coherence in a programme or project to evaluate it, means using many of the same skills and processes of stakeholder consultation which are they key to solid Results-Based design:

Identifying stakeholders

Getting stakeholder agreement on actionable problems

Identifying with stakeholders possibly multiple causes of a problem

Agreeing on which causes will be the target of the intervention

Clarifying assumptions about what interventions work and what is appropriate in the context of the intervention

Getting stakeholder agreement on what inputs, and activities will be necessary to achieve results

Putting this together into a coherent theory of change with all of the stakeholders

If we can successfully engage with stakeholders in these processes, we have a reasonable chance of conducting a useful evaluability assessment.

The bottom line: The earliest of these studies outlined the basics, and recent guides provide the details on how to do it, but we still need the people with process skills to make evaluability assessments more than just a buzzword.

Greg Armstrong is a Results-Based Management specialist who focuses on the use of clear language in RBM training, and in the creation of usable planning, monitoring and reporting frameworks. For links to more Results-Based Management Handbooks and Guides, go to the RBM Training website.

About Me

Greg Armstrong brings what we know about how adults learn to helping international development workers use Results-Based Management in their work. If it is done right, it can be enjoyable, and productive, helping us explain our work to others.