Rethinking Evaluation - Sustaining a Focus on Sustainability

Taken together these dimensions of sustainability – economic, fiscal, environmental, and social – are complex. It will be hard and costly to try to address them systematically in all evaluations. At the same time, we evaluators cannot afford turning up with empty hands and concerns about missing data.

The UN Sustainable Development Goals (SDGs) have brought renewed attention to sustainability. Although the DAC evaluation framework includes sustainability as one of its five criteria, looking back on years of using this DAC evaluation criterion, one has to ask - how well have we done? And here I mean in evaluation practice rather than results.

More often than not have I seen sustainability used in different ways than it was originally conceived. The definition - “[s]ustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable” (OECD/DAC key terms for evaluation) - focuses clearly on the outcomes of the intervention and their sustainability.

Many evaluations, however, assess whether the projects themselves will be sustained, often concluding this to be the case when funding is secured from government or another donor. That is right, especially for facilities that continue to be run by the public sector and require government funding. But, the same is true for public-private partnerships (PPPs) where a recent evaluation of ours showed that the impact on government expenditure (in other words: fiscal sustainability) was hardly ever assessed.

Sustainability is also often taken as synonymous with environmental sustainability. When I was leading project evaluations, we hardly ever had the time and resources to assess environmental impacts and whether a project would leave a lasting footprint, positive or negative. At IEG we are in the process of evaluating environmental pollution projects the World Bank Group, which will shed some light on past practices, including data that is available today and remaining gaps.

But, under the SDGs environmental sustainability goes much further than a “simple” question of pollution. It is about the use and depletion of natural resources, about consumption patterns that are out of bounds, and the distribution of consumption patterns. For instance, when we look at access to electricity, our recent evaluation showed how underserved countries especially in Africa are. Sustainable Development Goal 7 is committed to expanding access to affordable and clean energy, increasing renewable energy sources, and attaining energy efficiency as a measure of improved consumption patterns. But, it will not be sufficient for evaluations of the power sector to assess efficiency gains that must be achieved in other parts of the economy.

And, closely linked to the World Bank Group’s goals of poverty reduction and greater shared prosperity, there is the question of social sustainability. Upheavals during the past years have often been rooted in growth that has excluded a broader base, where wealth, access, and voice have been captured by the few. The commitment to inclusive growth necessitates that we understand better the distributional effects of interventions, whether they were designed to target groups previously excluded or not. Almost more important for us is to evaluate and understand interventions that we believe to have no distribution effects to shed more light on the actual distribution of results that they have. IEG is in the process of evaluating the World Bank Group’s experience in this area to generate some early insights.

Taken together these dimensions of sustainability – economic, fiscal, environmental, and social – are complex. It will be hard and costly to try to address them systematically in all evaluations. At the same time, we evaluators cannot afford turning up with empty hands and concerns about missing data. We need to debate how we would evaluate interventions through these lenses of sustainability, stimulate that the right questions are asked during the design of interventions, and incentivize that relevant data starts being collected.

About the Author

Director General and Senior Vice President, Evaluation, World Bank Group

Caroline Heider joined the Independent Evaluation Group on October 1, 2011. She reports directly to the World Bank Group’s Boards of Executive Directors through the Committee on Development Effectiveness.

Comments

I would like to add other dimensions: in some cases i tend to evaluate sustainability (1) conventionally - in relation to the project/ intervention (the focus then is on its resources) and (2) its "legacy" i.e. sustainability of the project outcomes. The latter answers the question: what would sustain the project outcome?. It can be best captured by the change in behavior of the key stakeholders or/and building partnerships that will internalize and push forward those outcomes...

Thank you for this reflection on sustainability - and indeed the other posts on aspects of evaluation.
Two comments:
1. As part of the social sustainability you discuss, I would like to include institutional aspects and contextual aspects. First, it is critical that development interventions lead to a real, demonstrable and sustainable increase in the capacity of the relevant insitutions, be they those of government or civil society. This capacity must include how to design interventions from the perspective of sustainability, as well as the usual planning, implementing and evaluating capacities. Second, contextual aspects refer to how and why behavioural changes occur - or might be fostered - in the light of socio-cultural patterns in the local context. Ignoring such patterns has often led to interventions whose benefits could be neither sustained nor replicated.
2. Before sustainabiity became a watchword with its strong focus on environmental issues, my own thinking focused on sustainability at the local (=community) level, such that the intangible areas of vision, networking/relationships (see Kassem's comment above) figured as prominently as institutional development, capacity and funding. Indicators for intangible aspects need creativity, but can be developed.
So yes, achieving... and evaluating... sustainability is complex; but then social change always is :)

Connect With IEG

The Independent Evaluation Group evaluates the work of the World Bank Group to find what works, what doesn't, and why. IEG evaluations provide an objective assessment of World Bank Group results, and identify lessons learned from experience. Through independent evaluation, IEG is helping the World Bank Group achieve its twin goals of eradicating extreme poverty and boosting shared prosperity.