For better results, evaluate performance

The proposed Concurrent Evaluation Office will encourage analysis in the government for a start. Charan Singh writes.

The ministry of rural development is setting up a Concurrent Evaluation Office (CEO), a first of its kind in India. The ministry, with an annual budget of Rs 800 billion wants to understand the impact of its programmes so that it can successfully help reduce poverty and empower people in rural India. This is a laudable step by the Union Cabinet because in a diverse country such as ours, evaluations by skilled professionals will be able to credibly substantiate efficient utilisation of substantial developmental expenditure on welfare schemes.

The purpose of concurrent evaluation is to assess the performance of policies, programs, and processes and to learn what works in what context, all on an ongoing basis. As the scope of government operations and those of the public sector have grown significantly over the years, there has arisen the need to extend such offices to different ministries. Globally, a growing number of governments are working to improve their performance by creating systems to measure and help them understand their own performance. These systems for monitoring and evaluation are used to measure the quantity, quality and targeting of goods and services — outputs — that the State provides, and to measure the outcomes and impacts resulting from these outputs. Such evaluations have been introduced in many countries like Australia, Benin, Burundi, Canada, Chile, Colombia, Ghana, Kenya, Malaysia, Mexico, Senegal, South Africa, Uganda, Britain and the US.

Evaluation is different from research because it seeks utilisation-focused answers to stakeholders’ questions, simply compared with exploration and proof of a hypothesis. Audits are generally undertaken for fiduciary and legal accountability purposes, i.e., to examine whether there has been any mishandling of funds or legal wrongdoing, while evaluations are conducted to assess the effectiveness and efficiency of a policy, programme, or a project.

Evaluation, in general, aims to provide an objective assessment of the results of the project and helps to identify and disseminate lessons learned from experience. The objective of evaluation should be to enhance a learning culture, support institutional governance and oversight, strengthen the institutions of credibility and promote greater understanding of an institution’s work amongst its stakeholders. An evaluation office helps provide transparency, continuous learning and oversight by experts in the operations of the project. In order to garner unqualified credibility for the assessment and for the analysis to be meaningfully accepted and its recommendations implemented, it is important that an evaluation office is perceived to be independent.

To establish credibility, the evaluation office has to be meticulous in its staffing, selection of projects and topics, methodology, timeliness of completion and the dissemination of its findings. The staffing pattern, carefully planned, would have a mix of skilled internal and external professionals to warrant a balanced approach to evaluation. The internal members bring in domain knowledge while the external members provide a fresh outlook, and lend credibility and independence to the evaluation. Given the limited resources, the topics and areas chosen should be such that they appeal to a wider population and more stakeholders. The methodology adopted should generally be interactive and in consultation with policy makers, the designing and implementing team of the project, beneficiaries, local NGOs and other stakeholders. The interaction should be through surveys and structured interviews as well as desk research, and specific in-depth case studies. It would be useful to use the services of external consultants who not only bring expertise but also credibility as they are generally perceived to be independent by both, the beneficiaries and stakeholders. It is important that evaluation be conducted within a stipulated time frame to retain the relevance of its findings. The findings should be disseminated to a wider audience as this will help establish norms of credibility and transparency.

A beginning has been made and now it would be vital to develop a national evaluation policy framework to guide the practice of evaluation in the government or public sector and help develop common methodologies and terminology. Evaluation should be thought of as a part of the policy, planning, budgeting and monitoring cycle and must be given a separate dedicated cadre in the civil services. Successful evaluation needs to have a clear framework that is coordinated at the centre of government. The focus of evaluation should not just be on output but also on outcome, impact and sustainability of projects.

It is hoped that after gaining the experience of concurrent evaluation at the ministry of rural development, the government will also consider an independent office to evaluate select projects and their impact on the population. But will the expected beneficiaries finally get the promised benefits? The evaluation approaches should include assessing outcomes against stated objectives, benchmarks, and expectations, or assessing what might have happened in the absence of the project, programme, or policy — a counterfactual analysis.