UNICEF and the Global Goals

UNICEF is committed to doing all it can to achieve the Sustainable Development Goals (SDGs), in partnership with governments, civil society, business, academia and the United Nations family – and especially children and young people.

Evaluation database

Evaluation report

2008 India: Third Party Assessment of GOI-UNICEF Evaluation of the Quality Package for Primary Education

Executive summary

Background:The Sarva Shiksha Abhiyan (SSA) is the Government of India‘s (GOI‘s) national programme for universal elementary education (UEE). Launched in 2001, the programme has sought to ensure that all children in India completed five years of primary school by 2007 and elementary school by 2010. While SSA improvement in the quality of education is also a key focus area for SSA, it has not been able to take up this issue as a primary concern, given the high repetition rates, dropout rates and dismal learning achievement in primary schools.UNICEF‘s education programme (2003-07) supported SSA by focusing on improving the quality of primary education so as to make learning more attractive and effective. Thus, UNICEF India conceptualised a holistic, gender-sensitive, quality education package (QP) for implementation in primary schools in 2003. QP entailed a multi-pronged approach, involving strategies for the facilitation of across-the-board improvements in the entire curricular package. QP, therefore, included teaching-learning materials, classroom transactions, teacher training, assessment and examinations, while simultaneously building community-school linkages and dealing with infrastructure and environment issues. QP was implemented jointly by GOI and UNICEF in 13 states and 25 districts in the country from 2003 onwards. Each state team adopted QP differentially depending on the prevalent system of education in the state and its districts. UNICEF commissioned ICRA Management Consulting Services (IMaCS) in consortium with Pragmatix Research and Advisory Services Private Limited in November 2007 to undertake a third party assessment of QP in 4 of the 13 QP states.

Purpose/Objective:The objective of the assignment was to analyse the effectiveness of QP and evaluate how learning outcomes were affected by the six QP components:I. Development and supply of essential teaching-learning materials (TLMs)II. In-service teacher training (TT)III. Academic support to improve active learning, continuous, supportive student assessment and rational class management (AS)IV. Improvement of classroom environment (CE)V. Improvement of school environment and facilities (SE)VI. Promotion of community participation (CP)

Methodology:A specifically devised methodology in two distinct modules was used to understand the impact of QP. In the first module, a Quantified Participatory Assessment (QPA) was undertaken at the district level to understand in qualitative and quantitative terms how QP schools were performing vis-à-vis non-QP schools and as per UNICEF‘s multidimensional inputs. During the second module, an overall analysis of the policy environment at the state-level and the role of different stakeholders at both the district and state levels were done. QPA collected qualitative and quantitative information from 141 schools in the 4 districts (of which 103 were QP schools and 38 were non-QP) to ascertain how QP schools were performing vis-à-vis non-QP schools in terms of the six QP dimensions and chosen output indicator—learning achievement of school children. The learning achievement was tested in language (local) and mathematics for children of Classes 3 and 5, since these were the classes for which UNICEF-commissioned learning assessment tests were available. Field teams comprising of 5 to 6 members spent a day in each school, observing school and classroom environments and facilities, discussing the quality of teacher training with teachers, academic support and the use of TLMs, and meeting community representatives to discuss community participation in school activities. The assessment was completed between February and March 2008, after which the data was collated, cleaned and analysed. The findings have been reviewed by UNICEF‘s State Education Program Officers (EPOs).

Findings and Conclusions:Overall, from an analysis of stakeholders reflections in conjunction with district-level QPA findings about QP, it is apparent that conceptualising quality in terms of six separate dimensions, UNICEF has changed the policy and programme context in how quality can be addressed in the field, not only in the UNICEF-supported-districts but in the entire state. This is true in both comparatively advanced (AP and Orissa) and lagging QP states (MP and Bihar). However, as there are different degrees of difficulties in implementing the different dimensions, there has been a noticeable focus on infrastructural elements such as TLMs, CE and SE (in order of priority). Generally, across all the states, the other three dimensions have generally been neglected (TT, AS and CP).The conclusions have been categorised according to the benchmarks for impact assessment—relevance, effectiveness, efficiency, sustainability and impact—and are presented below. Relevance: Because of UNICEF‘s initiative, quality of primary education has become essential (no longer ―just desirable) to both the discourse and the implementation of primary education across the country. At the district level (instrumentally relevant administrative unit), however, there is a single-minded focus on ensuring access in quantitative terms. Thus, QP has not been able to demonstrate the relevance of focussing on quality to achieve UEE. The infrastructural aspects of QP have been more readily adopted by SSA. However, innovations such as the learning ladder based on cards, which help the learner to take an independent path of learning, have been successively diluted by the SSA authorities, for instance, in AP. The erstwhile system has integrated the TLMs developed by the Chittoor district but only by diluting their innovative design (and thereby reducing the potential for improving quality). Hence, QP was relevant conceptually and has helped to shape the policy discourse on both why and how the quality of primary education in the country should be augmented. However, this relevance was diluted during implementation in most of the districts/states and as a result contrary to QP‘s conceptualisation, quality itself seems to have been compartmentalised (into different dimensions) and the focus on integrating all the dimensions together has been lost. Effectiveness: Despite having well-established offices in the states as well as good working relationships with the key state departments, UNICEF was not effective in rolling QP as it lacked the critical support from teachers. For successful implementation, the teachers must own the package and treat it as a broad canvas or a framework to facilitate learning and also innovate in the classroom.Efficiency: IMaCS‘ earlier study5 for UNICEF for designing the analytical framework for cost-benefit analysis was stymied by the lack of financial data within UNICEF regarding the cost of QP; programme costs are not easy to disentangle from support costs. As a result, this impact assessment was not able to determine the efficiency of QP. Further, it was difficult to estimate the costs.Sustainability: Given the fragmentation witnessed during implementation, the sustainability of QP without UNICEF support, as envisaged earlier, is not possible. QP itself has not yet demonstrated its full potential in a cost-effective manner; however, various dimensions of the package such as the TLMs (and even TT) have been replicated by the SSA. This certainly can be constituted as a success for the package. However, from the point of conceptualisation the incorporation of only some parts (dimensions) of the package instead of the whole package can also be viewed as a failure. Impact: QP has demonstrated a new approach to education to both teachers and students, so much so that there is a demand for it to be introduced into higher classes. However, QP‘s impact has been limited because of feeble implementation; particularly due to its lack of ability to engage teachers effectively.

Recommendations:Overall UNICEF should invest greater energy and efforts in engaging and involving the state governments and agencies in implementing QP in its true spirit. Clarify the entire QP implementation system so that it is easily understood by all stakeholders, particularly community representatives, parents and teachers. Reconstruct QP to embrace the whole school and do not restrict to just a few grades. Provide special training to teachers and all key stakeholders in quality management in education, supported by relevant and high-quality training material on quality management in primary and elementary education. Turn QP from a supply-driven to a demand-driven programme, by involving teachers and other stakeholders effectively from design to implementation and monitoring. Create a more prominent and inspiring role for mothers in implementing QP in schools, to foster greater participation from parents. Make QP dynamic, so that it can respond quickly to changes in policy and educational technologies.Teachers Involve teachers more in QP design and implementation, to build ownership and foster innovation. Train and orient leaders of teacher organisations to participate more effectively in QP design and implementation. Re-design teachers‘ capacity building programmes to improve personal effectiveness of teachers (soft skills) first, followed by competence in instruction and content. Develop illustrated and attractive advocacy literature for teachers in place of conventional and uninspiring teacher manuals. Build new mechanisms for on-the-job mentoring of teachers in QP. Create institutional self-evaluation toolkits for teachers, as this is not only a more powerful tool of organisational diagnosis, but it also fosters ownership of findings among the teachers. Complement QP package with an ICT kit, to increase its adoption and effective use. Strengthen the SLM inputs as these are directly used by students.CRC and BRC Identify competent and trained staff for the CRCs and BRCs, drawn from among teacher educators in universities, colleges of education and DIETs, who can effectively mentor teachers, support QP implementation, and generate resource groups in states. Free CRC and BRC coordinators from routine administrative work, so that they can perform their roles and discharge their responsibilities effectively.

Lessons Learned (Optional):Teachers are the main variable, but have not been engaged effectively by QP: Having failed to engage the teacher on all the integrated dimensions, QP has not been able to differentiate itself from the supply-driven approaches. It has thereby fallen short of its goal of instituting a (support) system for delivering quality in the classroom. Supplying more or better TLMs does not increase their use: QP intervention has only improved the supply of TLMs, but has not affected their usage in class by teachers. TLM use, however, has not been a focus area for QP implementation. Need better implementation planning for a successful QP: Managing the complex interfaces required for the success of QP across varying interests (and abilities) of different stakeholders (especially parents and communities) needs a logical framework or result-based management approach culminating in a time-bound implementation matrix.