2 Evaluation of Social Service Delivery: A Practitioner s Perspective The issue of evaluation raises two further important questions: Evaluation by whom? Evaluation for what? These critical questions need to be well understood, because the answers to these questions can provide us with important political and programmatic insights in the delivery of the social good. In this paper, I propose that in evaluating the delivery of any social service, the following four questions need to be borne in mind: 1. Has the social service been delivered to the beneficiaries in the intended manner? Is it having the intended effect/ impact on them? 2. What other aspects of the beneficiaries/ stakeholders has this social good or the form in which it has been delivered impacted? 3. What aspects of the political/ cultural/ social setting in the area have impacted to the success/ failure of the initiative? 4. How do we gain insights about how the beneficiaries/ stakeholders can evaluate the quality of service delivery on their own in the long-term? This four-question framework is expected to lead to improved delivery of the social service. In the course of this paper, I cite three specific initiatives, all in the context of universalisation of elementary education in India. In the first instance, the evaluation of learning outcomes is done by teachers, but increasingly gives an opportunity for the parents to understand and participate. In the second, which stands on the shoulders of the first, the programme is designed for evaluation by an independent agency for schools that volunteer to be evaluated. And the third that describes the evaluation of a service delivered in this case describes the findings of the evaluation of outcomes in an initiative using IT in elementary schools. As we begin a discussion on the aspect of evaluation, it is worth understanding what The World Bank, which is among the largest organisations funding poverty alleviation and other development projects around the world, has to say about evaluation. The Poverty Net website of the World Bank has written extensively about evaluation of poverty alleviation programmes: Impact evaluation is an assessment of the extent to which interventions or programs cause changes in the well-being of target populations, such as individuals, households, organizations, communities, or other identifiable units to which interventions are directed in social programs. One way of conceptualizing net effects (or outcome) is the difference between persons or other targets who have participated in a project and comparable individuals, or entities who have not participated in the project. Therefore, an impact 2

3 evaluation must estimate the counterfactual, which attempts to define a hypothetical situation that would occur in the absence of the program, and to measure the welfare levels of individuals or other identifiable units that correspond with this hypothetical situation. 1 Getting illiterate parents to evaluate their child s learning Karnataka is a state in South India with nearly 8.3 million children enrolled in elementary school. More than 85 percent of children in the state are enrolled in government schools. One of the major problems faced by the system here is the low academic achievement levels of children who go to government schools. The government has initiated a number of programmes based on providing inputs to government schools. The assessment of these initiatives reveal favourable findings but the magnitude of change falls short of expectations. Until recently, the Government of Karnataka did not have a formal report card that was issued to parents of children at the primary school level (Grade 1-4). In 2001, the government took up an initiative to provide report cards of children s academic performance to the parents periodically. The progress report card indicated the child s achievement levels in terms of grades A to D. It was assumed that such grading would be a way to communicate with parents about their child s performance. A quick tour of several schools, and discussions with parents pointed to two basic flaws in this approach: 1. A number of children in government schools are first generation literates i.e. their parents are illiterate. This report card assumes that all parents would be able to understand what the grades meant. 2. It provides no scope for parents to identify the specific academic competencies that their child does not possess. In such a situation, the parents will be unable to assist the child to gain the competencies that the child lacks. In addition, there were technical difficulties in the format because of which teachers found it difficult to fill the report. Deeper probing as part of this informal evaluation of the utility of the existing tool to measure the quality of delivery of the social good (in this case, learning outcomes) provided some important insights: 1. Some of the important factors that can have a significant impact on the quality of learning outcomes is parental awareness of the expected learning outcomes, cooperation and pressure all acting at the same time. 3

4 2. Cooperation and meaningful pressure on teachers can be brought more effectively if the parents are aware of what their children are supposed in know in any grade. 3. The centrally printed and standardised textbooks couch the expected learning levels in highly technical language, difficult even for a teacher to understand. 4. Most teachers have a tendency to measure children s achievement on the content in the textbook (which requires rote learning), rather than the competencies that are intended to be covered in the lesson (which requires an understanding of the content/ competency). 5. The lack of understanding on the part of the parents about the expected learning levels results in an information imbalance between parents and teachers, thereby significantly weakening the bargaining position of the parents. These observations led to the following ideas: 1. Simplify the technically worded expected learning outcomes defined by the government (controversially called Minimum Levels of Learning) to commonly understood language reduce the information imbalance 2. Identify the key competencies that need to be listed in the report card (for monitoring purposes), without compromising on the learning outcomes to be attained by the child simplify the process for easy evaluation by parents 3. Prepare a report card format that is easily understandable by teachers and from which parents can identify which specific competency their child needs extra assistance provide an opportunity for parents to help their child learn 4. Communicate with parents about how to use this report card for the benefit of their child improve the chances of better quality delivery A 10-week intensive process involving classroom teachers from rural government schools, development of a draft report card, field testing with teachers and parents and subsequent modifications, led to the creation of a new improved and acceptable version of the report card. This report card has addressed some of the pitfalls of the previous one. However, preparation of the new version of the report card is no panacea it is only one small step towards effective evaluation of learning outcomes of children. The next steps required to make this approach effective and increasingly local are being worked out. However, the fact is that for learning to become effective and lasting, evaluation of learning outcomes needs to be simplified to a level in which even illiterate parents are able to evaluate their child s learning. Work is underway to further simplify the articulation of academic competencies, 4

5 which the child has to learn and simultaneously develop tools for effective evaluation of these competencies by illiterate parents. The question of whether the delivery of the social good (learning outcomes) has in fact improved because of the changes in the report card is indeed very important. However, it would be important to understand the other processes that are at work when such an initiative is launched that contribute to either the success on failure of this intervention. Learning Guarantee If your child comes to my school regularly for a year, I will guarantee that your child will know at least these identified competencies. What is the possibility of getting schools to commit to learning outcomes and then work towards ensuring that children in fact achieve those learning outcomes? The Learning Guarantee Programme (LGP) is a multi-layer strategic response developed by Azim Premji Foundation to the problem of low attainment levels by children in elementary schools across the country. Studies have indicated that primary-level learning achievement is low in the country where majority of grade 4 and grade 5 students in government have not acquired basic literacy and numeracy skills (Govinda and Varghese 1993; Saxena, Singh, and Gupta 1995) 2. The LGP is structured as an award based programme spread over three years. Azim Premji Foundation announces the programme and invites applications from government schools volunteering to participate in the programme. Upon signing up, the school receives a Prospectus, which details out the contours of the programme and the academic competencies that an independent group of academics will measure the children on. The school has the option of indicating the exact time when their school will be ready for evaluation. Any school that is able to demonstrate student achievement of a certain level gets the award. For the school to qualify for the award the following year, the school needs to demonstrate continued superior performance. The Foundation is putting in place a simple mechanism to track a number of schools that have opted to be part of the LGP. This tracking will involve two approaches: (a) Case studies of participating schools to observe what happens in these schools from the day they sign up for the programme to the time of evaluation. (b) Detailed research on a handful of schools on a variety of factors that affect learning outcomes. The independent academic team that will visit the school for the evaluation will only measure terminal performance. But the greatest learning value will come from the close observations in schools about 5

6 the process in which the learning guarantee is achieved. The evaluation process of learning guarantee programme will be inadequate if at least the following aspects are not covered: 1. How did children in the learning guarantee school perform on the terminal tests administered by the independent evaluators? 2. What other processes have occurred in the learning guarantee schools after they have signed up for the programme? 3. What are the enabling factors in the village and the school that enabled the achievement of the learning guarantee? Do these factors exist in the villages/ schools that did not sign up for learning guarantee? Can these factors be simulated in all other villages? 4. What was the involvement of the community to track the learning levels of children during the period prior to evaluation? Are there aspects of evaluation that can be communicated effectively to parents, so they may be able to evaluate their children s learning long after the independent evaluation team is gone? The Use of IT in Rural Government Schools Parents in rural communities in India have increasingly expressed a desire that their children should know English and should develop basic IT skills. Azim Premji Foundation decided to pilot a programme on the use of IT in rural government schools in Karnataka by setting up Community Learning Centres (CLCs). Typically the CLCs have about 6 computers and allow about 3 class periods of access to each child between grade 3 to 6 in the school. The CLC is managed on a day-to-day basis by a local youth who is provided basic training to work with computers in a school environment. We had the following main questions for our investigation on the effectiveness of CLCs in achieving universalisation of elementary education goals. Will the use of IT lead to 1. Attracting the out-of-school children back to school? 2. Improvement in attendance rates? 3. Improvement in academic achievement levels of children in these schools? In an attempt to gain an understanding of whether out-of-school children came back to school, we considered transition rates from one class to the next. Transition rate is a good indicator of whether more children joined a particular school in comparison to the enrolment rates of the school in the previous year. For example, it is expected that in a class of 25 children in Standard 1, all children will move to standard 2, barring those who are detained/ retained or those who drop out of school. If there are more children in standard 2 than the 25 in standard 1, this could happen only if children transfer from other schools to this school or if out-of-school children have joined this school, laterally to standard 2. 6

7 The adjoining graph showing the Transition Rate Comparison, points to interesting results. Some observations on the graph: Percentage Change Over the Previous Year Transition Rate Comparison Year a. We began preparatory work to set-up CLCs in December This 2 led to the setting-up of 34 CLCs between 3 March and August in the 34 government 5 schools. An increase in 6 enrolments would be 7 expected in June/ July 2001 and b. In June/July 01, and June/July 02 (since the setting up of the 34 CLCs), there have been consistent and increases in enrolments, over the previous year. This points that more children have joined these set of schools in various grades than in the previous two years in a row. c. The spurt in transition rates in Standard 5 and 6 seen are largely on account of change in government policies. d. We do not have data, however, to conclusively state how many of these children were out of school children, and how many have transferred from other schools. e. We will not be able to conclusively state based on the quantitative information available, whether children joined the school because of the on-going efforts by the government to enrol all children into school, or because they were attracted to school by the computers. Interviews with parents and children however, reveal that a number of children came back to school because they were excited at the prospect of playing on the computer. (Most of the educational content in the CLCs is given in the form of education games.) Attendance Rates in school are also expected to show an improvement. However, it is widely believed, that attendance records maintained in many schools are not dependable. Only anecdotal evidence has been gathered, which overwhelmingly supports the hypothesis that children are indeed attending school more regularly since the CLCs have been set-up. 7

8 The statistical analysis of the data on academic achievement levels of students from standard 3 to 6 revealed that the learning levels of children in schools with the CLC were significantly better than students of those schools who were not exposed to the intervention. It is even more revealing to see that there is a dramatic drop in the number of children who were in the lowest quartile of scores between the pre and post test in CLC schools (from 118 to 32 children in language; 130 to 37 children in math), as compared to only a marginal drop (from 112 to 88 children in language; 164 to 116 children in math) in the control group schools. The software that was provided to children in the CLC schools was designed to partially address only the academic competencies of grade 1, 2 and 3. Of the children tested on grade 2, 3, 4 and 5, the most important target group of this initiative was the lowest performing children in each of these classes. To that end, the programme seems to have met the objectives. The pilot project of setting up 34 Community Learning Centres (CLCs) has given a number of insights into the use of technology in government schools. Our plan was to provide financial support to the youth to manage the centre for a period of 12 months from start of the programme. We have stopped providing any direct financial support to these schools since April 1, Until now, we have not had any instances of the community refusing point blank to participate in making the CLC financially sustaining. Only part of the sustenance has happened through revenue generation at the CLCs, the balance coming from Gram Panchayat (the local village council), local milk dairy, etc. In our eagerness to study the impact based on the three identified areas of improvement in enrolment, attendance and achievement, if we ask only questions pertaining to the direct impact of the programme, we will likely miss a number of important insights into factors that have enabled the programme. Applying the four-question framework would be critical to understanding the programme from a practitioner s point of view. Let me illustrate the importance of looking beyond the three questions that we have asked ourselves in the case of usage of IT in rural schools. After we stopped paying the Fellowship at the end of 12 months to a school in Anoor village (about an hour and half away from Bangalore city), the community got together and decided that the youth will (a) be allowed only one day a month of leave, (b) keep the CLC open for an extra hour if there is a power cut during school hours so children don t lose out on computer time. They have also started asking for new content in CDs to be given to them. When parents start taking such active interest in the affairs of the CLC, there are obvious implications on school governance. It is entirely possible for us to completely lose sight of this if we have a narrow focus in evaluation. While we actively debate the evaluation of social service delivery, there are also a number of voices that caution us against narrow interpretations of measurement and evaluation. As Mintzberg says: 8

9 Next, consider the myth of measurement, and ideology embraced with almost religious fervor by the Management movement. What is its effect in government? Things have to be measured, for sure, especially costs. But how many of the real benefits of government activities lend themselves to such measurement? Some rather simple and directly delivered ones do especially at the municipal level such as garbage collection. But what about the rest? Robert McNamara s famous planning, programming, and budgeting systems in the U.S. federal government failed for this reason: Measurement often missed the point, sometimes causing awful distortions. (Remember the body counts of Vietnam?) How many times do we have to come back to this one until we finally give up? Many activities are in the public sector precisely because of measurement problems: if everything was so crystal clear and every benefit so easily attributable, those activities would have been in the private sector long ago. 3 The larger canvas Norris, Smolka, and Soloway 4, in their paper on Convergent Analysis, urge researchers to provide information to practitioners on the available research regarding the topic being researched. This, they argue, provides valuable insights to practitioners about similar initiatives taken up elsewhere, and facilitates learning. The essence of evaluation is to learn. Learn whether an initiative has worked learn why it has worked learn how to enable the direct stakeholders of the initiative to evaluate the impact and quality in the long term. The delivery of a social good is a complex process and can t be seen as the delivery of a service in a simplistic manner. For example, delivery of the social good is not necessarily only a matter of working out the logistics of distributing, say a vaccine, but involves complex social and political processes. A comprehensive evaluation of social service delivery needs to be able to comprehend and capture these aspects. 1 Poverty Net website, World Bank Group, 9

ACCELERATED LEARNING PROGRAM An End line Evaluation Study July 2004 Conducted Jointly By RVEC and The Azim Premji Foundation 1. INTRODUCTION 1.1 Background Azim Premji Foundation is a not for profit organization

EXPLORATORY DATA ANALYSIS USING THE BASELINE STUDY OF THE KHANYISA EDUCATION SUPPORT PROGRAMME Charles Simkins and Carla Pereira 1 Introduction The Khanyisa Education Support Programme aims to facilitate

WHAT DO THEY KNOW? A summary of India s National Achievement Survey, Class V, Cycle 3, 2010/11 Supported by SSA TECHNICAL COOPERATION FUND Contents Introduction 3 States of change why we evaluate 4 The

WHAT WORKS IN INNOVATION AND EDUCATION IMPROVING TEACHING AND LEARNING FOR ADULTS WITH BASIC SKILL NEEDS THROUGH FORMATIVE ASSESSMENT STUDY OUTLINE The OECD s Centre for Educational Research and Innovation

EFFECTIVE SCHOOL MANAGEMENT CREATE INDIA POLICY BRIEF 4 FEBRUARY 2011 EFFECTIVE SCHOOL MANAGEMENT COMMITEES Community based organisations, such as School Management Committees (SMCs) and other Panchayati

PRE-PRIMARY EDUCATION IN INDIA CREATE INDIA POLICY BRIEF 1 DECEMBER 2010 PRE-PRIMARY EDUCATION IN INDIA Various types of pre primary schools are available in India and more children are now attending pre-school

6. A framework to plan monitoring and evaluation Key ideas Monitoring and evaluation should be an integral part of (and therefore affect) all stages of the project cycle. As mentioned earlier, you need

Identity and Needs in the Modern World: Roles of Orality and Literacy 15 th Inuit Studies Conference, Paris, Oct. 26, 2006 An overarching goal in Inuit communities is to maintain Inuit culture as a dynamic

Profile of New Brunswick High School Students: Their Reading Skills Research Program on Post-Secondary Education and Training Opportunities in New Brunswick Report #2 February 27, 2007 Undertaken by Tomasz

Funding success! How funders support charities to evaluate A review of Evaluation Support Accounts with Laidlaw Youth Trust and Lloyds TSB Foundation for Scotland The process of working with ESS has greatly

City District Government Faisalabad Strategic Policy Unit Whole School Development: A Holistic Approach to Education The trouble with education (GUIDANCE NOTE) In 2000 the Dakar Conference in Senegal rallied

Annual National Assessment (ANA): A SADTU perspective Executive summary The Annual National Assessment (ANA) is an assessment instrument introduced by the Department of Basic Education (DBE) in 2011 to

Teacher Education for inclusive education in Finland Hannu Savolainen University of Jyväskylä Teacher Education for inclusive education in Finland Background of the Finnish comprehensive education system

Andhra Pradesh School Choice Project Proposal 1. Background: In recent years, access to primary education has expanded tremendously in India and gender gaps have narrowed. Approximately 95% of both boys

Lecturer Job Description - Maths Graduate Date : May 2014 Reports to : Section Leader, Skills for Life Introduction Bicton College s Values are embodied in the following statements: We are student focussed

Perspectives in Learning: A Journal of the College of Education & Health Professions Volume 11, Number 1, Spring 2010 Columbus State University Taking the Lead Role in Intern Supervision: The Mentor Teacher

Personnel What exactly are the new roles of the School Psychologist and Social Worker? While these aren t new roles, School Psychologists and Social Workers will continue to work in schools by gathering

Position Papers from Azim Premji Foundation Azim Premji Foundation is a not for profit organization established in 2001 with a vision to facilitate a just, equitable, humane and sustainable society Page

IBE/2006/EFA/GMR/CP/78 Country profile prepared for the Education for All Global Monitoring Report 2007 Strong Foundations: Early Childhood Care and Education South Africa Early Childhood Care and Education

Assessment FOR Learning, the Achievement Gap, and Truly Effective Schools* Rick Stiggins ETS Assessment Training Institute Portland, OR * A presentation at the Educational Testing Service and College Board

The Current Situation and Future Prospects of Japanese English Teaching YAMAOKA Kenji 1. The Dichotomy between Grammar-translation and Communication 1.1 Grammar-translation Methods for the Examinations.

RESEARCHER S VOICE IN WEB-BASED COURSE MANAGEMENT SYSTEMS Written by Marianne S. Castano, Ed.D. Research and Development Associate GEMS (Gender Equity in Mathematics and Science) A Project of the Education

What is Quality Schools Development? Why are we doing this? All students deserve quality schools in their neighborhoods that help them thrive academically, socially, and emotionally. Right now, several

How to Measure and Report Social Impact A Guide for investees The Social Investment Business Group January 2014 Table of contents Introduction: The Development, Uses and Principles of Social Impact Measurement

SLDS Spotlight Data Use Through Visualizations and Narratives Data, on their own, may not tell a story to the untrained eye. It takes analytical skills and an understanding of the data to draw out meaningful

FINDINGS FROM AN INDEPENDENT EVALUATION OF THE AMNH s ONLINE SEMINARS ON SCIENCE COURSE: GENETICS, GENOMICS, GENETHICS Inverness Research studied the AMNH Seminars on Science program for eight years, from

BUILDING EVALUATIVE CAPABILITY IN SCHOOLING IMPROVEMENT POSITION PAPER 9 Learning for Leadership: Building Evaluative Capability through Professional Development for School Leaders D. Le Fevre Why a Focus

TEFP1011 Susanne Hashmi-Lewis - Kings College Guildford Improving the Progression, attainment and enjoyment of KS3 students in Mathematics using single sex classes in a co-educational, comprehensive school

A Guide to Curriculum Development: Purposes, Practices, Procedures The purpose of this guide is to provide some general instructions to school districts as staff begin to develop or revise their curriculum

Professional Graduate Certificate in Education (FE and Skills Sector) PGCE This teacher training programme is for individuals who wish to gain a nationally recognised Level 6 teaching certificate in the

The MetLife Survey of Challenges for School Leadership Challenges for School Leadership A Survey of Teachers and Principals Conducted for: MetLife, Inc. Survey Field Dates: Teachers: October 5 November

Modern student progress looks like this An online high school uses data to help visualize each student s journey Overview In Colorado, the mandate is clear: ensure students at each grade level are being

UMEÅ INTERNATIONAL SCHOOL OF PUBLIC HEALTH Master Programme in Public Health - Programme and Courses Academic year 2015-2016 Public Health and Clinical Medicine Umeå International School of Public Health

Education for All Global Monitoring Report Proposed post-2015 education goals: Emphasizing equity, measurability and finance INITIAL DRAFT FOR DISCUSSION March 2013 The six Education for All goals have

Literacy catch-up programme Fischer Family Trust Wave 3 Summary Target audience Evidence of impact Method of delivery Practitioner support Learners aged six and seven who are reading and writing at a low

A Concept Note COMMUNITY PARTICIPATION IN EDUCATION Community participation An emphasis on participation has links with the interest in democracy in community organization and in self-help and political

Four Simple Ways to Improve New York City School Data A flood of data is re-shaping American public education, nowhere more than in New York City. With 1.1 million public school students and a decade-long

READING MATERIAL Advanced Securities Law UNIT 2 Public Issues: Initial Public Offering- II ADVANCED SECURITIES LAW 2 In the previous Unit we began our study of initial public offers ( IPOs ). We looked

Introduce diagnostic teaching Alan Bell and the Toolkit team Summary Many students who appear to understand a topic at the end of the teaching unit, do not retain that even for a few months. Better long-term

INTRODUCTION TO HOSPITALITY Course Overview and Syllabus COURSE DESCRIPTION This introductory course provides an overview of the hospitality and tourism industry, its growth and development, industry segments

11 th Improving Student Learning Symposium, 2003 Measuring the response of students to assessment: the Assessment Experience Questionnaire Graham Gibbs and Claire Simpson, Open University Abstract A review

Bring on the Data: Two New Data Tools from Strive Carly Rospert and Geoff Zimmerman 1 Community Impact Report Card The Community Impact Report Card is an online, web-based business analytics application

Migrant Education Program Evaluation Toolkit A Tool for State Migrant Directors Summer 2012 Developed by the U.S. Department of Education Office of Migrant Education through a contract with The SERVE Center

Australia (South) Open Access College Middle Years Program This school aims to provide those who are not able to attend regular schooling with the opportunity to continue their education in the Open Access

GUIDE TO THE EDUCATION SYSTEM IN THE UNITED STATES BY ANTONELLA CORSI-BUNKER ELEMENTARY AND SECONDARY SCHOOL LOGISTICS Gaining Admission to Public Schools Foreign national students accompanying parents

TRAINING CATALOGUE ON IMPACT INSURANCE Building practitioner skills in providing valuable and viable insurance products 2016 List of training courses Introduction to microinsurance and its business case...

Assessment of children s educational achievements in early childhood education purpose pedagogic and managerial of such assessments. This paper outlines the evaluation of a national pilot baseline assessment

A commitment from The Children s Plan Your child, your schools, our future: building a 21st century schools system SUMMARY Building a 21st century schools system Summary 1 Summary Chapter 1 Our ambition

A study conducted by the Center for Adult Learning Leadership A member of the Adult Education Service Center Network Major Advantages and Disadvantages of Open, Fixed and Managed Enrollment According to

Interview with Hugh McLaughlin and Helen Scholar, editors of Social Work Education: The International Journal As new editors of Social Work Education: The International Journal tell us a bit about yourselves

TECHNICAL REPORT #33: Exploring the Use of Early Numeracy Indicators for Progress Monitoring: 2008-2009 Jeannette Olson and Anne Foegen RIPM Year 6: 2008 2009 Dates of Study: September 2008 May 2009 September

CALIFORNIA S NEW EDUCATION FUNDING FORMULA: What is it? Who benefits? What does it mean for students? How can I get involved? In July 2013, California dramatically reformed the way we fund our schools.

2013 RESEARCH IN REVIEW Mathematica informs evidence-based decision making through rigorous studies and analyses. Mathematica s Research in Review is a snapshot of some of this year s research results.

Alliance Scorecarding and Performance Management at TechCo A Vantage Partners Case Study With the assistance of Vantage Partners, TechCo, a microelectronics company based in California, developed and implemented

Report of External Evaluation and Review Bethlehem International School Limited trading as Bay of Plenty English Language School Confident in educational performance Confident in capability in self-assessment

The Importance of Failing Well Lance G King www.taolearn.com A study of intellectually gifted students at a New Zealand high school has revealed one significant factor that distinguishes the highest achievers

INCREASING COMPLETE IMMUNIZATION IN RURAL UTTAR PRADESH The Government of India has recommended that a child must be vaccinated against six vaccine-preventable diseases (polio, tuberculosis [TB], diphtheria,