Nij

NIJESH SOLK MWIL START GYAMYATH (Save us from the time of trial) From Austerity to Prosperity and back again ESAI CONFERENCE 2013 M.Brown, G.McNamara, J.O’Hara Centre for Educational Evaluation (CEE), DCU

FOCUS OF THE STUDY This study examined the changing landscape of evaluation policy and practice in the ROI and NI since the inception of the Stanley letter 1831 where evaluation in the form of school inspection and self evaluation aims to promote and by turn of phrase improve the quality of education provided in schools. From this, a framework for the mutual terms of co-existence between IE and EE is proposed. Theoretical Assumption ‘Both systems should exist because we need both and because they might even benefit from each other’ (Nevo p.6, 2002) Centre for Educational Evaluation (CEE), DCU

OUTLINE OF THE STUDY: MAY 2009 - MARCH 2013 Phase 1 :- Tentative Schemata Review of the literature on the rise of evaluation in education and the different international models of evaluation that exist, coupled with an analysis of principal and inspector interviews in the 19th and 20th century. Phase 2: - Quantitative Analysis All Island survey of school Principals perceptions of internal/external evaluation in the Republic of Ireland and Northern Ireland. Analysis was carried out using parametric and non parametric techniques. Phase 3: - Qualitative Analysis Interviews with 46 Principals and Inspectors from both regions. A mixed methods way of thinking rests on assumptions that there are multiple legitimate approaches to social inquiry and that any given approach to social inquiry is inevitably partial (Greene, 2007 p.20). Phase 4: - Final analysis (March 2013) Proposed Framework: for the mutual terms of co-existence between Internal and External Evaluation. Centre for Educational Evaluation (CEE), DCU

VALUE ADDED ASSESSMENT (THE GLOBALISED FUTURE) OECD REVIEW OF EVALUATION AND ASSESSMENT (Australia) The development of “value-added” models represents significant progress as they are designed to control for the individual student’s previous results, and therefore have the potential to identify the contribution an individual teacher made to a student’s achievement. (OECD p.96, 2011) OECD REVIEW OF EVALUATION AND ASSESSMENT (Portugal) In value-added models, students’ actual test scores are often compared to the projected scores, and classroom and school scores that exceed the projected values are considered as positive evidence of instructional effectiveness. (OECD p.132, 2012) OECD REVIEW OF EVALUATION AND ASSESSMENT Belgium (Flemish Community), Chile, Czech Republic, Denmark, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Portugal, Slovak Republic, Sweden and the United Kingdom (Northern Ireland). Centre for Educational Evaluation (CEE), DCU

VALUE ADDED ASSESSMENT THE IRISH FUTURE JUNIOR CYCLE FRAMEWORK The DES will provide each school with a Data Profile…The Data Profile will also provide schools with information on their patterns of achievement relative to schools with a similar school context…These data will help schools to refine their assessment and moderation practice. They will also be a valuable source of information for schools’ self-evaluation processes…In the event of an unusual pattern of achievement, the Inspectorate of the DES will be advised, and support and evaluation measures will be provided for the school (DESROI, 2012a, p.27). Centre for Educational Evaluation (CEE), DCU

SELF EVALUATION: THE PRESENT Over a four-year period from 2012, all post-primary schools should engage in school self-evaluation and produce three-year improvement plans for numeracy, literacy and one aspect of teaching and learning across all subjects and programmes (Department of Education and Skills 2012, p.2). The performative culture is so deeply ingrained in schools and education systems that I can foresee a game of permanent artifice, where schools squeeze their individual circumstances into a self-evaluation document designed solely to impress inspectors, and hold themselves in a state of perpetual readiness to live up to their claims, the model prisoner. In this context, ‘bleak indeed is the desire for perfection’ (Marshall 1999, 310)’ (Perryman 2009, p.629) Centre for Educational Evaluation (CEE), DCU

FRAMEWORK FOR THE CO-EXISTENCE OF INTERNAL AND EXTERNAL EVALUATION AT A CONCEPTUAL LEVEL Evaluation needs to be perceived as a means of understanding rather than judgement’ (Nevo 2002, p.10). AT A METHODOLOGICAL LEVEL Evaluation should be based on the continuous dissemination of information, enabling mutual learning. However, Ryan et al (2007, p.208) ask ‘what should we expect from novice, schoolbased evaluators? Should we have the same standards and expectations for the school teams (i.e., internal evaluators) as we do for evaluators conducting external evaluations?’ AT A COMMUNICATION LEVEL There needs to be mutual respect and trust between internal and external evaluators. However, within the social fabric of many regions, such as the ROI, where the economic downturn has led to cut backs in education, it would be reasonable to suggest that very little trust in government policy and practice in general actually exists. Centre for Educational Evaluation (CEE), DCU

FRAMEWORK FOR THE CO-EXISTENCE OF EVALUATION AT AN INFLUENTIAL LEVEL In the past, many Evaluation systems tended towards, Reward, Coercive and Legitimate power. Table 2.3:Ravens (1965) typology for the six bases of power Positional vs transformational bases of power POWER BASE POSITIONAL DEPENDENT ON CHANGE IN TARGETS INTERNAL BELIEFS REQUIRES SURVEILLANCE Reward Yes No Yes Coercive Yes No Yes Legitimate power Yes No Yes Referent power No Yes No Expert power No Yes No Informational power No Yes No However, in order for internal and external evaluation to mutually and beneficilally co-exist: Evaluation frameworks should centre on the Referent, Expert and Informational aspect of Evaluation. Centre for Educational Evaluation (CEE), DCU

FRAMEWORK FOR THE CO-EXISTENCE OF INTERNAL AND EXTERNAL EVALUATION USE OF THE INFLUENTIAL ASPECT OF EVALUATION During my time as Principal the emphasis was on schools evaluating themselves with the Inspector as the "critical friend“ [Expert Power]. For the approach to be successful it required a good working relationship between the Principal and the District Inspector… [Referent Power]. By meeting frequently we grew to trust and respect one another with the result that I never felt threatened or worried when [Dr.Shevlin] (DI) would ask probing questions… With [Dr. Shevlins] support we tried new approaches to things like literacy, discipline, behaviour and then wrote up our policies [Informational Power]. It was this professional relationship between the Inspector and the Principal which greatly assisted school improvement in the case of St Colm's… (Hansard Archive, Northern Ireland Assembly, 2011) Evaluator Influence (St. Colm’s High School Belfast) Centre for Educational Evaluation (CEE), DCU

FRAMEWORK FOR THE CO-EXISTENCE OF INTERNAL AND EXTERNAL EVALUATION AT A CULTURAL RESPONSIVE LEVEL I can find no logical explanation as to why our evaluations should not be culturally responsive or that we should not behave in culturally responsible ways in our work as evaluators. (Hood, 2001) Indeed: Evaluation efforts have typically failed to consider cultural background and context in their design, implementation, analyses, and recommendations. However: It is not possible to effectively derive evaluative meaning from educational programs, designed to serve culturally diverse students, unless the evaluator themselves are more culturally responsive. (Hood, 2001) Centre for Educational Evaluation (CEE), DCU

CONCLUSION EVALUATION Those of us who are proponents of external evaluation should find ways to empower schools and teachers to participate as equal partners in the evaluation process and make use of it; and those of us who believe in internal evaluation as a means for school autonomy and teacher professionalisation must admit the legitimacy of accountability and the right of the public to know. They, in their turn, should seek external evaluation as a partner for dialog rather than an object for rejection. (Nevo 2010, p.784) ASSESSMENT Every time you use a measure of pupil attainment for some extraneous purpose you risk creating a perverse incentive. So every time you find yourself doing that or participating in a system that requires you to do that; ask what incentive is this creating? Whom is it damaging? Pupils, head teachers, schools, employers and I think that would be a very useful first step (O’Neill, 2011). Centre for Educational Evaluation (CEE), DCU