Turnaround Strategy Evaluation Overview

Thanks to a generous grant from the Walton Family Foundation, the School Turnaround Strategy will be evaluated over a three year period. The external evaluation will assess implementation, impact on participating student and adult behaviors, and the overall value of the strategy on the district. APS will use the evaluation findings to inform ongoing efforts to improve school and student outcomes and to contribute to the national conversation on school turnaround and student support.

Evaluation Goals

Assess implementation and impact of student-focused initiatives on academic and behavior outcomes, and the impact of professional learning opportunities on school climate and school performance.

Assess progress of the implementation of changes in operating models and impact of partnering with non-profit organization managers on student, school and community outcomes.

Assess APS’ approach to implementing the Turnaround Strategy, particularly its selection of community partners and organizational/staffing efforts in supporting targeted schools.

Evaluator Overview and Scope of Work

Mathematica Policy Research was selected to serve as the external evaluator for the APS School Turnaround Strategy. Mathematica’s evaluation approach aims to provide rich, descriptive information about implementation and rigorous causal evidence about the overall Turnaround Strategy and the key components of the Strategy. In addition to project management, the scope of work includes three key tasks:

Implementation Study

The implementation study will be guided by four main objectives, which will provide both formative and summative information:(1) describe how the APS Turnaround Strategy is designed, structured, and operationalized to achieve outcomes; (2) identify adaptations to the design and activities that occur during implementation for rapid-cycle formative feedback to APS; (3) obtain an understanding of educators’ experiences to provide APS with information to inform implementation support; and (4) assess the extent to which schools develop and deliver well-defined Strategy components with fidelity.

Impact Study

The impact study will have three main objectives: (1) provide rigorous, timely estimates of the combined impacts of the Turnaround Strategy on school-level outcomes; (2) evaluate the contribution of individual components of the Strategy, such as partnerships or Vacation Academies, so results can be used in continuous program improvement; and (3) assess the cost-effectiveness of the Strategy and its components.

Reporting and Dissemination

What does this mean? Over the course of the three-year study, Mathematica will conduct observations of key interventions as well as conduct interviews and focus groups with APS staff and stakeholders. The district project team will coordinate with Mathematica to contact schools and schedule all activities to eliminate/mediate any burden to school staff.