Partner Institutions: University of Virginia and District of Columbia Public Schools

Education Issue: Student outcomes in the District of Columbia Public Schools (DCPS) are among the lowest of any school district in the U.S. In Washington, D.C., as in many other districts and states, policymakers have focused on improving educational quality as a way to improve student outcomes. In recent years, a research consensus has coalesced around the notion that teacher quality is an important determinant of student development and achievement. However, there is no similarly wide agreement on how to systematically promote improvements in teaching quality.

In AY 2009-10, DCPS implemented IMPACT: The DCPS Effectiveness Assessment System for School-Based Personnel. IMPACT is a high-stakes teacher evaluation system intended to improve teaching and student achievement. IMPACT was intended to set clear expectations, provide systematic, informed feedback on teaching performance, facilitate collaboration, inform teacher training and support, and retain/reward effective instructional personnel. During IMPACT’s first 3 years, DCPS increased the pay of teachers rated as “highly effective,” while it terminated the employment of teachers rated as “ineffective.”

Partnership significance and goal: Under the proposed project, the partnership of the District of Columbia Public Schools (DCPS), University of Virginia (UVA), and Stanford University will focus on the development and analysis of student, teacher, and school administrative data from DCPS to understand the effects of IMPACT on student achievement and guide its ongoing design refinements. In addition, researchers will work closely with policymakers to understand better the strategic and institutional aspects of IMPACT to insure sufficiently nuanced research designs and interpretations.

Partners and Partnership Activities: Scott Thompson (Deputy Chief, Teacher Effectiveness Strategy) and Jason Kamras (Chief of Human Capital) of the DCPS are collaborating with Professors James Wyckoff (University of Virginia) and Thomas Dee (Stanford University) to assess and improve the DCPS teacher evaluation system. The partnership was initiated in February 2012 with a memorandum of agreement to share data and explore a set of questions. Since then the Partners have designed a set of initial research questions, developed an analytic database to address those questions, and conducted descriptive and causal analysis (using a regression discontinuity design) on aspects of IMPACT. Additional data acquisition and management are planned as are ongoing discussions regarding expanded research questions. Each of the partners is especially interested in human capital policies that improve student achievement.

The Partners will meet every two weeks to discuss the progress of ongoing research and data collection, adjust research based on policy considerations, and plan for future research. Less structured interactions by phone and email will occur as needed. The partnership will assemble a project advisory board that will meet annually with the Partners to review research products and provide advice on future research. The Partners will update the advisory board on a regular basis and be in contact with individual advisory board members as issues arise and as consistent with their expertise. The partnership will provide the opportunity for mentoring of DCPS research staff by UVA faculty. An important shared goal of this collaboration is to increase DCPS research capacity. The partners will explore informal (advising and mentoring) and formal (coursework and collaborative research) mechanisms toward this end.

Setting: This study will be conducted onsite in the District of Columbia and at the University of Virginia in Charlottesville.

Population/Sample: The analytic sample will include all teachers and students in DCPS from 2009-10 (IMPACT’s first year) through 2014-15 (when available).

Initial Analysis: Partners will carry out descriptive analyses and regression discontinuity analyses, specifically to examine the role that substantial IMPACT changes implemented in 2012-13 have played in modifying conclusions drawn from prior analyses. The planned analyses will allow the research team to examine possible differences across time in teacher applicant characteristics to DCPS as a result of IMPACT; to examine teacher mobility within DCPS and between DCPS and DC charters as a potential consequence of IMPACT; and to examine aspects of teaching that are most malleable in the context of strong incentives. Partners will also explore whether findings differ by teacher groups (i.e., instructors of tested grades/subjects versus the remainder who represent the majority of the DCPS teacher population), by different weights applied to evaluation system component scores, by use of the PARCC assessments versus the prior assessments, by teacher uptake of instructional coaching, and by student/community characteristics (e.g., parent’s education, household income, community safety).

Using multiple administrative datasets (e.g., student-level longitudinal data from DCPS and DC charters, longitudinal data from the NAEP- Trial Urban District Assessment (TUDA), and the knowledge of DCPS policymakers and managers, partners will catalog the changes in the design of IMPACT and in the broader DC context and develop measures that attempt to account for the changes. Analyses will also include heuristic robustness checks to probe the empirical relevance of potential confounds (e.g., Do changes in teacher applicants or teacher mobility appear to lead, rather than follow, policy changes in DCPS?).

Partners will examine the psychometric properties of the measures that compose the evaluation system, specifically the reliability of each measure, the inter-correlations among these measures, and the correlations between these measures and student outcomes.