Drawing upon decades of experience, RAND provides research services, systematic analysis, and innovative thinking to a global clientele that includes government agencies, foundations, and private-sector firms.

The Pardee RAND Graduate School (PRGS.edu) is the largest public policy Ph.D. program in the nation and the only program based at an independent public policy research organization—the RAND Corporation.

Download eBook for Free

Research Questions

To what extent does content expertise of the rater influence scores on TNTP Core?

To what extent are teacher observation scores (overall and on each of the four rubric domains) valid predictors of the effectiveness of math and ELA teachers, as captured by their estimated contributions to student performance (or value-added scores)?

Nearly every public school teacher in the United States receives some form of feedback based on classroom observations, and these assessments are used by policymakers and administrators as indicators of teacher quality and input for professional development purposes. The TNTP Core Teaching Rubric, introduced in 2014, is unlike others in that it is scored based on student behavior rather than teacher actions. In this report, RAND researchers assess whether the TNTP Core Teaching Rubric produces scores that are representative of teachers' overall instructional practices and whether raters' content expertise influence scores on TNTP Core.

For this study, volunteer teachers of mathematics and English language arts (ELA) from more than 20 districts or charter school networks across eight states had their classroom instruction videotaped three times. This sample was complemented with a sample of approximately 100 4th-and 5th-grade teachers drawn from the Measures of Effective Teaching study for whom videotaped instruction was available. Raters trained in the TNTP Core Teaching Rubric then scored the videos from both samples. Analyzing the raters' scores, RAND researchers found a modest relationship between teachers' TNTP Core scores and student achievement gains in ELA, and no statistically or practically significant relationships between TNTP Core scores and math achievement gains. Further, raters struggled to agree on their judgments about the quality of a lesson, suggesting a considerable amount of uncertainty about the extent to which TNTP Core scores represent teachers' overall instructional practices. The report includes a number of recommendations to improve the use of TNTP Core.

Key Findings

Observers often disagree in their ratings of instructional practice based on the TNTP Core rubric, and these disagreements may be related to their content expertise.

There is a considerable amount of uncertainty about the extent to which TNTP Core scores represent teachers' overall instructional practices.

Recommendations

Consider setting higher standards for rater certification (specifically by requiring raters in training to have their scores align with those of master raters at a higher frequency).

Research has shown that raters tend to change their approach to scoring over time — a phenomenon known as rater drift. To mitigate this, consider using more frequent post-certification calibration and validation exercises during a rating period.

Keep in mind that high-quality evaluation and feedback require many observers with different backgrounds to rate many lessons.

Consider collecting additional sources of evidence that support claims about the quality of teacher practice, i.e., using TNTP Core as one measure in a multiple-measure system.

Research conducted by

The research described in this report was prepared for TNTP and conducted by RAND Education.

This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.

The RAND Corporation is a research organization that develops solutions to public policy challenges to help make communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit, nonpartisan, and committed to the public interest.