This critique was originally published for the Schools of Education (SOE) Knowledge Community in February 2010. At the time, the National Council on Teacher Quality (NCTQ) was poised to launch its first National Review of Teacher Preparation after several years of conducting reviews of individual states. Eduventures’ research found a number of methodological flaws with NCTQ’s research methods that call into question their reported findings.

Fast forward to 2013 and NCTQ has partnered with US News and World Report in order to publish the 2013 Review. In anticipation of the release, Eduventures reached out to both NCTQ and US News in an attempt to understand any methodological changes that were made to increase the validity and reliability of their results for the forthcoming report. US News deferred comment to NCTQ, and NCTQ declined to comment.

While SOE’s are able to respond to NCTQ’s findings, NCTQ’s program review methodology may still not accurately reflect the quality of educators produced by each teacher education program:

NCTQ evaluates inputs in order to draw conclusions regarding the quality of teacher preparation program outputs. Addressing inputs alone without taking into considerations outputs such as the resulting quality of prepared educators, or the impact of teacher education graduates on P-12 student achievement, does not provide a holistic picture of program quality.

The list of inputs NCTQ used to evaluate teacher preparation programs is incomplete. The NCTQ study does not consider other inputs such as the quality of instruction, student support services, professional development provided, teacher induction programming, and many other factors that are related to the quality of teacher preparation programs.

The majority of NCTQ’s standards are not evidence-based, and appear to reflect the specific viewpoint of NCTQ. In its Guide to Ratings Methodologies, NCTQ frequently couches its rationale with nuances such as “While there is no research basis for this….”, or “While there is no research evidence…” Rather than providing evidence, the rationale NCTQ provides for many standards appears to be opinion-based, and, in some cases, the rationale includes broad generalizations that many experts would recognize as untrue.

It is unclear how well the evidence that NCTQ uses to measure a program’s adherence with each standard actually measures each standard. For example, to rate the overall quality of a given teacher preparation program’s coursework, NCTQ only chose to review a small sample of course syllabi, descriptions, and in limited cases, textbooks. While these materials may capture some of what occurs in a classroom or course, they are a small proxy for an entire program’s coursework and the limitations of this methodology should be recognized. It is questionable whether these materials alone provide a valid, reliable measure for the quality of the broader program’s coursework.

Eduventures agrees that SOEs should be held accountable for preparing high quality teachers. However, accountability should be grounded in research high quality, and include outcomes. We do not see evidence that NCTQ has worked to correct these methodological flaws prior to its launch of its national study in partnership with US News & World Report.

The federal government and state and local policymakers increasingly emphasize outcomes and results in producing highly effective teachers that increase student achievement, turning away from inputs alone as indicators of teacher quality. The quality of outputs and employer ratings are not taken into consideration—an integral piece to consider when judging the quality of preparation program.

Lastly, the lack of clarity and transparency with regards to the common standards that NCTQ is using and the processes and procedures used to analyze the collected data make the methodology of NCTQ’s studies problematic. While its studies provide a high-level look across teacher education program models, their methodology flaws ultimately limit the validity of the study’s conclusions and make the findings somewhat unreliable and distract from the most important goal of improving the quality of teacher preparation in the U.S. As a result, Eduventures analysts recommend that policymakers, the public, and the press should keep these limitations in mind when reviewing the results of NCTQ’s existing and future studies.