The Raj Chetty VAM Balloon Gets Popped

BOULDER, CO (April 10, 2014) – A
highly influential but non-peer-reviewed report on teacher impact suffers from
a series of errors in methodology and calculations, according to a new review
published today.

Professor Moshe Adler reviewed two
recent reports released in September 2013 as National Bureau of Economic
Research working papers. Dr. Adler’s review for the Think Twice think tank
review project is published today by the National Education Policy Center,
housed at the University of Colorado Boulder School of Education.

Adler is an economist affiliated
with both Columbia’s Urban Planning Department as well as the Harry Van Arsdale
Jr. Center for Labor Studies at Empire State College, SUNY, and the author of Economics
for the Rest of Us: Debunking the Science That Makes Life Dismal (New
Press, 2010).

Adler reviewed Measuring the
Impact of Teachers, parts I and II, written by Raj Chetty, John N.
Friedman, and Jonah E. Rockoff. Part I is subtitled, Evaluating
Bias in Teacher Value-Added Estimates, and Part II, Teacher Value-Added
and Student Outcomes in Adulthood. Taken together, the two-part report
asserts that students whose teachers have higher value-added scores achieve
greater economic success later in life.

The documents (as is standard for
NBER working papers) were not peer-reviewed, yet as Adler points out, the
research on which they were based has gained extraordinary attention – turning
up as references in President Obama’s 2012 State of the Union address, in
expert court testimony by the principal author (Chetty), in extensive news coverage, and even as a
justification for Chetty’s 2012 MacArthur Foundation “genius” award.

That sort of credibility, Adler
suggests in his review, may not be warranted – as demonstrated by a series of
problems that he finds with the new two-part report and the research that
undergirds it.

The report’s own results reveal
that calculating teacher value-added is unreliable, Adler writes. Additionally,
the report includes a result that contradicts the central claim; it relies on
an erroneous calculation to support a favorable result; and it assumes that the
miscalculated result holds across students’ lifetimes – “despite the authors’
own research indicating otherwise,” the reviewer notes.

Finally, Adler explains, the
studies relied on by the report as support for its methodology don’t actually
provide that support.

“Despite widespread references to
this study in policy circles, the shortcomings and shaky extrapolations make
this report misleading and unreliable for determining educational policy,”
Adler concludes.