'Cutting funding for early childhood education in order to meet the current budget shortfall is probably not a great idea'

By Dylan Matthews

Raj Chetty is a professor of economics at Harvard University. His most recent paper, "How Does Your Kindergarten Classroom Affect Your Earnings?" (PDF), co-authored with John Friedman, Nathaniel Hilger, Emmanuel Saez, Diane Schanzenbach and Danny Yagan, examines Project STAR, a landmark education experiment conducted in the 1980s in Tennessee, which randomly assigned students to classes of different sizes and to different teachers for kindergarten through third grade. Past studies have focused on the effect of teachers and lower class size on test scores; Chetty et al stand out for focusing on adult outcomes, such as earnings. Read David Leonhardt's column on the study for more context. A lightly edited transcript of our conversation follows.

Do you want to describe what the initial experiment in Tennessee was, in the ’80s?

Sure. So what we studied in this paper is a famous education experiment called Project STAR, which randomly assigned roughly 12,000 students in Tennessee to different classrooms from grades kindergarten to third grade. And because of random assignment, you can interpret any of the effects that you see of these classrooms on later outcomes, like test scores or adult earnings, as the causal effect of being assigned to that classroom. So, for instance, some students were placed in small classes, some were placed in larger classes, some students had more experienced teachers, some had less experienced teachers, and we can evaluate the causal effect of all of these features of the early childhood education experience on later outcomes using this experiment.

One of the most interesting things about your results is some of the findings on teacher quality. As you laid out, most of the experimental design seems focused on classroom size, but you found some significant effects of the quality of teachers involved as well.

The experimental design actually allows you to evaluate both the effects of class size and the effects of teachers, because students were randomly assigned to classrooms within a school. So the way it worked is, your child shows up at a school, and there's seven different classes, let's say classrooms, and your name is picked out of a hat and you're randomly assigned to one of the seven classes. So seven classes of course have different teachers, and they're also different sizes, so you can evaluate the effects of all of these things. And what we find is that if you, for instance, have a more experienced teacher, that is, a teacher who's taught for more years, you are likely to be earning more as an adult than a student who was randomly assigned to a novice teacher.

One of the limitations of the study is that we don't have that much data on teacher characteristics, so that's why I emphasize experience. The idea is not that experience is the only thing that matters, it's just that we can't measure that many other aspects of what makes a good teacher. So for instance a teacher who relates well to students, or is a good communicator, or is compassionate -- we don't have data on those measures.

So the way we try to capture all that is using a summary measure of the quality of a teacher, which is your peers' test scores. So the idea here is if everybody else in your class is learning a lot from the teacher, they are going to have high test scores, so what we look at is whether, if you are randomly assigned to a class where all the other students are doing well on tests and getting high test scores, do you do better as an adult? And that's where we find really powerful effects. If you're randomly assigned to a group where the peers are scoring high on tests, you're earning more as an adult, you're more likely to go to college, you attend a better college, you're more likely to be saving for retirement, you're more likely to own a house, etc.

And you have found similar benefits to being in small classrooms.

That's right. Being in a small class also yields a variety of improvements on outcomes.

One thing I found interesting about the effect of test scores as relates to earnings is that it seems like some of the gains you find in early childhood, that might not show up on later test scores, later emerge when you're looking at earnings data.

RC: Yes, and that's in fact, I think, the most striking finding and perhaps the most surprising finding from the study, which is if you had just, past education studies just look at test scores, and you see as you had mentioned the effects of these early childhood interventions, like being in a better kindergarten class, they fade out over time. So kids who had better teachers and were in smaller classes in kindergarten aren't doing all that better, really, on tests in middle school and high school. But what's surprising is that those effects reemerge in adulthood. And I can talk about why we think that is.

Why do you think that is?

One explanation for this fadeout and then reemergence of the impact of kindergarten is through non-cognitive channels. So we find, for a limited subset of the students we have measures of non-cognitive ability in eighth grade. So what that means is measures like, they ask teachers to evaluate whether the students are being disruptive in class, whether the students are putting in a lot of effort, whether they're motivated and so on. Now, we find persistent effects of your kindergarten class on these non-cognitive measures. There's no fadeout, or very little fadeout on the non-cognitive stuff.

So one potential explanation of all of the findings together is, a good kindergarten teacher teaches you the material that you're tested on in kindergarten, and so you do well on kindergarten tests. That same good teacher also imparts non-cognitive skills, like they teach you how to be a disciplined learner, how to put in a lot of effort, how to be patient. And those other, non-cognitive skills aren't really directly tested in later cognitive tests, like the kinds of tests that are administered in fourth grade and eighth grade. But they get picked up in these non-cognitive tests in eighth grade, and then it's quite intuitive that these non-cognitive skills matter when you're an adult. It helps to get a good job and to do well in general if you're a disciplined person, if you're perseverant and so on.

And there are some recent studies to back that up. James Heckman's latest kick is about how that's more important than IQ.

Exactly. It's very consistent with what Jim Heckman and others have been arguing about what early childhood education does.

I wanted to ask about one specific number in the study that got some pickup and I want to be sure that we're interpreting it correctly, and that's the $320,000 value for a teacher. That doesn't mean, if you paid people this much, these qualities would emerge.

Absolutely not. That's a measure of the value generated by an excellent teacher, like a 99th percentile teacher, instead of a 50th, average teacher. It is not a measure of what salary we should be paying people, because it's not clear that paying teachers that salary would actually get better teachers. It might, but it also might not. So there's no correspondent, necessarily, between the value that the teacher generates and the right salary for teachers. What we're saying here is just that teachers just have very important impacts on long-term outcomes.

Just so we're clear on how that number is computed, what we do is we say, suppose you're randomly assigned to a teacher that's at the 99th percentile instead of the 50th percentile, as a child you earn on average $1,000 more per year. If you add that up over the course of the child's working life and adjust for interest rates and discounting, you get a total present-value gain of about $16,000 per child. There are about 20 kids in each class, so multiply $16,000 by 20 and you get $320,000.

One of the interesting aspects of the study and the STAR experiment in general is there is no control for people who had no early childhood education period. Obviously, that would be a horrible control group to put people in, but there is a paucity of options in a lot of parts of the country. Is there anything in the literature, or anything even within the study, suggesting the benefits of early childhood education period, as opposed to the specific nature of the education?

This is not a study you can use to answer that question because the experimental design is about once you're in school, what effects does it have. Although the study is titled, "How Does Your Kindergarten Classroom Affect Your Earnings?", what it's really about is random assignment to classes in grades K-3, which is what the study did, and we find similar effects in grades one, two and three. So everybody has access to schools, right, you're supposed to be in school from grades one through three.

And what we're saying is that it matters what school you go to. It matters a great deal, actually. And I think one of the policy implications, relating to your point about a paucity of options in some places, is that it's, I think, quite problematic that the way we finance schools in the U.S. directly leaves more disadvantaged populations with worse schools because it's property-tax finance, so that you have larger classes, lower-quality teaching, just less resources in general in lower-income areas. And I think the results from this study show that that perpetuates inequality.

It's interesting that you bring that up, because the debate within education policy seems to be between people who focus more on accountability measures to make sure that teachers are performing well and people who focus more on ensuring that you have adequate funding. Obviously, it's much more complex than that, but are there any implications you can draw from the study, besides the inequity of the property tax system, in terms of that?

I mean, the study is more limited in terms of what we can say. I think the $1 million question at this point is, "Who are these very good teachers who are generating $320,000 of extra value? And how can we get more of them? And how can we make better kindergarten classrooms?" As I said, we don't have a great deal of data on the characteristics of these great teachers, so we can't really answer that question definitively.

You may think trying to recruit better teachers and only keeping teachers who are doing well could be a good idea, but it also might not be a good idea because then you start getting teachers teaching to the test, so we don't want to take a strong stance on those issues. We're trying to conduct some follow-up research that gets more directly at these issues by getting more data on teacher characteristics in a separate study.

I obviously don't want you to say anything that's going outside the realm of the study or that's going too far, but this seems to be dense in policy implications, and are there any takeaways we can take from this without follow-up or are you sort of waiting and seeing at this point?

I don't feel comfortable making direct policy prescriptions about how we should recruit teachers or what we should do about class size necessarily. I think the broad picture, the broad message here is, things like cutting funding for early childhood education in order to meet the current budget shortfall is probably not a great idea, because we're going to pay the price in the long run, even though it might be a convenient way to close the budget gap in the short run, and that's what many states are doing. So I think the message is more broadly, the period of early childhood education, even though it might look like the effects kind of disappear in terms of test scores, they matter actually, quite a bit in the long run. In terms of resource allocation, at a broad level, we need to really keep that in mind for policy.

Dylan Matthews is a student at Harvard and a researcher at The Washington Post.

The STAR project didn't control for how students were assigned to schools. The income of someone's parents is more highly correlated with their future earnings than their kindergarten teacher. Local funding of schools, and forced attendance at ones local school, regardless of ability, means that more experienced teachers, which unions often cause to have high wages, regardless of ability, seek employment at higher paying schools. We thus have a reversal, where students with higher potential earnings are sought by higher paid teachers.

This ridiculous data mining result is probably completely bogus, but people that choose seek confirmation for their beliefs, rather than truth are drawn to it. Without data showing that teacher experience and student wealth were evenly distributed across schools, this result is worse than useless. I predict more than $320,000 will be spent in attempts to validate its conclusions. Yet, very little will be done to actually improve education.

I believe that the random assignment of students is itself a painful problem. We need differentiated education, letting everyone progress go as fast and far as they can.

"Without data showing that teacher experience and student wealth were evenly distributed across schools, this result is worse than useless. I predict more than $320,000 will be spent in attempts to validate its conclusions. Yet, very little will be done to actually improve education."

Wait - students were randomly assigned to classes _within_ schools, not between different schools. So it would be fairly simple to control for family income level if it were part of the data set. All they are claiming is that students that were randomly assigned to certain classrooms eventually earned on average $16k more than kids randomly assigned to the average classroom. If all of those kids came from wealthy households then you'd have a skewed sample, which any peer reviewed researcher would toss out of the study.

I haven't read the paper whose results are being discussed, and there are very often really terrible statistics in papers analyzing education, but it is not AT ALL self evidently true that staticvars' criticisms are correct.

In fact, it wouldn't be all that hard, actually, to do some gross control for what staticvars points to as making the results reported here illegitimate with fairly simple data on mean family income, funding per student, mean education level, and similar types of data on a per school basis. Those types of information should do a decent job of controlling for the fact that some schools have students who are poorer and have less educated parents, as would a basic HLM analysis.

The more troubling point is that better teachers might be drawn to schools with students already at an advantage. If there is solid school level data on parental income and education data, however, this still is not necessarily a problem that would make a legitimate statistical analysis impossible.

I think an even bigger problem that this is the way teacher quality was scored. Average peer test score is a terrible way to do this. Average peer pre-test scores seems like it is much more likely to measure peer effects than teacher effects, and peer effects are known to be very strong...

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.