Teachers

Seven questions for Brian Jacob

OUR last two interviews focus on teachers. Today we talk with Brian Jacob, a professor of education policy and economics at the University of Michigan's Gerald Ford School of Public Policy. Mr Jacob is also a faculty research fellow at the National Bureau of Economic Research and an executive committee member of the National Poverty Center. Previously he was a policy analyst in the New York City mayor's office and he also served on the front lines, teaching middle school in East Harlem.

DiA:You've done research into teacher effectiveness and its effect on student achievement. How much of a lasting effect can a stand-out teacher have on his students?

Mr Jacob: Parents, educators and casual observers have always recognised that teachers play an absolutely critical role in student learning. Taking advantage of advances in education data systems that link students to teachers, a number of recent studies have documented considerable variation across teachers in their effectiveness at raising student achievement. This research suggests that students who are lucky enough to have a teacher in the top quartile will gain roughly 10 percentile points more than otherwise identical students who have a teacher in the bottom quartile. This impact is extremely large if one recognises that the black-white achievement gap nationally is roughly 34 percentile points. Moreover, there seems to be substantial variation even across teachers within the same school.

What is more surprising, at least to some, is that the same studies have documented that a teacher's effectiveness at raising student achievement is not correlated with many things that we commonly associate with teacher quality. For example, teachers with Master's degrees are no more effective, on average, than teachers with a BA. Teachers who are traditionally certified are no more effective than alternatively certified (and, in some cases, uncertified teachers). First- and second-year teachers are somewhat less effective than their peers, but subsequent experience is not strongly linked with effectiveness. That is, a tenth-year teacher is no more effective than a 20th-year veteran, on average.

And these results do not consider the myriad other ways in which a good teacher can influence a child, including fostering a love of learning and teaching them study skills that will benefit them for years to come.

DiA:You've also done a lot of research on teacher labour markets, and more specifically the effect of employment protection on teacher effort. Steven Brill wrote a fascinating article in the New Yorker last year on how difficult it is to fire a teacher in New York City. Is this the case all over the country? And what effect do these policies have on teacher effort?

Mr Jacob: In all large urban school districts across the country, it is extremely difficult for principals to fire a teacher. It is likely somewhat easier in smaller districts. In practice, however, a remarkably small number of teachers are dismissed for cause in any given year. Similarly, principal evaluations of teachers tend to be extraordinarily generous, in part due to the fact that the evaluations have little affect on a teacher's promotion or compensation. In the Chicago Public Schools in 2007, for example, only 15 out of the 11,621 teachers who were evaluated received a rating of unsatisfactory, and only 641 (roughly 5.5%) received a rating of satisfactory. The remaining teachers were rated excellent or superior.

Critics of teacher employment protections argue that it not only prevents school leaders from removing "bad apples", but also reduces effort among all teachers. Unfortunately, there has been little concrete evidence on this point.

In some recent work, I take advantage of a unique policy change to examine how the reduction of employment protection for teachers impacts teacher effort and student achievement. In 2004, the Chicago Public Schools (CPS) and Chicago Teachers Union (CTU) signed a new collective bargaining agreement that gave principals the flexibility to dismiss probationary teachers for any reason, and without the documentation and hearing process that is typically required for teacher dismissals. In practice, this dramatically reduced the costs associated with firing young teachers in the district.

In the first year of the policy, the 2004-05 school year, over 1200 teachers—roughly 12% of all probationary teachers—were dismissed.

To see how this policy influenced one measure of teacher effort, I compare changes in teacher absences before and after the policy for probationary versus tenured teachers. The results suggest that the policy reduced annual teacher absences by roughly 10% and reduced the prevalence of teachers with 15 or more annual absences by 20%. The effects were strongest among teachers in elementary schools and in low-achieving, predominantly African-American high schools, and among teachers with high-predicted absences.

In additional analyses, I find some evidence that the policy increased student achievement at the elementary level. My results suggest that the reduction in teacher absenteeism likely explains only a small part of the achievement effects, suggesting that the policy may have impacted teacher behaviour in many ways.

DiA:Many education reformers support policies that incentivise good teaching by way of bonuses or performance pay. But this only works if we come up with a fair and accurate way to evaluate teachers. How do you propose we evaluate them?

Mr Jacob: I firmly believe that teachers should be evaluated on the basis of how they influence student performance. However, I do not believe that teacher evaluation need or should be limited to student achievement on standardised tests. Such “value-added” measures of teacher effectiveness should be a part of teacher evaluation, but not the only part. I believe that a supervisor evaluation—which could be conducted by the principal, the department head, mentor teacher or other person—should be an important component of a teacher's evaluation. And, finally, I think that a piece of the teacher's own personal evaluation should be based on the performance of a broader set of students in the school, whether it be the entire school, the teacher's grade-level group, the teacher's subject group or something else. This final component will mitigate any incentive teachers might have to eschew collaboration with colleagues and will promote collective school-level approaches to reform.

A few other components are key to a compensation/promotion system.

First, rewards should be allocated on a basis of absolute performance—that is, every teacher in a school should have an opportunity to “win” rather than reserving bonuses for the top 5% or 10% or 25% of teachers.

Second, any type of compensation system should focus on broad categorisations of teachers, and not attempt to finely distinguish between every teacher in a school. In my research, I have found that a principal's holistic, “subjective” evaluation of a teacher is highly correlated with an “objective” value-added measure of teacher effectiveness for the top and bottom teachers. That is, principals can very clearly identify the most and least effective teachers in their building—roughly, the top 15% and bottom 15%. In contrast, principal evaluations are not correlated with achievement-based measures of effectiveness for the middle 70% or so of teachers. Similarly, the precision of most achievement-based measures of teacher effectiveness do not allow one to reliably distinguish between, for example, a teacher at the 45th percentile and a teacher at the 55th percentile. For this reason, I believe that any teacher evaluation system should focus on grouping teachers into 3 to 5 categories.

Finally, I think a teacher evaluation system should be embedded in a broader career ladder system for teachers that ties changes in pay and job security to changes in job responsibilities. A number of districts and states have experimented with career ladder systems, but there has not been any rigorous evaluation of such systems relative to a traditional system or relative to each other. This is an area that could definitely benefit from experimentation and evaluation.

Mr Jacob: In Chicago, yes. At least as far as one can tell by looking at a few key proxies for teacher effectiveness. As part of the study described above, I matched information on all teachers who were eligible for dismissal with records indicating which teachers were dismissed. By comparing the characteristics of dismissed versus non-dismissed probationary teachers within the same school, I estimate the relative weight that school administrators place on a variety of teacher characteristics, including proxies for teacher productivity.

I find strong evidence that principals do consider teacher productivity in determining which teachers to dismiss. Principals are significantly more likely to dismiss teachers who are frequently absent and who have received worse evaluations in the past. Moreover, elementary teachers whose students had made smaller achievement gains in the past were more likely to be dismissed than their peers. In addition, principals were significantly less likely to dismiss teachers with stronger educational qualifications as measured by things such as the competitiveness of their undergraduate college, whether they ever failed the teacher certification exam and whether they had a Master's degree.

These results, coupled with the findings presented above, suggest that reforms along the lines of the Chicago policy might improve student achievement.

At the same time, this analysis reveals that roughly 35% of principals—including those in some of the worst performing schools in the district—did not dismiss any teachers despite this new flexibility. The apparent reluctance of many Chicago principals to utilise the additional flexibility granted under the new contract may indicate that issues such as teacher supply and/or social norms governing employment relations are more important factors than policymakers have realised.

DiA:Has No Child Left Behind done anything to improve the quality of teachers in America's schools?

Mr Jacob: This is a difficult question to answer. NCLB required states to ensure each classroom was staffed with a “highly qualified” teacher. To be deemed highly qualified, a teacher was required to have a BA degree, be fully certified and demonstrate subject-matter knowledge. However, beyond these basic requirements states were given considerably flexibility in implementing this provision, including how the new standards would be applied to existing teachers. For the most part, states allowed existing teachers to easily meet the new requirement. So, I do not think that this provision had much impact on the quality of existing teachers.

As for new teachers, there is some evidence that NCLB led districts to expand alternative certification routes, which likely raised teacher quality. In New York City, for example, researchers have documented that highly educated recruits from Teach for American and the Teaching Fellows replaced uncertified teachers since the introduction of NCLB.

In addition, the accountability provisions of NCLB may have influenced teacher quality indirectly by affecting personnel decisions within schools. For example, a principal facing the pressure to meet AYP may have moved the more effective teachers into the classroom (from administrative positions) or mandated additional professional development for existing staff. Unfortunately, there is no systematic evidence on the prevalence or nature of these effects.

DiA:Will Barack Obama's changes to NCLB do anything to improve the quality of teachers in America's schools?

Mr Jacob: I think there are a number of promising features in the administration's proposal for reauthorisation. I like the idea of focusing on teachers and teacher quality. I like the idea of focusing limited state resources on a set of perpetually failing schools, and then taking stronger action to shut down or “turn around” these schools. And I think the increased support for charter schools is good, as long as it is accompanied by rigorous oversight and one does not assume that these charters alone can “fix” the public schools.

Ultimately, the impact of the changes in NCLB will be determined by how states and districts implement the reforms. For example, NCLB should have provided states and districts with the authority and motivation to shut down perpetually failing schools. However, for a variety of political, financial and other factors, many states and districts did not take the type of direct and decisive action necessary to dramatically improve student learning.

DiA:How would you go about improving the quality of teachers in America's schools?

Mr Jacob: Most importantly, I would recognise that there is no silver bullet. We will need to experiment with and evaluate a variety of different approaches.

First, we need to restructure teacher evaluation and compensation systems. We should deemphasise experience and instead evaluate teachers on the basis of student performance in a way similar to what I outlined above.

Second, teacher compensation should be reconfigured to pay a premium to teachers in hard-to-staff subjects and/or schools. Given the current labour market, it is simply more expensive to hire a teacher for high-school chemistry than general elementary education. Forcing salaries to be equal across fields simply ensures that teachers in hard-to-staff fields will be less qualified than their peers.

Finally, we should invest more to devise and evaluate professional development programmes, which would include both new-teacher mentoring as well as further subject-specific professional development.

Readers' comments

I'm afraid I don't see that FC. It says that teacher quality appears to make up about 10 percentile points. That's obviously huge and a big factor. But it says the black-white gap is 34 points. So teacher quality appears to cover about 1/3rd of this gap.

Obviously that makes it a prime candidate for reform - and is something on which we can obviously focus reform efforts. It may be that the other missing points are parental involvement or other things which aren't really something that can be changed via rules & regulations - but it would still seem we ignore other factors at our peril.

A. D. - agreed, just by eliminating the Teacher Quality disparities we can't eliminate the entire black-white gap. But based on this blog post and other articles I've read it is becoming clear that Teacher Quality is the largest single factor that can be impacted by education policy. Given that we have for decades been spending a whole lot of money on factors that have had little to no impact on education outcomes, it seems obvious (to me at least) that we should start focusing on the elephant in the room.

Put differently:
- which other factors can we impact via policy and get a 10 percentile point improvement?
- which of those factors are more cost-effective in terms of dollars/improvement?
- which other factors can we impact by redirecting existing spending, rather through additional appropriations?

The US already spends more per student than other OECD countries, and we're getting worse outcomes - at least in terms of 12th grade Math & Science results.

Doug makes a good point, but when looking at the totality of Education Reform this an outstanding piece of journalism - thanks R.M. for bringing this issue to the fore.

The massive point not to be missed is that Teacher Quality is key. Not parents, not child IQ, not school district, not charter vs non-charter, not money per student, not testing, or not testing, not native born or not native born, race, skin color, etc. etc. What matters is Teacher Quality. All those other factors are relevant only to the extent that they affect Teacher Quality.

The policy implications are huge. All our policy efforts should henceforth be focused on improving Teacher Quality. Make it so.

I enjoyed this interview but the last answer struck me funny. "I would recognise that there is no silver bullet. We will need to experiment with and evaluate a variety of different approaches." It sounds like Dr. Jacob expect through trial and error to find the silver bullet. I would expect that by evaluation we might get smarter about which approaches will work where and with whom, but I'd also guess there will always be ideas in opposition to one another in education, which look stupid if you're using them and brilliant if you're not.

We have a national teacher shortage, so measures making it easier to fire teachers (while important) won't automatically lead to principals firing bad teachers.

But, as many pointed out, teacher pay is quite low compared to the costs of the bachelor's and master's degrees we want teachers to have, so few choose teaching once they've attended higher education for more than 4 years.

I find it no coincidance that the country with one of the best educational systems in the world, Finland, pays teachers salaries between $60,000 to $80,000.

Put another way, that's how much money an Engineer in the U.S. earns after graduating with a BS or Master's - i.e., that seems to be the salary level that is necessary and sufficient to encourage bright students to stay in high education for 4+ years.

Raise teacher pay in exchange for union cooperation in making teacher dismissals much easier.

Great article. I am a two year second career substitute teacher who previously spent 23 years in finance. What I have seen in over 80 sub assignments at over 20 schools:

1, Classroom management skills are #1. Wong and Wong’s textbook is right.
2. Students need to constantly hear the importance of learning. When a math class hears the starting salaries and job demand for surveyors (or any other high need area) you have their attention in five minutes.
3. A class size over 20 is a line in the sand we should not cross. Due to budgets we have, but we need to try to get back to where we were.
4. Last point, I am amazed at how good the teachers are. They may take a day off but it is to have a sub administer a test so they can get caught up on grading/ planning prep/ etc. They work very hard for a low salary. Thanks to all of you who teach.

You note that teacher evaluations by superiors are very "generous" yet suggest that evaluations of performance should include principal and self evaluation. Although you note that evaluations by principals for those teachers in the top and bottom 15% correlate with achievement based measures, this still implies that evaluations are not indicative of performance for 70% of teachers.

In private industry, an axiom is that if you can't measure the success of an effort that you shouldn't allocate resources to the effort.

I believe that standardized tests established by a national authority that can't be "dumbed down" which use rather than an absolute score a percentile ranking are the best way to go. When evaluating a teacher, the percentile ranking of their current students on the standardized test are compared to the percentile ranking in that subject for the same students in the previous year of study to determine if, on average, the students improved or not. Using a percentile ranking adjusts for factors outside of a teacher's control such as language abilities, home life, etc.

"For example, teachers with Master’s degrees are no more effective, on average, than teachers with a BA. Teachers who are traditionally certified are no more effective than alternatively certified (and, in some cases, uncertified teachers)."

All of the malarkey peddled by the Education Establishment is a scam ? Imagine that !