Tag Archives: poverty

Every year since 1974, thousands of people have come to Washington, DC to rally against Roe v. Wade. Protestors argue that pregnant women should be stripped of the ability to choose whether or not they want to have an abortion. Referencing the unborn fetuses pregnant women carry inside their bodies, these anti-abortion advocates call their demonstration the “March for Life.”

Politicians who support these efforts use similar language. Senator and former Republican presidential candidate Marco Rubio, for example, declared the “simple truth that all human life is sacred” to be the most recent march’s inspiration.

Yet neither Rubio nor the vast majority of marchers can credibly claim to have “pro-life” views.

I do not think fetuses should be viewed the same way as people, but let’s imagine you disagree with me. Suppose, based on that disagreement, you believe an abortion kills an innocent person. You think enabling the death of innocent people is wrong, and you thus think abortion must be opposed in all circumstances. Isn’t that a “pro-life” view?

Well, it depends. The logic of the ostensibly “pro-life” part of that reasoning is that, because X kills innocent people, and because killing innocent people is wrong, nobody should be able to choose X for any reason.

Here’s the problem with that logic: “X” could be any number of things. Drone strikes kill innocent people. More generally, war always does. So does the death penalty. And many other policies, while less active and direct than drone strikes, war, and putting people to death, effectively kill people. Refugees are potentially given a death sentence when the countries to which they’re fleeing don’t let them in. Thousands of people die each year due to inadequate access to health care. And societies’ refusal to invest in substantial benefits for poor people both here and around the world leads to preventable deaths all the time.

People who truly have a “pro-life” position, therefore, oppose all of these things. Those who are anti-war, anti-death penalty, pro-inclusive immigration, pro-universal health coverage, and pro-substantial benefits for the poor in addition to believing that fetuses are people and abortions are wrong may have a coherent, “pro-life” philosophy.

Needless to say, that’s not a description of Rubio. He, like so many other anti-choice Republicans, opposes aborting fetuses but none of the preventable deaths mentioned above.

That doesn’t mean Rubio wouldn’t offer a justification for his positions. He’d likely argue that drones, refugee bans, and military actions save more innocent lives than they sacrifice, that the death penalty is reserved for bad people who deserve it, and that providing health care and money for poor people slows economic growth, discourages work, and harms the very people such measures are intended to help. He’d be wrong about all of these things – the United States perpetrates far more violence than we prevent, there are innocent people on death row, and meeting the needs of poor people, which we have the resources to do, would be perfectly consistent with a strong economy and boost long-term economic mobility – but that’s not the point. The point is that Rubio does not allow the idea that “all human life is sacred” to guide his policy positions. Instead, he balances the sacrifice of human lives against other things he thinks are important and decides which he thinks matters more.

In the realm of abortion, Rubio and others have decided that a fetus’s right to be born is more important than a woman’s right to make a personal, intimate decision about her body. Again, if you believe fetuses are people and that life starts at conception, that may be a defensible position. But if you also oppose raising taxes on rich people to provide health care and other basic needs to kids after they’re born, or if you support war, or the death penalty, your position definitely isn’t “pro-life.” If you contend that “human life is sacred” only when that belief deprives women of rights but not when it consigns innocent people to death or cuts a little bit more into your fortune, you don’t really believe it.

I once began a K-12 education talk by putting the following two questions on a screen.

1. What is the single policy change that would most improve the quality of K-12 education?
2. What is the single policy change that would most reduce the opportunity gap between low-income and high-income students?

I asked audience members to, by a show of hands, indicate which question spoke to them more. They had three choices:

A) Question 1
B) Question 2
C) Doesn’t matter, since both question 1 and question 2 have the same answer

Stop and think for a second about which choice would have prompted you to raise your hand.

If you would have selected choice C, you would have been joined by about 90 percent of the audience at my talk. I expected that result. In a culture in which politicians routinely say things like “education is the closest thing to magic we have here in America” and cite low graduation rates in low-income areas as evidence of our education system’s failures, that view is unsurprising.

It’s also completely wrong. The overwhelming evidence that choice C is incorrect falls into at least five primary buckets:

1) There are large gaps in test score performance in the United States before students enter kindergarten. The graph shown below, from the Economic Policy Institute, documents the extent of these gaps (there are gaps in various cognitive and noncognitive skills as well), and as Sean Reardon has shown, there is evidence that they close during the school year, only to reopen during the summer months. The gaps have declined in size since the late 1990s, but they are, in Reardon’s words, “still huge.”

Inequitable access to preschool for low-income students is definitely part of the problem here, but gaps are apparent in infancy and probably due mostly to differences in housing, nutrition, medical care, exposure to environmental hazards, stress, and various other factors.

2) Decades of research into the causes of the gap in test scores between low-income and high-income students in the United States has consistently found a limited contribution from school-based factors. In the US, variations in school quality seem to explain no more than 33% of the discrepancies in test score performance; this number, which has been around since 1966, considers the influence of a student’s classmates to be a school-based factor (it arguably isn’t) and thus seems to be a conservative upper bound. Most studies put the school-based contribution to what is commonly called the “achievement gap” closer to 20%, with about 60% attributable to “student and family background characteristics [which] likely pertain to income/poverty” and the other 20% unexplained.

3) Economic success in this country is less common for low-income students who are successful in school than for high-income students who are unsuccessful in school. The graph below, made using data from the Pew Economic Mobility Project, compares the distribution of adult economic outcomes for children born into different quintiles of the income distribution with different levels of educational attainment. If education were the prime determinant of opportunity, we’d expect educational attainment to determine these adult economic outcomes. Yet the data show that children born into the top twenty percent who fail to graduate college typically fare better economically than children born into the bottom twenty percent who earn their college degrees. In fact, the born-into-privilege non-graduates are 2.5 times as likely to end up in the top twenty percent as adults as are the born-poor college graduates.

4) The test scores of students in the United States relative to the test scores of students around the world aren’t all that different than what students’ self-reports of their socioeconomic status would predict. The Programme for International Student Assessment (PISA) has an “index of economic, social, and cultural status” which incorporates family wealth, parents’ educational attainment, and more. There is a gap in test score performance between students who score high on this index and students who score relatively low on it in every country in the world. The size of the gap varies by country, as does the median test score, but there is a strong correlation overall between students’ socioeconomic status and their performance on standardized tests. The first graph below, in which each data point relates the average socioeconomic index score for a decile of a particular OECD country’s students to that decile’s average performance on PISA’s math test, depicts this relationship.

As the next two graphs show, test score performance for the bottom socioeconomic decile in the United States falls right on the OECD bottom-decile trend line, and while U.S. test scores for the second decile are a little below the OECD trend (as are U.S. scores for the next few deciles), socioeconomic status seems to explain American students’ performance on international tests pretty well overall.

5) The distribution of educational attainment in the United States has improved significantly over the past twenty-five years without significantly improving students’ eventual economic outcomes. While people with more education tend to have lower poverty rates than people with less education, giving people more education neither creates quality jobs nor eliminates bad ones, as Matt Bruenig has explained. A more educated population (see the first graph below), therefore, just tends to shift the education levels required by certain jobs upwards: jobs that used to require only a high school degree might now require a college degree, for example. The “cruel game of musical chairs in the U.S. labor market” (as Marshall Steinbaum and Austin Clemens have called it) that results is likely part of why poverty rates at every level of educational attainment increased between 1991 and 2014, as shown in the second graph below.

Source: Matt Bruenig

Source: Matt Bruenig

Bruenig’s analysis lacks a counterfactual – the overall poverty rate may well have increased if educational attainment hadn’t improved, rather than staying constant – but it’s a clear illustration of the problem with primarily education-focused anti-poverty initiatives.

None of this evidence changes the fact that education is very important. It just underscores that direct efforts to reduce poverty and inequality – efforts that put more money in the pockets of low-income people and provide them with important benefits like health care – are most important if our goal is to boost opportunities for low-income students.

50 years ago, Assistant Secretary of Labor Daniel Patrick Moynihan wrote a report called The Negro Family: The Case For National Action. The central argument in what has come to be called the Moynihan Report was that “The Breakdown of the Negro Family Has Led to a Startling Increase in Welfare Dependency,” and that “a national effort towards the problems of Negro Americans must be directed towards the question of family structure.”

The Moynihan Report, published in March of 1965.

New York Times columnist Nicholas Kristof thinks “Liberals Blew It” by excoriating this report. His conclusion, however, is based on a fundamental misunderstanding of the document’s critiques. The Moynihan Report’s faults lie not with its assertion that family stability is desirable, nor with its documentation of an increase in single-parent households, but with its insistence that family structure and Black “pathology” are primary drivers of poverty and inequality. This privilege-defending and inaccurate cultural narrative, however it was intended, implies that poor people of color are to blame for the effects of institutional racism and classism and diverts attention away from the real causes of inequity.

Those who denounced the Moynihan Report for that reason didn’t “blow it;” in fact, they presciently predicted how the report would be used to justify the false claim that “lifestyle issues” are the root cause of poverty. The real mistake is made not by people who recognize that connecting all types of families to money, basic necessities, and power is the best way to help them overcome hardship, but by those who continue to lend credence to the idea that the decline of traditional families has drastic consequences.

The lone piece of evidence Kristof cites in support of his claim that single-parent households lead to poor outcomes for low-income children is “an essay by Sara McLanahan of Princeton and Christopher Jencks of Harvard” in the March issue of EducationNext. While Kristof correctly notes that McLanahan and Jencks suggest that “growing up with just one biological parent reduces the chance that a child will graduate from high school by 40 percent,” he fails to discuss the broader context that calls his thesis into question.

First, the review McLanahan and Jencks cite (a review published by McLanahan and two colleagues in 2013, though the fact that McLanahan authored it is not mentioned in her and Jencks’s essay) found smaller associations or no relationship between family structure and other outcomes for children. As McLanahan and Jencks note about the review’s other findings regarding education, for example:

The absence of one’s biological father has not been shown to affect a child’s verbal and math test scores…The evidence for other indicators of educational performance, such as high school grades, skipping school, and college aspirations, is mixed, with some studies finding that father absence lowers school attendance and aspirations and others finding no effect.

Second, McLanahan et al themselves acknowledge that the relationships they do find may not be causal; the researchers write that “family disruption is not a random event…[T]he characteristics that cause father absence are likely to affect child well-being through other pathways.” Shawn Fremstad and Melissa Boteach explain in a comprehensive report published in January that while other studies that look at this issue find similar associations, most researchers are much more wary than McLanahan and colleagues of suggesting a causal link between family structure and indicators of children’s well-being.*

In fact, Kristof’s suggestion that the rise in single-parent households is a major driver of poverty and inequality is incompatible with some key details. For instance, while poverty levels in the United States remain far too high, they have fallen significantly over the past 50 years, in large part due to the safety net. If family structure were, as Moynihan contended, “the principal source of most of the aberrant, inadequate, or antisocial behavior that did not establish, but now serves to perpetuate the cycle of poverty and deprivation” (a contention that highlights the absurdity of Kristof’s argument that “liberal denunciations of Moynihan were terribly unfair”), this reduction in poverty would not have happened, as the number of “nontraditional” families has exploded during this time period.

Third, and most significantly, little research explores more plausible causal explanations for the relationship between economic and social disadvantage and family structure. It may very well be the case that the hardships associated with poverty make traditional families less likely, or that many of the factors that contribute to poverty and inequality also disrupt family stability.

Survey data suggests that these alternate interpretations are more likely to be accurate than Kristof’s; if nontraditional households were a cause rather than an effect or merely a correlate of disadvantage, we’d expect more support for traditional family structure among more advantaged individuals. The reverse is true, however; a study of survey results in 2012 noted that, “relative to higher income respondents, low-income respondents held more traditional values toward marriage, had similar romantic standards for marriage, and experienced similar skills-based relationship problems.” That study is consistent with Fremstad and Boteach’s summary of earlier research: “If anything, working-class people seem to value the cultural and religious aspects of marriage as much or more highly than more-educated adults.”

Relatedly, as Jared Bernstein pointed out last year, “policy interventions to encourage marriage have been shown to be quite ineffective” (and costly; as Bernstein also noted, “[t]wo pilot programs introduced in the George W. Bush years cost $10,000 per couple”). Wanting children to grow up in stable households is of course a laudable goal, but the evidence indicates that achieving household stability is not about culture, preferences, or a particular type of family structure. Instead, it is about a broad social justice agenda that addresses economic and social barriers to equality.

On some level, Kristof recognizes that direct means of addressing economic and social disadvantage should be the focus of anyone interested in “helping American kids.” He correctly decries how our racially- and economically-biased system of mass incarceration has torn families apart, and he also appropriately advocates that we “support programs to boost the economic prospects for poorer families.” Criminal justice reforms, safety net programs like SNAP (food stamps), the Earned Income Tax Credit (EITC), and Medicaid, and pretax income-boosting policies like the minimum wage and elements of the full employment agenda will likely promote family stability, and, more importantly, they are a sampling of some of the best methods we have to reduce poverty and inequality directly. To the extent that Kristof is advocating for this set of ideas, he is absolutely right to do so.

But the general thrust of Kristof’s piece, like the narrative in the Moynihan Report before it, undermines the fights for racial and class equality. In the future, the report’s defenders would do better to stop castigating the activists who disagree with them and start listening to and reflecting on advocates’ legitimate concerns.

*Fremstad and Boteach also note that “McClanahan’s review and much of the existing research do not clearly distinguish between the effect of family structure per se and the effect of family instability,” a clarification consistent with Kristof’s correct observation that children raised by loving gay parents do very well.

Jared Bernstein and I just published a comprehensive PowerPoint presentation on inequality in the United States (available for download here). This presentation is first and foremost intended as a resource. Part 1 of the presentation documents the increase in inequality over the past 35 years; the trend is evident from every major data set and income definition. Inequality deniers have fortunately become a rapidly-dwindling breed, but should you encounter one, Part 1 should help set the record straight.

Part 2 discusses why inequality matters. As we summarize on our slides, inequality “reduces opportunities, undermines the democratic process, distributes growth unevenly, and may even have negative macroeconomic effects.” We provide the mechanisms behind these reasons and evidence documenting them as well. The slide notes contain a more in-depth look at these issues, as they do throughout the presentation, for anyone interested.

Fortunately, increasing inequality is not inevitable; it is “a problem that better policies can directly address.” Part 3 of our presentation explores some of these policies, measures that can begin to address inequality’s causes and counter its effects. While our slides in this section are in no way exhaustive, they’re a good starting point for the type of comprehensive social justice agenda I’ve mentioned previously.

I believe the section on equality of opportunity (or lack thereof) in Part 2 is particularly relevant for education stakeholders, as it highlights the importance of this comprehensive agenda (as opposed to a narrower, education-only policy focus). Consider the following chart, the extended version of our featured graph from slide 26:

Source: Pew Economic Mobility Project

This chart uses data from the Pew Economic Mobility Project to compare adult economic outcomes for two different types of students: students who grow up in the bottom income quintile but manage to graduate college (“poor college grads”) and students who grow up in the top income quintile but don’t make it through college (“wealthy non-graduates”). Wealthy non-graduates (49%) are almost twice as likely as poor college graduates (27%) to end up in one of the top two income quintiles as adults.

As we discuss in our presentation, inequality presents a variety of education-specific obstacles for low-income students, and addressing these obstacles (and thus facilitating college completion) is an important part of our agenda. At the same time, the less favorable distribution for low-income college graduates shown above provides a critical reminder of the importance of a comprehensive social justice agenda. If we truly want to provide low-income students with opportunities equivalent to those of their higher-income peers, we need strategies that address the needs of low-income families, strategies that address some of the direct effects of growing up disadvantaged. That means we must get the economy back to full employment. It means we need a strengthened safety net, higher labor standards, and a reversal of the decline in unionization. It also means we need better-regulated markets and improved fiscal policy. A push for quality education can’t have its intended effect unless it’s part of this larger agenda.

Jared and I hope that this presentation, by documenting inequality’s rise and consequences, provides a compelling basis for these claims. While the historically high level of inequality represents a serious threat to fundamental American values – opportunity, democracy, and broadly shared prosperity – we believe we can combat this problem if we acknowledge it and work together on solutions.

StudentsFirst Vice President Eric Lerum and I recently began a debate about approaches to teacher evaluation. During Part 2 of that debate, the conversation touched on the relationship between anti-poverty work and education reform. We resume that conversation below.

Here were the relevant parts of our original exchange, in case you missed it:

Lerum: The larger point that is made repeatedly is that because outside factors play a larger overall role in impacting student achievement, we should not focus on teacher effectiveness and instead solve for these other factors. This is a key disconnect in the education reform debate. Reformers believe that focusing on things like teacher quality and focusing on improving circumstances for children outside of school need not be mutually exclusive. Teacher quality is still very important, as Shankerblog notes. Improving teacher quality and then doing everything we can to ensure students have access to great teachers does not conflict at all with efforts to eliminate poverty. In fact, I would view them as complementary. But critics of these reforms use this argument to say that one should come before the other – that because these other things play larger roles, we should focus our efforts there. That is misguided, I think – we can do both simultaneously. And as importantly in terms of the debate, no reformer that I know suggests that we should only focus on teacher quality or choice or whatever at the expense or exclusion of something else, like poverty reduction or improving health care.

Spielberg: I believe you discuss [a] very important question…Given that student outcomes are primarily determined by factors unrelated to teaching quality, can and should people still work on improving teacher effectiveness?

Yes! While teaching quality accounts for, at most, a small percentage of the opportunity gap, teacher effectiveness is still very important. Your characterization of reform critics is a common misconception; everyone I’ve ever spoken with believes we can work on addressing poverty and improving schools simultaneously. Especially since we decided to have this conversation to talk about how to measure teacher performance, I’m not sure why you think I’d argue that “we should not focus on teacher effectiveness.” I am critiquing the quality of some of StudentsFirst’s recommendations – they are unlikely to improve teacher effectiveness and have serious negative consequences – not the topic of reform itself. I recommend we pursue policy solutions more likely to improve our schools.

Critics of reform do have a legitimate issue with the way education reformers discuss poverty, however. Education research’s clearest conclusion is that poverty explains inequality significantly better than school-related factors. Reformers often pay lip-service to the importance of poverty and then erroneously imply an equivalence between the impact of anti-poverty initiatives and education reforms. They suggest that there’s far more class mobility in the United States than actually exists. This suggestion harms low-income students.

I wholeheartedly agree that anti-poverty initiatives and smart education reforms are complementary. I’d just like to see StudentsFirst speak honestly about the relative impact of both. I’d also love to see you hold donors and politicians accountable for their overall impact on students in low-income communities. Then reformers and critics of reform alike could stop accusing each other of pursuing “adult interests” and focus instead on the important work of improving our schools.

Lerum: So I’m beginning to understand where some of the miscommunication is coming from. You speak a lot about how you view StudentsFirst’s (and other reformers’) discussion of poverty from the perspective of what you expect us to talk about, rather than from the perspective of our stated objectives. That is, what you deem as “lip service” is merely an acknowledgement of something that is not our primary focus. There are many folks in education reform – I have a few on my team – who could spend hours talking about poverty reduction and could very easily work in another field that more traditionally aligns with what you think of as efforts geared toward reducing poverty. But the route we’re taking is one where reducing poverty, achieving social justice, lifting the long-term opportunities for our country – they all intersect. And therefore what we focus on at StudentsFirst are the policy levers – what we think of as levers for reform or change. For example, creating the conditions for other reforms to flourish or for educators and school leaders to use their resources more wisely (fiscal transparency, structuring smarter compensation systems, creating more school-level autonomy) are levers, whereas something like instituting a STEM program or increasing funding for social and mental health services would be specific programs or initiatives. Both are great for kids. Both are needed in order to ultimately reduce poverty. But we’re squarely focused on the former, while critics seem to be expecting we would focus on the latter. This disconnect is made worse though because critics seem to believe that an approach that involves initiatives is the only way to combat poverty. There’s a lack of appreciation and understanding of what’s intended by reform efforts that target levers.

Spielberg: I actually wasn’t talking about the distinction between levers and initiatives; I was talking about accurate messaging and political activity.

My two critiques from above (rephrased and with my questions for you added) were:

1) StudentsFirst leaders and board membersfrequentlysuggest that education can improve the lives of low-income kids as much or more than alleviating poverty. That suggestion isdemonstrablyfalse. You could say the following, but don’t: “Research is clear that school-related factors cannot fix the achievement gap, but it’s also clear that schools make a difference. They seem to account for about 20% of student achievement, and our organization believes we can maximize the impact of this 20% with an intense focus on certain policy levers. We fully support other organizations that work on the anti-poverty efforts that are most important for low-income kids.” Why won’t you speak honestly about the limitations and relative importance of the reforms you push when compared with other efforts?

I appreciate, as you mentioned in a comment on Part 2 of this conversation, that you “created a school-based mental health program and piloted a half-dozen evidence-based mental/social/emotional health programs” in DC, and I’d love to talk more about the other issues you raised in your response, but I think your thoughts on the above points and questions are most relevant to typical reformer critiques.

Lerum: On the policy discussion, I would just end with this then. Saying that education can’t solve the achievement gap is demonstrably false only works if you base it on the education system we have now. To say that today’s education system cannot and has not solved the problem of poverty or the problem of the achievement gap thus far is correct. It’s also correct that in 60 years we haven’t solved the problem of segregation. But I got into this work because, like every reformer I know, I believe completely that we can do better than this. We don’t even know what’s possible because we haven’t actually tried. We’ve never run a public school system at scale completely differently. We’re not very good at breaking the mold of a model that hasn’t worked. But there are reasons – an increasingbody of research – to believe that if we do, we just might get somewhere. That’s a theory of change. You can disagree with it. But you do not have the evidence that it won’t work because everything that’s been tried or done thus far has been done within some confines or under some of the restraints of the existing system. There are many limitations, that’s true. I think we’ve done a pretty good job of talking about those limitations through our advocacy work.

I would also add that there’s little evidence that other approaches that are championed as counters to reform will have a tremendous impact on kids either. I would love to see this “comprehensive assessment of de Blasio’s policies” that you spoke of earlier – but it doesn’t exist. Rather, there is simply a different theory of change – that certain other levers, be they class size or overall funding or whatever will have a greater impact than reforms we’re advocating for. What we need is a way to model, using rigorous research, what the potential impact of various reforms would be. That doesn’t exist right now either. But what I’m trying to get you to agree to here is that by attacking one side as only having a theory that’s not proven while not acknowledging that the anti-reform side isn’t exactly operating with a track record of success seems to me to be disingenuous, but more importantly, allows opponents to occupy this space wherein they own the debate on what’s good for solving poverty, what the right approach is to combat social ills, etc. And I just believe that way of thinking hasn’t gotten us very far and doesn’t advance social change.

As to the political issues you raise – we consistently say that we will support public officials who support the policies we believe are right for kids. I understand you have issues with our agenda – but there’s nothing inconsistent about a single-issue organization supporting candidates that support and will advocate for their issues. That almost always means as an organization we will support candidates with whom I may not agree with on a personal level when it comes to any number of other issues. But that is not unique to StudentsFirst and I do not think it is reasonable to expect us to answer for their stances on other issues or to ask them to change their stance on other issues. The issues we prioritize are those on our policy agenda and we work to stick with that approach, as do countless other organizations in other fields.

Spielberg: While I would agree with you (and said in Part 1 of our conversation) that the research on many in-school reforms is mixed, the suggestion that you seem to be making – that school-based reforms alone could potentially solve the opportunity gap – is contradicted by existing research and logic. Research has never attributed more than one-third of the variation in student outcomes to school-based factors, we know that “children from rich and poor families score very differently on school readiness tests when they enter kindergarten,” and there is even “some evidence that achievement gaps between high- and low-income students actually narrow during the nine-month school year, but they widen again in the summer months.” Though I suppose it’s theoretically possible that these studies are wrong, that could be said about almost anything, and the findings you link about teacher attrition and charters in no way support that conclusion. Our knowledge about the disadvantages of growing up in poverty and the past several decades of research suggest that this theoretical possibility is negligible, which is why I called that statement demonstrably false.

I certainly understand the sentiment behind what you’re saying – we are in agreement that we haven’t yet maximized education’s contribution to anti-poverty efforts, and I think it’s important to remember and highlight that fact – but all the evidence points to a relatively low upper bound on what education reforms alone can accomplish. Recognizing that anti-poverty work matters more than schools does not preclude us from arguing that what happens in schools is very important for low-income kids.

I really appreciate having had this conversation and want to thank you again for going back-and-forth with me, but I believe we’re at a bit of an impasse. My questions deal with the out-of-school factors, like having access to health care, that very clearly matter for low-income students, and I don’t think your response addresses the issues I raised. You’re absolutely right that StudentsFirst isn’t alone in narrowing its policy focus, but the fact that other organizations also do so doesn’t qualify as a defense of that approach. Talking about what’s “right for kids” means considering more than just education policy.

StudentsFirst Vice President Eric Lerum and I recently began debating the use of standardized test scores in high stakes decision-making. I argued in a recent blog post that we should instead evaluate teachers on what they directly control – their actions. Our conversation, which began to touch on additional interesting topics, is continued below.

Lerum: To finish the outcomes discussion – measuring teachers by the actions they take is itself measuring an input. What do we learn from evaluating how hard a teacher tries? And is that enough to evaluate teacher performance? Shouldn’t performance be at least somewhat related to the results the teacher gets, independent of how hard she tries? If I put in lots of hours learning how to cook, assembling the perfect recipes, buying the best ingredients, and then even more hours in the kitchen – but the meal I prepare doesn’t taste good and nobody likes it, am I a good cook?

Regarding your use of probability theory and VAM – the problem I have with your analysis there is that VAM is not used to raise student achievement. So using it – even improperly – should not have a direct effect on student achievement. What VAM is used for is determining a teacher’s impact on student achievement, and thereby identifying which teachers are more likely to raise student achievement based on their past ability to do so. So even if you want to apply probability theory and even if you’re right, at best what you’re saying is that we’re unlikely to be able to use it to identify those teachers accurately on an ongoing basis. The larger point that is made repeatedly is that because outside factors play a larger overall role in impacting student achievement, we should not focus on teacher effectiveness and instead solve for these other factors. This is a key disconnect in the education reform debate. Reformers believe that focusing on things like teacher quality and focusing on improving circumstances for children outside of school need not be mutually exclusive. Teacher quality is still very important, as Shankerblog notes. Improving teacher quality and then doing everything we can to ensure students have access to great teachers does not conflict at all with efforts to eliminate poverty. In fact, I would view them as complementary. But critics of these reforms use this argument to say that one should come before the other – that because these other things play larger roles, we should focus our efforts there. That is misguided, I think – we can do both simultaneously. And as importantly in terms of the debate, no reformer that I know suggests that we should only focus on teacher quality or choice or whatever at the expense or exclusion of something else, like poverty reduction or improving health care.

If you’re interested in catching up on class size research, I highly recommend the paper published by Matt Chingos at Brookings, found here with follow-up here. To be clear about my position on class size, however; I’m not against smaller class sizes. If school leaders determine that is an effective way for improving instruction and student achievement in their school, they should utilize that approach. But it’s not the best approach for every school, every class, every teacher, or every child. And thus, state policy should reflect that. Mandating class size limits or restrictions makes no sense. It ties the hands of administrators who may choose to staff their schools differently and use their resources differently. It hinders innovation for educators who may want to teach larger classes in order to configure their classrooms differently, leverage technology or team teaching, etc. Why not instead leave decisions about staffing to school leaders and their educators?

The performance framework for San Jose seems pretty straightforward. I’m curious how you measure #2 (whether teachers know the subjects) – are those through rigorous content exams or some other kind of check?

I think a solid evaluation system would include measures using indicators like these. But you would also need actual student learning/growth data to validate whether those things are working – as you say, “student outcome results should take care of themselves.” You need a measure to confirm that.

I honestly think my short response to all of this would be that there’s nothing in the policies we advocate for that prevent what you’re talking about. And we advocate for meaningful evaluations being used for feedback and professional development – those are critical elements of bills we try to move in states. But as a state-level policy advocacy organization, we don’t advocate for specific models or types of evaluations. We believe certain elements need to be there, but we wouldn’t be advocating for states to adopt the San Jose model or any other specifically – that’s just not what policy advocacy is. So I think there’s just general confusion about that – that simply because you don’t hear us saying to build a model with the components you’re looking for, that must mean we don’t support it. In fact, we’re focused on policy at a level higher than the district level, and design and implementation of programs isn’t in our wheelhouse.

Spielberg: I believe you discuss three very important questions, each one of which deserves some attention:

1) Given that student outcomes are primarily determined by factors unrelated to teaching quality, can and should people still work on improving teacher effectiveness?

Yes! While teaching quality accounts for, at most, a small percentage of the opportunity gap, teacher effectiveness is still very important. Your characterization of reform critics is a common misconception; everyone I’ve ever spoken with believes we can work on addressing poverty and improving schools simultaneously. Especially since we decided to have this conversation to talk about how to measure teacher performance, I’m not sure why you think I’d argue that “we should not focus on teacher effectiveness.” I am critiquing the quality of some of StudentsFirst’s recommendations – they are unlikely to improve teacher effectiveness and have serious negative consequences – not the topic of reform itself. I recommend we pursue policy solutions more likely to improve our schools.

Critics of reform do have a legitimate issue with the way education reformers discuss poverty, however. Education research’s clearest conclusion is that poverty explains inequality significantly better than school-related factors. Reformers often pay lip-service to the importance of poverty and then erroneously imply an equivalence between the impact of anti-poverty initiatives and education reforms. They suggest that there’s far more class mobility in the United States than actually exists. This suggestion harms low-income students.

I wholeheartedly agree that anti-poverty initiatives and smart education reforms are complementary. I’d just like to see StudentsFirst speak honestly about the relative impact of both. I’d also love to see you hold donors and politicians accountable for their overall impact on students in low-income communities. Then reformers and critics of reform alike could stop accusing each other of pursuing “adult interests” and focus instead on the important work of improving our schools.

2) How can we use student outcome data to evaluate whether an input-based teacher evaluation system has identified the right teaching inputs?

This concept was the one we originally set out to discuss. I’d love to focus on it in subsequent posts if that works for you (though I’d love to revisit the other topics in a different conversation if you’re interested).

I’m glad we agree that “a solid evaluation system would include [teacher input-based] measures…like [the ones used in San Jose Unified].” I also completely agree with you that we need to use student outcome data “to validate whether those things are working.” That’s exactly the use of student outcome data I recommend. Though cooks probably have a lot more control over outcomes than teachers, we can use your cooking analogy to discuss how Bayesian analysis works.

We’d need to first estimate the probability that a given input – let’s say, following a specific recipe – is the best path to a desired outcome (a meal that tastes delicious). This probability is called our “prior.” Let’s then assume that the situation you describe occurs – a cook follows the recipe perfectly and the food turns out poorly. We’d need to estimate two additional probabilities. First, we’d need to know the probability the food would have turned out badly if our original prediction was correct and the recipe was a good one. Second, we’d need the probability that the food would have turned out poorly if our original prediction was incorrect and the recipe was actually a bad one. Once we had those estimates, there’s a very simple formula we could use to give us an updated probability that the input – the recipe – is a good one. Were this probability sufficiently low, we would throw out the recipe and pick a new one for the next meal. We would, however, identify the cook as an excellent recipe-follower.

This approach has several advantages over the alternative (evaluating the cook primarily on the taste of the food). Most obviously, it accurately captures the cook’s performance. The cook clearly did an excellent job doing what both you and he thought was a good idea – following this specific recipe – and can therefore be expected to do a good job following other recipes in the future. If we punished him, we’d be sending the message that his actual performance matters less than having good luck, and if we fired him, we’d be depriving ourselves of a potentially great cook. Additionally, it’s not the cook’s fault that we picked the wrong cooking strategy, so it’s unethical to punish him for doing everything we asked him to do.

Just as importantly, this approach would help us identify the strategies most likely to lead to better meals in the long run. We might not catch the problem with the recipe if we incorrectly attribute the meal’s taste to the cook’s performance – we might end up continuously hiring and firing a bunch of great cooks before we realize that the recipe is bad. If we instead focus on the cook’s locus of control – following the recipe – and use Bayesian analysis, we will more quickly discover the best recipes and retain more cooks with recipe-following skills. Judging cooks on their ability to execute inputs and using outcomes to evaluate the validity of the inputs would, over time, increase the quality of our meals.

Let’s now imagine the analogous situation for teachers. Suppose a school adopts blended learning as its instructional framework, and suppose a teacher executes the school’s blended learning model perfectly. However, the teacher’s value added (VAM) results aren’t particularly high. Should we punish the teacher? The answer, quite clearly, is no; unless the teacher was bad at something we forgot to identify as an effective teaching practice, none of the explanations for the low scores have anything to do with the teacher’s performance. Just as with cooking, we might not catch a real problem with a given teaching approach if we incorrectly attribute outcome data to a teacher’s performance – we might end up continuously hiring and firing a bunch of great teachers based on random error, a problem with an instructional framework, or a problem with VAM methodology.

The improper use of student outcome data in high-stakes decision-making has negative consequences for students precisely because of this incorrect attribution. Making VAM a defined percentage of teacher evaluations leads to employment decisions based on inaccurate perceptions of teacher quality. Typical VAM usage also makes it harder for us to identify successful teaching practices. If we instead focus on teachers’ locus of control – effective execution of teacher practices – and use Bayesian analysis, we will more quickly discover the best teaching strategies and retain more teachers who can execute teaching strategies effectively. Judging teachers on their ability to execute inputs and using outcomes to evaluate the validity of the inputs would, over time, increase the likelihood of student success.

3) As “a state-level policy advocacy organization,” what is the scope of StudentsFirst’s work?

You wrote that StudentsFirst “[doesn’t] advocate for specific models or types of evaluations” but believes “certain elements need to be there.” One of the elements you recommend is “evaluating teachers based on evidence of student results.” This recommendation has translated into your support for the use of standardized test scores as a defined percentage of teacher evaluations. I was not recommending that you ask states to adopt San Jose Unified’s evaluation framework (as an aside, the component you ask about deals mostly with planning and, among other things, uses lesson plans, teacher-created materials, and assessments as evidence) or that you recommend across-the-board class size reduction (thanks for clarifying your position on that, by the way – I look forward to reading the pieces you linked). Instead, since probability theory and research suggest it isn’t likely to improve teacher performance, I recommend that StudentsFirst discontinue its push to make standardized test scores a percentage of evaluations. You could instead advocate for evaluation systems that clearly define good teacher practices, hold teachers accountable for implementing good practices, and use student outcomes in Bayesian analysis to evaluate the validity of the defined practices. This approach would increase the likelihood of achieving your stated organizational goals.

Thanks again for engaging in such an in-depth conversation. I think more superficial correspondence often misses the nuance in these issues, and I am excited that you and I are getting the opportunity to both identify common ground and discuss our concerns.

Click here to read Part 3a of the conversation, which focuses back on the evaluation debate.

Click here to read Part 3b of the conversation, which focuses on how reformers and other educators talk about poverty.

My girlfriend’s very hospitable and generous family gave me some great gifts for the holidays when I stayed with them in upstate New York. As I rocked my new Teach For America T-shirtin the Rochester airport on Christmas Eve, my cursory overview of Nate Silver’s new book, The Signal and the Noise,inspired me to write this post.

While most people probably know Silver for his election predictions and designation in 2009 as one of the world’s 100 Most Influential People, Silver has been my baseball stat guru for considerably longer than he’s been doing political analysis. In one of my favorite books of all time, Baseball Between the Numbers, Silver penned a brilliant examination of clutch hitting that I still quote at least four or five times a year. I have generally found Silver’s arguments compelling not just because of his statistical brilliance, but also because of his high standards for data collection and analysis, evident in the following passage from the introduction of his book:

The numbers have no way of speaking for themselves. We speak for them. We imbue them with meaning…[W]e may construe them in self-serving ways that are detached from their objective reality…Before we demand more of our data, we need to demand more of ourselves.

In few fields are Silver’s words as relevant as education. While the phrase “data-driven” has become ubiquitous in discussions of school reform and high-quality instruction, most people discussing education have very little understanding of what the statistics actually say. As I’ve written before, many studies that reformers reference to push their policy agendas are methodologically unsound, and many more have findings very different than the summaries that make it into the news.

It’s hard to know how many reformers just don’t understand statistics, how many fall victim to confirmation bias, and how many intentionally mislead people. But no matter the reason for their errors, those of us who care about student outcomes have a responsibility to identify statistical misinterpretation and manipulation and correct it. Policy changes based on bad data and shoddy analyses won’t help (and will quite possibly harm) low-income students.

Fortunately, I believe one simple practice can help us identify truth in education research: read the full text of education research articles.

Yes, reading the full text of academic research papers can be time consuming and mind-numbingly dull at times, but reading articles’ full text is vitally important if you want to understand research findings. Sound bites on education studies rarely provide accurate information. In a Facebook comment following my most recent post about TFA, a former classmate of mine referenced a 2011 study by Raj Chetty to argue that we can’t blame the achievement gap on poverty. “If you leave a low value-added teacher in your school for 10 years, rather than replacing him with an average teacher, you are hypothetically talking about $2.5 million in lost income,” claims one of the co-authors of the study in a New York Times article. Sounds impressive. Look under the hood, however, and we find that, even assuming the study’s methodology is foolproof (it isn’t), the actual evidence can at best show an average difference of $182 in the annual salaries of 28-year-olds.

As I’ve mentioned before, there’s also a poor statistical basis for linking student results on standardized test scores to teacher evaluation systems. Otherwise useful results can give readers the wrong impression when they gloss over or omit this fact, a point underscored by a recent article describing an analysis of IMPACT (the D.C. Public Schools teacher evaluation system). The full textof the study provides strong evidence that the success of D.C.’s system thus far has been achieved despite a lack of variation in standardized test score results among teachers in different effectiveness categories. Instead, the successes of the D.C. evaluation system are driven by programs teachers unions frequently support, programs like robust and meaningful classroom observations that more accurately measure teacher effectiveness.

I would, in general, recommend skepticism when reading articles on education, but I’d recommend skepticism in particular when someone offers a statistic suggesting that school-related changes can solve the achievement gap. Education research’s only clear conclusion right now is that poverty explains the majority of student outcomes. The full text of Chetty’s most recent study defending value-added models acknowledges that “differences in teacher quality are not the primary reason that high SES students currently do much better than their low SES peers” and that “differences in [kinder through eighth grade] teacher quality account for only…7% of the test score differences” between low- and high-income schools. In fact, that more recent study performs a hypothetical experiment in which the lowest-performing low-income students receive the “best” teachers and the highest-performing affluent students receive the “worst” teachers from kinder through eighth grade and concludes that the affluent students would still outperform the poor students on average (albeit by a much smaller margin). Hayes made the same point to Rhee that I made in my last post: because student achievement is influenced significantly more by poverty than by schools, discussions about how to meet our students’ needs must address income inequality in addition to evidence-based school reforms. We can’t be advocates for poor students and exclude policies that address poverty from our recommendations.

When deciding which school-based recommendations to make, we must remember that writers and policymakers all too often misunderstand education research. Many reformers selectively highlight decontextualized research that supports their already-formed opinions. Our students, on the other hand, depend on us to combat misleading claims by doing our due diligence, unveiling erroneous interpretations, and ensuring that sound data and accurate statistical analyses drive decision-making. They rely on us to adopt Nate Silver’s approach to baseball statistics: continuously ask questions, keep an open mind about potential answers, and conduct thorough statistical analyses to better understand reality. They rely on us to distinguish statistical significance from real-world relevance. As Silver writes about data in the information age more generally, education research “will produce progress – eventually. How quickly it does, and whether we regress in the meantime, depends on us.”

Update:Gary Rubinstein and Bruce Baker (thanks for the heads up, Demian Godon) have similar orientations to education research – while we don’t always agree, I appreciate their approach to statistical analysis.

Update 3 (7/8/14): The Raj Chetty study linked above seems to have been modified – the pieces I quoted have disappeared. Not sure when that happened, or why, but I’d love to hear an explanation from the authors and see a link to the original.