Hoxby’s Other “Stubborn Facts”

Stanford University economist Caroline Hoxby released yesterday an update to her 2007 study of charter schools in New York City.1 In the study, she compares the state examination results of students enrolled in the City’s charter schools (i.e. those students “lotteried-in”) to the results for those students who applied to a charter but were not selected for admission (i.e. the “lotteried-out”). In many respects, this is a good approach as it aims to account for the possibility that charters enroll more motivated families and that it is this motivation, rather than any particular charter school effect, that is the cause of stronger student achievement.

Hoxby’s findings are encouraging: by the third grade, the average charter school student was 5.8 points ahead of the lotteried-out counterpart in math and was 5.3 points ahead in English Language Arts.2 As Hoxby follows students’ achievement from 2001 to 2008, she also finds that the average charter school student gained 3.6 more points each year in math and 2.4 more points each year in ELA. For an average charter student continuously enrolled in grades four through eight, the effect is larger with annual gains of 5.0 points in math and 3.6 in ELA above the performance of the lotteried-out student. (Last year, nine charters enrolled students across all of these grades.)

To put this in some context, Hoxby explains that the difference between a student not meeting standard and meeting standard is about 31 points in math and 44 points in ELA. She also points out that, on average, students in neighboring and affluent Scarsdale typically out-perform students in New York City by 35 to 40 points. In this context, Hoxby claims that the compounded gains for an average student continuously enrolled in third to eighth grade in a charter nearly closes the “Harlem-to-Scarsdale” achievement gap and implies — going outside of her dataset — that the trend will continue.

Such a dramatically-presented conclusion is sure to feature prominently in charter advocates’ efforts to expand the number of charter schools across the city and state. And if it’s true, then why shouldn’t we? The answer actually depends on how policymakers weigh the goal of improved student achievement against other worthy goals, such as greater educational equity and meaningful diversity. And on these other objectives, nagging questions dog the charter sector.

For example, Hoxby finds that 92 percent of charter students are black or Hispanic, compared to 72 percent in district schools and concludes that “the existence of charter schools in the city therefore leaves the traditional public schools less black, more white, and more Asian.” Such racial segregation is consistent with research on charter schools in other states including North Carolina, Texas and elsewhere.

Although this statistic is likely to be a function of charter schools’ location in largely black and Hispanic neighborhoods, Hoxby also reports that fewer white students are applying to the charters; although 14 percent of residents in the charter school neighborhoods are white non-Hispanic, only 4 percent are applying. Additionally, Hoxby confirms that the city’s charters enroll fewer English language learners (4 percent) than district public schools (14 percent) — despite their location in Hispanic communities.

On another key measure of economic diversity, it’s puzzling that Hoxby compares charter school free- and reduced-price lunch participation rates, which she pegs at 91 percent, to the citywide rate of 72 percent. For a report that prides itself on apples-to-apples comparisons, this charter-to-city comparison is a glaring inconsistency. As Edwize has demonstrated here and here, a better comparison is between charters and the surrounding district/feeder schools. By this comparison, charters have fewer students eligible for free lunch,3 suggesting that the city’s charters are more affluent than the neighboring district schools.

It’s also worth remembering that charter schools do not have attendance zones and can, in theory, draw students from across neighborhoods. But as the demographic statistics reveal, if charters are attracting applicants from beyond their immediate neighborhoods, it is not generating more racial, ethnic, and economic diversity.

Incidentally, Hoxby’s updated report makes no mention of student attrition from charter schools to district schools, despite the fact that she has access to student-level data over many years. Presumably she can, and should, confirm or reject the anecdotal reports of this occurrence, as student attrition biases achievement results. Attrition is also a measure of equity, as district schools must educate all of the students who come through their doors while charters, with site-specific discipline codes and the power to expel students, do not.

Moreover, the report provides tacit evidence that all of the students and families who apply for a charter are in fact different from the other public school parents and students who don’t apply. Whether they are more motivated, place a higher emphasis on education, or some other unmeasurable characteristic, Hoxby finds that “lotteried-out students’ performance does improve and is better than the norm in the U.S. where, as a rule, disadvantaged students fall further behind as they age” (emphasis in the original). Given this and the other demographic findings, it is unclear how the report “lays to rest the persistent myth that charter schools “cream” the best students,” as one charter advocate maintains. Quite the opposite — the achievement results for all charter applicants suggests there is likely to be some truth to the creaming “myth.”

These issues of equity and parity need more prominence in the charter debate. Although academic achievement remains the bottom-line, policymakers should be cognizant of the effect of charters on racial and economic diversity and whether charters uphold their full responsibility as public schools. I, for one, am reluctant to abandon such goals in a narrow pursuit of higher test scores.

Finally, Hoxby’s much vaunted methodology deserves a comment. Although she maintains that this is the “gold-standard” approach when seeking to determine the unique impact of charter school effects, she fails to address two complications that would otherwise moderate her findings:

First, it is conceivable that low-performing charters are undersubscribed and may not have a large “lotteried-out” population to study. In effect, this could under-weight the results of poor performing charters and biases her data in favor of higher-performing charters. A technical version of the paper or scholarly access to her data could ascertain if this is in fact occurring. And despite the reports sweeping conclusions, there are low-performing charters in her study: Hoxby finds that 31 percent of students attend a charter that is performing no better than or worse than the schools attended by lotteried-out counterparts in math and 24 percent are doing no better or worse in ELA.

Second, it’s not clear that Hoxby’s methodology takes into account peer effects — the straightforward concept that students not only learn from their teachers but are also influenced by their peers. Charter schools benefit from the fact that 100 percent of their students hail from motivated families; as a result, a charter student is surrounded by peers who are there by choice — rather than by attendance zone. In contrast, the “lotteried-out” students may not benefit from such an intensive peer effect.

* * * * *

Good news about academic achievement is always welcome, and the hardworking charter school teachers (many of whom are UFT members) and students deserve the credit for this success. And as it’s hardly an insight to say that the political, economic, and social stakes are high in the reform of public education, I have no doubt that we’ll continue to hear a lot about the updated “Hoxby study” in the months to come. Let’s hope that the discussion includes all of the findings and implications of her study. That only seems fitting, given the goal of New York’s Charter Schools Act to “increase learning opportunities for all students, with special emphasis on expanded learning experiences for students who are at-risk of academic failure” (emphasis mine).

2 Although, the untransformed standardized score in ELA is not statistically significant at the more rigorous .05 level of confidence (i.e. the p-value).

3 Free-lunch eligibility, rather than free- and reduced-price lunch eligibility, is a more meaningful socio-economic measure in New York City as it identifies a more severe level of poverty and as the data are collected by the city’s Department of Education to determine the allocation of federal funds to the city’s district schools.

6 Comments:

“Attrition is also a measure of equity, as district schools must educate all of the students who come through their doors while charters, with site-specific discipline codes and the power to expel students, do not.”

Perhaps I’ve missed something here. Weren’t these charters started precisely because parents and many others felt that district schools were supposed to – but were not – educating all of the students who came/come through their doors?

Why let the perfect be the enemy of the good? At least some kids are being reasonably educated. Are they to be denied this until the district home public school system gets its collective act together so that all the kids who live within appropriate home school boundaries are guaranteed good educations in the?

There’s a basic problem with random assessment in this type of a study. One group got what they wanted-to leave the traditional schools-and the other did not. Might this color their attitudes toward schooling? Might some of these students become discouraged by having their hopes denied?

Hoxby’s so-called “study” is discredited up front by the fact that her primary public identity is as a longtime advocate of charter schools, vouchers and other “free-market solutions” in education.

Just for that reason, the so-called “study” should be treated as an advocacy paper, not as impartial academic research. It’s a major breach of journalistic standards and ethics for the media to cover it as though it were credible academic research, given Hoxby’s open partisanship in favor of charter schools and against public education. Failing to disclose Hoxby’s open partisanship is even worse — the media screwed up big time on this. But any mention of this so-called “study” should include that disclaimer in big red letters.