NYC students post double-digit test gains; statisticians are dubious

No one was surprised when Chancellor Klein announced today that the city’s students posted dramatic gains on state test scores this year. Charting a clear trajectory of improvement has been fundamental to his reforms. This year, he announced, nearly 80 percent of 4th graders and 60 percent of 8th graders passed the state math test, and about 60 percent of 4th graders and 40 percent of 8th graders passed the state English test. Gains in the last six years, the DOE points out in its press release, range from about 15 points in 8th grade English to more than 30 points on math tests at all levels.

Even before the mayor made his announcement this afternoon, discussion had begun over whether this year’s test scores are a sign of victory, as the mayor believes, or of score inflation and manipulation. In today’s Sun, Elizabeth Green speaks to statisticians who warn that, for many reasons, large-scale score increases are not always to be believed.

But we do have some evidence that suggests that this year’s scores aren’t entirely fabricated or based on tests that have gotten easier or easier to prepare for. New York City’s test scores have risen faster than scores in the rest of the state, although some cities did better than New York this year. And Kevin Carey at the Quick and the Ed points out that even though New York City’s performance on the National Assessment of Educational Progress hasn’t paralleled its improved state test scores, the city does outshine other urban districts on the NAEP. There are good things happening in many New York City schools where attention wasn’t always given to even the basics. And there are also a great number of things happening in New York City schools with the express purpose of raising test scores. So a jump in test scores shouldn’t be surprising.

What does the improvement actually mean? Statistically dubious as they may be, single-shot test scores are used in New York City to determine children’s promotions, teachers’ bonuses, principals’ job security, and schools’ report card grades — and, consequently, the schools’ very existence. And who’s going to look deeper when test scores keep going up and up, fewer children are retained, bonus money is distributed widely, principals land performance bonuses, and schools seem to be doing better than ever before?

The possible perfidy of state test scores in this city is not out of the ordinary. Indeed, as Green reports, studies have found that two-thirds of states post test score increases that outstrip their increases on national and international tests, suggesting that test score engineering may be taking place. But with all eyes looking to New York City’s radical and far-ranging incentives programs, misleading data here can have wide-ranging impact. When George Bush became president, he named Houston superintendent Rod Paige as his Secretary of Education, citing among Paige’s successes a dramatic reduction in Houston’s dropout rate. It was only after Paige engineered and began implementing the No Child Left Behind act that it became clear that Houston’s books had been cooked. We should be pleased that on one important measure — test scores — the city’s schools appear to be improving, but at the same time, we must keep Houston’s lessons in mind.