Newsweek’s Ranking of High Schools Is More Advocacy Than Science

A Newsweek ranking of the nation’s top 1,200 public high schools received billing on the magazine’s cover this week, and newsoutlets around the country reported on local schools’ positions on the list.

But the ranking method, based entirely on rates of student participation in certain standardized tests, is questionable. The ranking was devised, and is still implemented, by a Newsweek contributing editor who says he is using the ranking as an advocacy tool, to encourage more schools to push students to take these tests, and the classes that typically precede them. “I have yet to think of ways for schools to game this list without improving students’ lives,” rankings creator Jay Mathews told me. “I designed the list so that would happen. I’m hoping people game it in any possible way.”

Newsweek ranks public schools by what it calls a Challenge Index: the total number of standardized tests for college-level courses — Advanced Placement, International Baccalaureate and Cambridge tests — taken by all students at a school last year, divided by the number of graduating seniors. Schools with an index above 1 were included, though the magazine excluded 19 schools “because so many of their students score well above the average on the SAT and ACT.” (That included my alma mater, Bronx Science.)

The magazine has run the rankings occasionally since Mr. Mathews devised his system a decade ago. The longtime education reporter for the Washington Post (a sister company of Newsweek) told me he was inspired by research for his book, “Class Struggle,” in which he examined what factors made schools successful, and determined that access to college-level coursework is critical.

Mr. Mathews said the ranking’s simplicity is a virtue, making it easy to understand — and to criticize, as many education observers have done. “A rating system that rewards quantity without measuring quality produces some truly bizarre results,” education columnist Michael Winerip wrote in the New York Times last year. I asked Mr. Mathews why he doesn’t combine quality and quantity, by taking into account how students actually score on the tests. He said the magazine does display the percentage of students who achieve at least a certain score, but those aren’t used for the rankings.

The raw number of students taking tests shouldn’t be isolated from their performance on those tests, and the quality of the related courses, said Trevor Packer, executive director of the Advanced Placement Program for the College Board, which administers AP tests. Of the Challenge Index, Mr. Packer told me, “It is effective at generating discussion about access to rigorous curricula, but should not be deemed a thorough or comprehensive means of evaluating school achievement or excellence.”

To Mr. Mathews’s credit, he addresses a host of questions and criticisms about his rankings in a lengthy Newsweek FAQ. (A Newsweek spokeswoman referred questions about the rankings to Mr. Mathews.)

All rankings involve some arbitrary choices about what to include, as I’ve pointed out frequently (most recently here). Generally, rankings are produced with the goal of measuring something, not changing it. But Mr. Mathews said he sees the rankings as a “way of giving recognition to schools that are not restricting access” to advanced courses. He added, “I presented the list as a way of showing schools the light.” In a Washington Post column Mr. Mathews wrote about the rankings this week, he likened them to a carrot encouraging schools to offer more college-level courses, saying those that do “ought to be recognized and rewarded.”

Mr. Mathews said he’s convinced the measure is a good one: “There are only a few things I’m very confident about. In those cases, I do act as an advocate.” Mr. Mathews acknowledged that it’s unusual for a reporter, writing in a news magazine, to present what is framed as an objective ranking of “America’s best high schools,” yet use it as an advocacy tool. “I realize this is a very odd position to be in,” Mr. Mathews told me.

About The Numbers

The Wall Street Journal examines numbers in the news, business and politics. Some numbers are flat-out wrong or biased, while others are valid and help us make informed decisions. We tell the stories behind the stats in occasional updates on this blog.