A Rank Exercise
Jay Mathews' method for ranking America's best high schools is so narrow it may actually be misleading.

By Andrew J. Rotherham and Sara Mead
Friday, June 22, 2007
5:56 PM

Until a few years ago, America's elementary and secondary schools generally escaped our national obsession with lists. Almost every week another ranking of best communities, most beautiful people or top hospitals is published.

But in 1998 Newsweek, which is owned by the Washington Post, began publishing a list of "The 100 Best High Schools in America." The ranking is based on "The Challenge Index," a measure developed by Washington Post education reporter Jay Mathews. The list, published annually the past few years, has become increasingly influential. Other media outlets now cover it like a horserace, and high schools all over the country are reacting to the scrutiny.

Unfortunately, the Challenge Index is a flawed proxy for America's "best" high schools. Using publicly available student performance data, we have found that many schools in Newsweek's ranking have high dropout rates or glaring achievement gaps between racial and ethnic groups. At the same time, many schools that fail to make the Newsweek list may be doing a better job educating all of their students.

The Challenge Index is a simple measure: It's the number of Advanced Placement (AP), International Baccalaureate (IB), and Cambridge tests a high school's students take, divided by its number of graduating seniors. This simplicity is both its primary virtue and fatal flaw.

Our research shows that the Challenge Index's methodology is far too focused on a single, narrow indicator. While education policy is increasingly focused on closing achievement gaps, the Challenge Index pays no attention to differences in achievement or AP- and IB-test-taking rates for students from different racial and economic groups, nor does it ask whether high schools are achieving their most fundamental goal: enabling students to earn a diploma.

Mathews focuses on AP and IB because high scores on these tests are supposed to indicate that students have mastered challenging, college-level work while still in high school. He believes the percent of students taking the exams is the best available measure of how effectively a high school prepares its students for post-secondary success.

Education analysts generally agree that America's high schools need to increase their academic rigor to better prepare students for post-secondary opportunities. Expanding AP and IB courses is one way to do this, and there is evidence that students who take these courses do better in college. But AP and IB test taking is only one measure of school quality, and it ignores other vital indictors.

Consider Eastside High School in Gainesville, Fla., 17th on this year's list. Eastside did not make "adequate yearly progress" or AYP under the federal No Child Left Behind Act in 2006, although only about half of its students would have had to pass Florida's math and reading tests to meet such performance targets. Eastside also received a "D" on Florida's state school accountability system. The achievement gap between Eastside's white and black students is enormous: 80 percent of white students are proficient in reading, but only 13 percent of black students. In math, 89 percent of white students, but only 15 percent of black students, are proficient. The achievement gap for economically disadvantaged students is just as big. And Eastside's black and disadvantaged students perform well below state averages for their peers.

Or, take Southside High School in Greenville, S.C., 76th on this year's list. Southside did not make AYP in the 2006 school year and is rated "unsatisfactory" under South Carolina's state accountability system. The majority of Southside's students are low-income and minority students who tend to do worse on state tests. But according to South Carolina's state school report card, Southside's students do worse than students in demographically similar schools on state end-of-course tests in math and English. And it graduates only 59 percent of students, also worse than state averages.

How do such schools find their way into a ranking of the very best high schools in the nation? Ironically, the problems these schools face may actually help them reach their ranking. For instance, Southside's low graduation rate means the number of AP and IB tests its students take is divided by a smaller number of students to reach the final ratio. This problem likely accounts for the inclusion on the list of many schools with magnet programs within them: pockets of AP and IB test taking surrounded by a school with dropout problems. Sadly, this is a metaphor for America's achievement gap problems more generally.

We don't dispute that there are many great schools on Newsweek's list, and good things are happening inside some schools -- including the two we highlight here -- that are not posting great numbers overall. But it's unfathomable how anyone could look at the performance of Eastside, Southside or some other schools on the Challenge Index and say they deserve to be ranked in the top fraction of 1 percent of American high schools, which is what Newsweek's top 100 ranking signifies.

A successful high school should show high levels of student achievement, graduate almost all of its students and not let any demographic subgroup lag far behind. National education policies, and increasingly state policies, reflect these values. To be sure, graduation rates and student achievement are hardly the only indicators of a school's quality. They are, however, reasonable minimums.

But Mathews argues that such criteria automatically exclude virtually any school serving large numbers of poor or minority students. Yet the Newsweek list includes at least two schools that meet our criteria while serving majority low-income and minority student populations: The Preuss School in La Jolla, Calif., ninth on Newsweek's list, serves only low-income students, mostly from racial and ethnic minority groups, but virtually all of them graduate, and they do better than the statewide average on state accountability assessments. And No. 39 on this list, YES! College Prep, part of a network of high-performing public schools in Houston, is more than 70 percent low-income and more than 97 percent black and Hispanic, but its students exceed statewide averages on state tests, nearly all of them graduate, and all graduates go on to a four-year college or university. Other schools around the country, for instance Boston's MATCH Charter School, the top open-enrollment high school in that city, also show that higher standards are attainable even with students who are more challenging to educate.

The lack of schools that meet more ambitious criteria speaks to a larger problem: Nationwide, only about half of minority students finish high school on time, and a gap equivalent to about four grade levels separates white and minority high school students. That is the dismal status quo for too many of our nation's poor and minority students. Consequently, Mathew's concern about excluding schools with disadvantaged populations says far more about the need to generate a list that can run to 1,000 schools and sell magazines than it does about the validity of the Challenge Index as proxy for America's best high schools.

Mathews makes two arguments for why the Challenge Index is a valid measure of the nation's best high schools. He points out that there are few sources of useful education data in general and even fewer with which to make valid interstate comparisons among high schools. This, says Mathews, is why he relies on AP and IB tests.

But since the Challenge Index's 1998 debut, a national revolution in education data, generated by more assessments and better calculation of graduation rates and other indicators, means much better comparisons can be made. In fact, U.S. News and World Report, whose college rankings have become extremely influential (and, more recently, controversial), is considering publishing a high school ranking that takes into account the better data available today.

Mathews also argues that "best" is an inherently subjective term and what one person says is the "best" movie, restaurant, or school is different from someone else's judgment. He's right, but there are limits to the elasticity and reasonableness of "best." A list that includes a school that only graduates 64 percent of its students, as does Wilbur D. Mills High School in Little Rock, Ark., 36th on this year's list, simply exceeds those limits.

The nation can do a much better job of educating low-income and minority students. And Newsweek can do a better job of recognizing those efforts than its ranking does today.