Rick Casey: Lies, damn lies and magic statistics

Commentary

RICK CASEY, HOUSTON CHRONICLE |
June 21, 2010

If you were depressed by last Sunday's discussion in this space of the fact that Texas students could pass the TAKS test with scores as low as 44 percent, consider this:

Even with such low standards, the Texas Education Agency decided to rate hundreds of Texas schools and scores of Texas districts as "academically acceptable" last year — the lowest "passing" category - by counting thousands of students who flunked the TAKS as passing it.

This academic alchemy was achieved through a statistical exercise called the "Texas Projection Measure."

This complex formula, based on statewide TAKS scores from the year before, is said to identify students who failed this year but are likely to pass next year. Or maybe the year after that. Or possibly the year after that.

Education reporter Ericka Mellon last month wrote about the fact that HISD's long-troubled Sam Houston High School won recognition from Gov. Rick Perry and even Education Secretary Arne Duncan for improving its rating. She noted that the school was bumped up one level because some of its students who failed portions of the TAKS test were counted as passing because of the "Texas Projection Measure."

That recognition drew the attention of Houston State Rep. Scott Hochberg, vice chairman of the House Public Education Committee. Hochberg, considered by members on both sides of the aisle to be something of a guru on Texas education, decided to figure out just how the Projection Measure works, talking to experts and poring over statistics at TEA's website.

Inflated numbers

The first thing to understand is that this bureaucratic boost, which began last year, is no small matter.

"It's not nibbling around the edges," said Hochberg.

For example, TEA reported that statewide the number of "exemplary" campuses, the highest rating, more than doubled from 1,000 in 2008 to 2,158 in 2009. But without the statistical projections that some failing students would later pass, the increase would have been only 44 campuses.

At the other end of the spectrum, the TEA reported that the number of "unacceptable" campuses had increased by 43, from 202 to 245. But without the magic of statistical projections, the real increase was 401 "unacceptable" campuses - almost 10 times the massaged number.

Hochberg said the new system is disturbing not only because of the large number of schools that are receiving upgraded ratings under it, but also because of the statistical formula itself.

You may have assumed, as I did, that it projected a student's future success based on improvements that student had made over the past few years.

It doesn't. It's based on a statewide analysis of all students indicating that at certain score levels students who failed math but passed English (or vice-versa) on average went on to pass both.

So it isn't a measure of progress, but of the statistical likelihood of progress.

Like Enron's legendary "mark-to-market" accounting, which booked future profits in current quarters based on a variety of fanciful assumptions, it is counting eggs before they hatch.

Really 'unacceptable'

Hochberg notes that a similar analysis might also show that certain students, statistically speaking, are heading in the other direction.

So Perry and Duncan celebrate the "improved" performance of our schools. But HISD officials, to their credit, are skeptical of the enhanced scores. An independent study they commissioned and released this week, based on actual performance rather than on statistical "projections," shows that about half the ninth-graders of the class of 2005 enrolled in college and less than a fifth of those graduated in four and a half years.

Nearly a third of those ninth-graders dropped out. HISD trustee Anna Eastman called it "unacceptable," just the opposite of how the TEA rates HISD.