Network News

Jay Mathews is looking for schools that qualify for next spring's America's Best High Schools list. If your school has not been contacted by Jay, and if it gave at least as many Advanced Placement, International Baccalaureate or Cambridge tests in 2010 as it had graduating seniors, please fill out this form and email it to Jay as soon as possible. Deadline is around March 1, but why wait?

USA Today series forces look at cheating

By
Jay Mathews

The Los Angeles Board of Education shocked the city, and much of the education world, last week by ordering six charter schools shut down after a charter official was found to have orchestrated cheating on state tests. It is rare for a school board to close that many charters at once. Even the local teachers union, often hostile to charters, advised against it.

But more surprising, and perhaps a sign of a significant shift in the national debate over testing, is the fact that the jump in scores at the Crescendo charter system was investigated at all. USA Today, in a series of stories launched this week, has compiled nationwide evidence of inexplicable test score gains, followed by equally puzzling collapses, that experts say suggest cheating but are ignored by the officials responsible for those schools.

Looking at test results in Arizona, California, Colorado, Florida, Michigan, Ohio and the District, the newspaper found 1,610 examples of grades at schools that increased three standard deviations or more over the average statewide gain on the same test. That means the students in that school and that grade “showed greater improvement than 99.9 percent of their classmates statewide,” the story by reporters Greg Toppo, Denise Amos, Jack Gillum and Jodi Upton said.

In 317 instances, USA Today found similarly large year-to-year declines in entire grades at schools. What could cause that? One possibility was that school officials found ways to inflate scores one year, then lost the ability or the will to distort the numbers the next year. It would be unfair to say educators broke the rules without examining the evidence, but that is the point. The school districts could not be bothered to take a look.

If we are going to use standardized tests, and I don’t see any alternative until we find a better way to measure student achievement, then we ought to take seriously the possibility that some schools are making their gains look much better than they are. USA Today cites several examples of schools that failed to investigate glaring cases of possible score inflation.

I think this is important, but I have a bias. My wife Linda Mathews, USA Today’s senior projects editor, conceived and edited the new series, “Testing the System.” Her 13 reporters, it seems to me, have gone deeper into this issue than anyone before, but readers should check out the stories and decide for themselves.

The initial installment acknowledges that sharp gains in achievement can be achieved honestly. It quotes Mike Feinberg, co-founder of the KIPP charter school network and a key subject of my last book, saying that “remarkable growth” is possible with “great teaching and more of it.” USA Today doesn’t say so, but I know that test score gains at one of Feinberg’s schools in Houston and another KIPP school in the District have been investigated and found to have no irregularities. Such is not the case in several other schools reported by USA Today.

At Portsmouth West Elementary School in Ohio, third grade math scores ranked in the 21st percentile in 2007, but in 2008, when those same students completed fourth grade, they soared to the 94th percentile, then declined the following year when they were in fifth grade. The school superintendent attributed the big jump to regrouping students by ability and adding an extra teacher to help with low-performing students.

At Stanton Elementary School in the District, the newspaper said, fourth grade math scores took a big jump, but there was something odd about the test papers. Erasure rates in that grade on that test were ten times the district average. “On a math test administered to 20 students, 345 answers were changed—97 percent of them to the correct answer,” the newspaper said. USA Today obtained D.C. school documents through a Freedom of Information Act request that revealed those scores were investigated but no violations were discovered. There was no explanation for the remarkable number of changes from the wrong to the right answer. The only indication of any action taken was a note saying that one teacher, not identified, was barred from proctoring tests in the future.

At Charles Duval Elementary School in Gainesville, Fla., fourth graders were in the 5th percentile in 2005 in math but in 2006 as fifth graders they reached the 79th percentile. The next two fifth grades also soared, the group in 2008 reaching the 91st percentile. But in 2009 fifth graders were back down in the 1st percentile.

State officials stepped in, USA Today said, but not to investigate if the high scores were falsely achieved. Instead, they said they were just there to get the school back on a upward track. The deputy superintendent for the school district attributed the drop in scores to “the sudden death of a beloved math teacher and the district’s failure to screen adequately for kids who needed extra help in reading,” USA Today said.

That is not, to say the least, a satisfactory explanation. Feinberg at KIPP has never seen such a downturn at any of his schools. When big gains are followed by big losses, he told USA Today, you have to ask “what were the adults doing that might not have been . . . ethical?”

It’s a good question. The many instances of hard-to-explain test score jumps in that many states suggest that school districts should spend some time and money finding out why. Once they have read the USA Today series, the many people who oppose using test scores to measure achievement are likely to adopt this issue as one of their principal arguments. I don’t agree with them on tests, but I share their concern about results that don’t make sense.

Up to now nobody wanted to get involved. School officials say to themselves, why buy trouble? How does it help us to investigate scores that make us look good? The answer is: If someone is putting out dishonest numbers, you and your students, as well as parents and taxpayers, are going to pay for it eventually. USA Today has more stories coming. Hopefully they will inspire a new constituency demanding a hard look at very weird numbers.

As you said, the test served the purpose. Where would students be without the test? Would educators inflate grades to move kids forward? Not blaming educators totally, but external testing provides the wakeup call for everyone.

Students cheat every day and think nothing of it.
Teachers lower their standards so everyone can pass.
Administrators inflate grades to make their schools look good.
Local officaials look the other way when they shouldn't.
Political officials lie about each other - Republicians and Democrate alike.
No one is held accountable.
And the beat goes on!!!!

The microscope under which these tests are placed lends itself to a host of manipulations. In high schools, the attention placed on tenth graders is disproportionate and disturbing. I recently visited a charter school where DC CAS signs dominated every wall in every classroom. Because high schools are evaluated based solely on these test results, the pull-out strategies, extra tutoring, Saturday academies, test taking seminars, trip incentives, and other lollipops extended to this one grade and this one test is almost laughable. After the test is over, the spotlight on these students moves on to next year's crop.

Standardized tests have a role to play in assessing student growth and achievement, but so do other less "scientific" methods like portfolios and more rigorous authentic assessments based on work produced versus "bubbles filled" correctly.

My students recently participated in such an assessment, and, while their work might very well increase their test score, it was neither the point nor the goal. Call it education.

2000 instances of cheating is a lot. But it's a blip in the 100,000-plus grades of schools in the 7 states studied, so USA Today is right to focus on what this says about the state officials who didn't care enough to investigate.

So: What might be a way to identify instances of statewide cheating by state officials? How about this: Compare school performance on state tests versus AP tests. The cheaters are those states in which schools score much higher on state tests than AP tests. Those states have designed dishonest state curricula and tests to deceive the public about student achievement. The non-cheaters -- those states in which schools score much lower on state tests than AP tests -- should be especially commended.

It's pretty obvious that where average standardized scores move up in by two sigma or more, cheating by school personnel has probably occurred.

The only way to stop such cheating is to keep the tests away from school personnel until they are administered by outside protors. Then have the tests taken away immediately after the testing by the outside proctors and given to disinterested graders.

I hate to say it but this is just another example of the general incompetence of the educational establishment in this country. Practically every profession or business, other than education apparently, recognizes that "cheating", in the broadest sense of the word, is something that is almost certain to happen if proper monitoring and control procedures aren't put in place. These procedures, once in place, also frequently serve purposes other than the primary purpose. A simple example in another field would be the cash-out procedures at the end of a shift at a retail establishment. In these procedures the register is "closed out" - the money is counted and matched against the register totals for the previous shift. There generally is at least 2 two people involved in this process, and sometimes three. If there is a discrepancy it frequently is resolved on the spot, if possible. It's essential that there be a division of responsibility so that people won't be in a position where they can cover up their own mistakes or bad behavior. These procedure also help to identify employees who are bringing in a lot of money and can be matched against inventory to determine what is selling and when. Since it's pretty clear to me that we are going to have to be making increasing use of standardized tests to measure the performance of teachers, schools, and school districts as well as students we can't rely on these entities to be administering these tests without a lot of monitoring and controls. Jay, somebody in the WP business/accounting unit could probably do a much better job of explaining this to you than me. Also, there are fairly simple statistical techniques for spotting cheating, if the tests are designed and graded properly. I'm sure that if you contact a statistics professor at one of your local universities they could explain to you how that works.

Some researcher might compare the frequency of such statistical quirks for individual schools on NAEP, a low-stakes (for schools) standardized tests, with that for schools on their local, high-stakes tests. It is obvious that a small number of statistical spikes is to be expected; but do the actual numbers discovered match those predicted?

When high stakes testing results in punitive results, people will cheat. We surely need to look at why we test and how we analyze the results. Are they a means to a punishment or to improvement? Have we lost sight of the reason for testing? Are we using the results productively? Perhaps a re-examination of our system is called for at this time.

Business is booming at this company which correctly states, "Cheating on tests isn't the exception anymore; it has almost become the rule."
http://www.caveon.com/

The higher the stakes, the more cheating there will be. The bubble-test result now reigns supreme and is the exclusive indicator which determines survival.

Griping that the school districts are not monitoring things enough shows very limited thinking to me. Not enough money will ever be set aside for keeping on top of regulation, enforcement, and oversight; no one wants to pay for what it would take to have an adequate number of investigators.

I'm just so thankful that the press is finally waking up to the fact that cheating on these tests is rampant and frenzied. Strangely the evidence has been there for all to see, and yet it's been ignored for so long. Why?

As others have said, if these tests are to be "high-stakes" they must be different each year and administered, collected and graded by a disinterested outside party. If a district cannot afford to do this, then it can't afford to administer the test.

Here's a fairly easy (and inexpensive) way to tell if test results are legitimate: Go into a classroom of students who scored really high on a standardized test and ask them to write a composition. If the scores are legitimate, the children's writing should be at grade level or above.

I taught for many years in a very low-income school. During this time I noticed that the parents of the high-performing students were often the ones to transfer their children to parochial, magnet, or private schools. Many just moved to "better" (i.e. more affluent) communities or sent their children to be babysat by a relative. (In my area it's legal to enroll your child in the school closest to babysitters or child care centers.)

Because the students transferring to these schools almost always had high test scores to begin with, the receiving school almost always had high test scores as well. To put it more succinctly: Any school that has a select student population (even if it is selected by parents) will have higher scores than a nonselective school. This applies to any school, from preschool to college.

Bald-faced cheating has been well-known and well-documented since at least the early 1970s. I first spoke to a Baltimore Sun columnist about such a problem in 1972. (He massively bungled what I told him about a school he had long been praising in print.) I did several op-eds for the Sun about such matters during the mid to late 1970s.

I'm speaking here about outright cheating, not about "teaching to the test." I'm talking about changing answers on answer sheets after the testing is done. I'm talking about teaching the students all the answers before they take the tests.

Everyone has always known about this. Why must journalists feign shock when news of outright cheating surfaces again? The guild does this every time.

By the way: Did you believe Michelle Rhee's glory tales? To anyone with an ounce of sense, her tales were always extremely implausible. Why did scribes feign belief?

Related:

When the state of Virgina conducted a state-wide test score scam in the mid-2000s, the Post refused to report it, even after the head of the state school board acknowledged that it had occurred.

Just last summer, the state of New York blew the whistle on its own multi-year statewide scam. The Post still hasn't reported this major act of (apparent) fraud.

Eek! A mouse! In fact, these scams have been conducted ever since "accountability" began to be tied to standardized testing in the early 70s. The scams began on the classroom or single-school level, then spread all the way up to the state level.

@bobsomerby,
In July 2010 many schools in NYS received double digit drops in the percentage of students receiving a score of proficient or better on the exams used for accountability purposes. The reason for this is that in prior years too many students who had received scores of "proficient" needed remedial work in their freshman year of college, so they raised the scores needed to get a proficient rating. I don't believe there were any allegations of fraud other than by the anti-standardized testing, anti-school reform crowd. I don't know if this is what you're talking about. If it's not could you be more specific, preferably with links. If it is what you're talking about could you provide a link to an objective source proving fraud. What happened in New York is what people should WANT to happen. We should constantly be measuring what's going on in schools and making adjustments to the way we teach our kids when measurements(ie standardized tests) show the methods we're using aren't working, and adjusting the measuring instruments (tests) when we have reason to believe they're not working properly.

Susan Edelman's recent report in the New York Post describes the process by which the state tests got "too easy," forcing the creation of that new set of tests. Which of course produced that precipitous drop in statewide passing rates.

Many people understood, through the years, that (at best) something had gone badly wrong with the tests or that (at worst) the tests were being scammed--were deliberately being made more easier.

Despite this, the NYC people (Bloomberg and Klein) kept trumpeting the rising test scores as proof of their ginormous success. I'd suggest a fraud in the test-making process, and in the self-serving proclamations.

I'll recommend that you read Edelman's account of how the state tests got so easy. (She writes about the statewide tests in lower grades and about the Regents exams.) This puts a lot of meat on the bones of an earlier, strikingly fuzzy report in the New York Times.

This has the obvious shape of a fraud, involving a very important state program. There was a somewhat similar statewide fraud in Virginia a few years ago, artificially driving up passing rates at almost all schools. You never read about that one either. Regarding these topics, the mainstream press corps very much tends to keep its trap shut.

One other possibility that hasn't been mentioned in this thread yet (apologies if I missed it) is that, at least in some states, the tests themselves could be flawed to the point that year-to-year comparisons are unreliable.

I suspect there's cheating, yes, but I also tend to suspect there are issues with the tests themselves—and I think the testers themselves have the same worry. When I lived in Florida, for example, the folks in charge of testing for the state refused to release any of the questions for reliability checking—a clear yellow flag, if not a red one.

@bobsomerby,
Thanks for the link, it's greatly appreciated. I've gotten use to people not responding to my requests for links or providing links to obscure blogs by people with axes to grind. I don't exactly have a high opinion of the NY Post, but this article looks legit. I live in Albany, NY and this issue got glossed over in the local paper. If the worst turns out to be true about this, then in my mind it is about on the same moral level as the judges in PA that were taking kickbacks to send kids to for profit detention centers who had committed relative minor offenses that would have only placed them on PINS anywhere else. The interesting thing is that Rupert Murdoch recently hired Joel Klein and purchased a educational software company specializing in testing and data systems that I assume Klein will be intimately involved with. Hmmmm! I wonder what effect this will have on any plans Bloomberg might have about running for president.
Here's a related aside for you. About 6 years ago I found out that the ACSD was allowing teachers to refuse to distribute textbooks to certain students, even when the students begged for them. The ACSD was and still is very much under the gun for poor performance. I gave the school district ample opportunity to start distributing the books, but when they refused to do that I notified the local paper, with a cc: to the school district. When I did that the school district immediately started distributing the textbooks(for that year - they went back to their old practices the next year), but the newspaper(the Times Union) didn't run an article about it - that year or in subsequent years when the practice was resumed. Eventually I got so sick of fighting the school district and getting no help from the Times Union that I filed a civil rights complain with US DOE, which made it past their first cut of cases they'll pursue. Wonderful press we have in this country. Hours and hours about Charlie Sheen's drunken escapades and tweets but almost nothing about rampant test frauds and immoral textbook distribution policies.

Maybe it's time to go all Jaime Escalante on these suspicious results and have the kids re-take the exams under strict outside monitoring. If the gains are real, the students should do well on the re-test.

Isn't it amazing how Jay and his wife can run with this "scandal" when it comes to administrators and fudging exam results - but when the CEO (Director General) of IBO International Baccalaureate Organization) was caught plagiarizing a speech and IB was caught plagiarizing Wikipedia for the mark scheme on one of its exams, Jay never managed to say a word. Not a peep. Nope. He just went on advocating IB as "a bit better" than AP, plugging his books and advancing his Progressive ideology. Mind you, I am not endorsing the manipulation of test scores by administrators. But Mathews' "selective" consternation is disingenuous. If these schools are going to embrace strict "no cheating" policies for the kids, then the grown-ups better knock this garbage off! And those who partake in this sort of duplicitous and intellectually bankrupt behavior should be fired! That includes kicking out fraudulent programs like IB as well as administrators who can't report the truth.

for lisamc31---Here at the Struggle blog we try to go with items that have news value. If I policed alleged plagiarism on every speech never heard by my readers delivered by persons they never heard of either, I wouldn't have much space for stuff that interests them. And why would I want to steal yr thunder? We want to make sure everyone knows that for anti-IB stuff, they should check in with truthaboutIB.

For CrimsonWife--- I agree entirely. Retests are the way to go. I suggested that when the problem with DC tests first came up.

for georgia198305--- I have written often that I think vouchers are a dead end. There will never be enough empty spaces in private schools to help very many low income kids. I don't see anything philosophically wrong with them, but I would prefer that we encourage charters instead, which can grow more easily and also can be closed more easily if they dont work.

I'm jes a little ol' blogger, not a paid journalist with a major newspaper. Apparently the UK and Switzerland thought the story had "news value". Considering you are the #1 media mouthpiece promoting IB in the U.S. and have spent years singing IB's praises, a little journalistic integrity seems in order.

Or you can keep promoting the IB scam and ignore the actual anti-intellectual leadership and practices of the organization.

Retests are NOT the answer. Why punish the kids for the crimes of the adults? VOID the results from that year and make sure strict oversight is in place when it comes time to report the next year's findings.

Congratulations to Mrs. Mathews, special projects editor at USA Today, for tolerating such a credulous spouse as a journalist all these years.
When I read just the first 2-page chapter of the uncut version in USA Today, I was immediately impressed with that paper's commitment to the story.
I wish Mr. Mathews understood that when I am next before a health insurance underwriter, I will do whatever is legal and undetectable to present myself as having better health metrics (ie. BP, BMI, cholesterol) than I normally present, so the insurance company incorrectly estimates its risks with me in my favor.
The prediction of "expected growth" in school tests is based on true scores, just like the insurance underwriter will presume to collect from me. You do not want to require an econometrician to make sense to you when the scores include a large error component. You would not want your children following the advice of pediatricians who measured your grand-children's growth in such a limited fashion and so unreliably as school systems have been "measuring" students and teachers.

---
Yes, dfbdfb, teachers are seeing expectations of "growth" calculated from a component they have no control over: last year's test scores.
Is the solution to pretest in the same year? No. We may allow of journalists but do not expect of our teachers that they are so careless of their own future as to heavily prepare students for pre-tests which they DO have greater control over. So, strategy now dictates what learning does take place in the first month or two months of the school year.

I don't recall, from the USA Today story, any report in which classes with outstanding gains were asked to and then did support them with portfolios of student class and homework.
That is what the USA story is about, thus far: absent interest of schools in doing anything about the fraud. What may be educational neglect is being covered up.

How would you explain the retests to the students? The adults cheated so they get a do-over? The adults may or may not have cheated, but you did very well and we all know you aren't really that good so we want to retest until you get dumber? The adults cheated so you have to pay the price? Or, the result of doing well is an accusation of cheating?

Are these the lessons we want students to learn from testing?

Why don't we just admit the entire testing thing was ridiculous to begin with and the tests are poorly made up and don't measure what we want them to?

If I'm not mistaken, her famous claim is probably more extreme than the claims described in the USA TODAY report.

THis takes us into the weeds a bit, but I think this is accurate:

USA TODAY is talking about the percentile standing for a particular grade group as compared to all other grade groups in a state.

Rhee was talking about the percentile achievements of her individual kids.

If 90 percent of a grade group scored in the 90th percentile or above, I would guess that this grade group would be close to the 99th percentile among a state's grade groups. That is to say, there are very few grade groups where so many kids score that high.

lissmc31's idea of voiding the scores for that year is not bad either, as long as they are not high school scores that decide if the kid will graduate or not. As for explaining a retest, you could say exactly what the College Board says when it invalidates AP scores: A question has risen about the validity of the scores so we are having the students retake them. Actually, with AP, there are other choices since the students paid for the tests and the tests can have some consequences for them. They can also elect not to retake them and have their scores canceled, or (at least in the 1980s, the last time I looked into this), submit evidence of their validity, although in the case of Escalante's kids the College Board did not appear to consider seriously the evidence they provided.

for lisamc31---So it appears no journalists in the US picked up on this big story, even the uncorrupted ones? I rest my case. None of us wanted to ruin yr scoop.

The kids don't care about the results, but they care plenty about having to take the tests. I just finished substituting at a school where the first item on the agenda was two hours of standardized tests. The kids who got done rapidly were bored stiff for the rest of the time. The two hours called for rearranging the rest of the schedule, so things they were looking forward to didn't get done, the kids who had to be tested separately missed regular classwork, and by the end of the day they were misbehaving and openly defiant. (Not because I was a sub; I was working as part of a team, and even the regular teachers were asking the kids, "What's wrong with all of you today?" and even the usually well-behaved kids were fed up and cranky.)

Tell them they have to go through the whole thing a second time because of adults' misbehavior, and few of them are going put out any effort. And what if it happens again? Sooner or later even the most test-anxious kid is going to wonder why he should even try if he can't trust the adults to do their part. (And this was middle school--high schoolers who are more cynical and sometimes well informed about the testing controversy are likely to fill in the answer key to form a four-letter word!)

Cheating was rampant in Houston during the Rod Paige era when NCLB was incubating. It's been investigated and written about extensively in Houston. NCLB was based on data created by fudging test scores, fake dropout and enrollment data, and by outright cheating from its inception. Most of the cheating was done by administrators who handled to tests after teachers handed it in or by teachers who were threatened by principals.

The national press was asleep at the wheel back then. Now it's waking up and it's shocked, shocked to find that cheating is going on in here!

But none of that really matters as long as businesses make money off the tests. That's the real point of it all, anyway.

A lot of cheating takes the form of drilling the students on the exact items on a test. Although the teachers might not see the test they are about to administer, they know that it is the same, or nearly the same, as the old test. Because these old tests are accessible to teachers, it is not "cheating" in the legal sense; that it, teachers would not be fired for doing this. However, teaching the exact items totally invalidates the results of the test and most educators know this. So in this sense it is cheating, but I am surprised to discover that many people don't agree with this. (See the Educated Reporter blog, March 10) Drilling students on tests is the strategy used by almost all schools in very poor areas. When such schools give the students a completely different test, scores almost always go down dramatically. To put this more bluntly, very little learning has taken place.

But as aed3 states, it doesn't matter anyway because this "reform" movement is about money and not about educating poor children.

We encourage users to analyze, comment on and even challenge washingtonpost.com's articles, blogs, reviews and multimedia features.

User reviews and comments that include profanity or personal attacks or other inappropriate comments or material will be removed from the site. Additionally, entries that are unsigned or contain "signatures" by someone other than the actual author will be removed. Finally, we will take steps to block users who violate any of our posting standards, terms of use or privacy policies or any other policies governing this site. Please review the full rules governing commentaries and discussions.