Evidence of effectiveness proves elusive

Alec MacGillisSun Staff

Second in a series

TALLADEGA, Ala. -- Graham Elementary School has no shortage of problems, so it might seem surprising that when it was awarded a $100,000-per-year federal grant earlier this year for "comprehensive school reform," the school used most of the money to buy educational video games.

The school, like 16 others in Alabama, purchased dozens of Sony PlayStations and, to go with them, reading and math programs from an education software company called Lightspan. Designed like video games, the programs came with colorful graphics, point totals and names like "Kazmania" and "The Three Decoders."

Graham Elementary Principal Frank Buck said he was persuaded partly by Lightspan's success claims at schools such as Lansdowne Elementary in Baltimore County, where 34 percent of kindergarten-to-second-graders supposedly posted major test score gains with the software.

Buck never called Lansdowne to check on this figure, but if he had, he would have learned that the company's claim was a bit of a stretch. Lansdowne Elementary administrators say that while the games were of some use, the school's gains had more to do with improved teaching, a partnership with the Baltimore Symphony Orchestra, and after-school clubs and tutoring.

"Like any company, they're going to make it look like the test scores were all because of them," said Lansdowne Principal Anne Gold. "There were many things happening here, and they were one piece. It wasn't all Lightspan."

Under pressure from the 2001 No Child Left Behind law, and from vendors using the law in their pitches, struggling schools across the country are spending heavily on education software programs that promise to raise their test scores.

But in many cases the products are unproven and the glowing claims behind them highly dubious -- often based on flawed studies or on data compiled by the companies themselves.

While federally funded studies on the impact of education technology are under way, most of the results won't be available for two years. By then schools will already have spent millions on software in their attempt to comply with the law's tough performance standards.

"We need to get beyond platitudes. We need proof," acknowledged Susan D. Patrick, the U.S. Department of Education's director of education technology. "That's one of the big issues I hear from districts and states: 'We need help with this, knowing what works.' ... There isn't anything out there now, and we realize that's a challenge."

Education software executives say they welcome objective research, but they deny making misleading claims. Mark Schneiderman, the industry's chief Washington lobbyist with the Software & Information Industry Association, said that No Child Left Behind's emphasis on accountability has made schools more discriminating in their software purchases.

"Schools are asking for evidence of products' effectiveness," he said. "They're going through a very thoughtful process when they make decisions."

Easier to take advantage

But many experts in education technology say that the 2001 law has made it easier for companies to take advantage of credulous administrators. Fast action is demanded: Schools must test all students annually from grades three through eight and show "adequate yearly progress" on scores not just school-wide, but also among minorities, special education students and other groups. Schools that fall short for several years running face funding cuts and possible state takeover.

Faced with so many requirements, few administrators spend the hours needed to sift carefully through vendor claims or hunt for better research.

"We're being bombarded; everyone says they have the turnkey solutions, but you don't have the time to review them," said Jayne E. Moore, the instructional technology director for the Maryland Department of Education. "It's overwhelming to try and discriminate. They'll provide their documentation, but how can you tell?"

The success stories are brandished with all the breathless fervor of weight-loss claims, but often they don't hold up under close inspection. Some examples:

Renaissance Learning, a Wisconsin-based software company, leads off a "scientific research case study" on its Web site by reporting that students at Sheridan Elementary in St. Paul, Minn., saw their scores on statewide math and reading tests increase an average of 43.5 percentage points over the four years since the school started working with Renaissance in 2000. Buried in the study is the fact that the school began using Renaissance's math software in only the last of the four years of steady increases.

Pearson Digital Learning trumpets, among many others, an elementary school in Hialeah, Fla. that, by using the Mesa, Ariz., company's SuccessMaker software, vaulted from a "D" state ranking to an "A" in the 1999-2000 school year. One has to look closer to discover that the school had been using SuccessMaker since 1994, long before the turnaround, which occurred in the same year as a major curriculum overhaul.

Curriculum Advantage, a Nebraska-based company, invokes the success of George Middle School in Portland, Ore. -- eighth-grade scores rose by 25 percent in reading and 50 percent in math over the 2000-2001 school year as a result of the company's Classworks software.
Left unmentioned in the company's two-page report on George Middle's gains is that it underwent radical reform prior to that year, including the unusual move to have a single teacher stay with the same group of students from sixth to eighth grade.
Also unmentioned is that George Middle has, while continuing to use Classworks, fared poorly more recently. It failed to meet No Child Left Behind's demand for "adequate yearly progress" for two years in a row and now is on Oregon's list of failing schools.
Joanne Benson, a school administrator quoted praising the software in the company's report, said it was wrong for Curriculum Advantage to cite George Middle as proof of Classworks' effectiveness:
"They're claiming a lot for themselves. As is clear now, by itself it isn't a magic silver bullet. We wouldn't have lost [the earlier gains] if it was."
Other elements are ignored
The failure to consider other factors in school improvement is a major flaw in companies' purported "scientific studies" of their products' impact, say experts in education technology. Many studies contrast the performance of one group of students using software versus another group in a previous year or a different class or school without taking into account key factors such as the arrival of a dynamic new principal or the quality of teachers.
In many cases, students' progress after using software is assessed using test questions that are virtually identical to questions used to drill students. And studies rarely track students for more than a year, giving buyers no way to know whether the software has any lasting impact on learning.
Often, the assessments are done by university researchers or private research firms that are paid handsomely -- sometimes more than $100,000.
Researchers concede that there is pressure on them to report glowing results but say they strive to be objective. In the many instances when studies show little or no impact from software, researchers say, the companies keep the results secret.
"If they aren't positive, people bury them and never report them," said Cheryl Lemke, head of the Metiri Group, a California education technology research and consulting firm. "People have a lot at stake. If it's a multimillion-dollar installation, no one wants to find out it doesn't work."
Even as they defend their research, software vendors concede that schools could find reasons to question their claims. In its 2004 annual report to the Securities and Exchange Commission, Plato Learning Inc. acknowledged "risks" in its studies, including:

"Limited sample sizes used in our studies may yield results that are not representative of the general population of students who use our products."

"We facilitate the collection and data for some of these studies."

"We select and pay researchers to aggregate and present the results."
Given such risks, the company stated, "Our sales and marketing efforts, as well as our reputation, could be adversely impacted if the public, including our existing and potential customers, perceives these studies to be biased due to our involvement." Despite such doubts, it added, "we believe that these studies accurately reflect the performance of our products."
Some of the most ambitious product claims have been made by Lightspan, a $54 million San Diego company that merged with Plato Learning last fall.
Lightspan's math and reading PlayStation games, which have been bought by more than 4,000 schools, were originally designed for home use by children, but after No Child Left Behind, the company started repositioning them for the classroom. (The company's original investors included Tribune Corp., The Sun's owner; Microsoft; Tele-Communications Inc.; and Comcast.)
In its literature, Lightspan stated that "more than 1,000 independent studies prove Lightspan products enhance student achievement, improve teacher effectiveness and help build stronger connections with families."
Promoted prominently was a 2002 "dissertation study conducted over a two-year period by Joseph Birch, Ph.D." at Simpson Elementary School in Camden, Del. The company said it "clearly demonstrated" Lightspan's effectiveness for reading and math learning.
The company neglected to mention one detail: Birch is not an independent researcher. He is the principal of Simpson Elementary, which received 16 computers, 150 PlayStations and Lightspan software as part of a $5.5 million federal grant for Delaware schools. He did the study at the same time as he was overseeing teachers to make sure they were using Lightspan effectively.
Birch said in an interview that he had felt no pressure from Lightspan to report positive results, and he saw no conflict between his research and his role in encouraging use of the software. He was somewhat surprised, he said, that his results were so prominently featured by the company, which paid his way to a convention in San Diego to talk about them.
He also mentioned something not featured in Lightspan literature: The product is no longer part of his school's regular curriculum. The school hasn't been able to afford new software or PlayStations to replace lost and broken equipment.
"We've kind of gone on to other things as a district," he said.
Software difficult to assess
Even when sound research methods are used, experts say, it is notoriously difficult to assess software. Too many factors are at work in classrooms.
Often, software programs will produce test gains simply because failing schools respond to having anything new to use and are grateful for whatever training the company representatives provide as part of the contracts.
Jackie Nunn, director of the Johns Hopkins University Center for Technology in Education, recalls a four-year "intervention" that her center did with a group of students at Baltimore's Northern High School in the late 1990s, which resulted in a 100 percent graduation rate.
Some education software was used, she says, but the students' success had much more to do with the guidance her center was giving. That didn't stop the software company involved in the project from claiming credit, she adds.
"Just providing attention to teachers and a set of kids will produce change," Nunn said. "It wasn't that little piece of technology we were using -- it didn't even work half the damn time."
Timothy Dixon, the technology coordinator at Judge Sylvania Woods Elementary School in Prince George's County, has observed a similar dynamic with programs: "A lot [of students] show short-term success simply because you're doing something different. But the question is, at the end of the year, what did they retain?"
The lack of solid research on software effectiveness has posed a problem for the Bush administration. To make up for what it calls a lack of accountability in past federal education spending, the administration required in No Child Left Behind that schools spend federal funds only on classroom approaches backed by "scientifically based research." But the software industry lobbied successfully for exempting the law's technology funding program from that standard, arguing that many programs are too new to have research behind them.
The scientific research standard still applies to software bought with the law's general funding, but schools are finding very little on the market that has rigorous research behind it. The "What Works Clearinghouse" Web site, created by the Education Department in 2002 to list legitimate studies, includes almost none for software.
To remedy that, the department is investing $10 million in 16 two-year studies of the effects of various kinds of software, in addition to $46 million in studies it is funding at the state level. Vendors are pushing to have their programs included, even as they acknowledge nervousness about being evaluated in a study out of their control.
With the results not due until 2006, though, companies are responding to the demand for "scientifically based research" simply by tacking that phrase onto older studies they had been referring to all along -- without meeting any resistance from government officials enforcing the law. Lightspan, for instance, described Birch's work as "scientifically based."
Patrick, the federal education technology director, acknowledged this is a problem. "Most of the [existing] studies don't fit the rigor" intended by the law, she said.
Once burned, twice shy
Some school officials have grown highly wary of vendor claims -- typically after finding that previous expensive software purchases didn't turn out as well as they had hoped.
In Denver, several schools purchased Lightspan games during the past five years with federal grants, with few positive results. Schools had a hard time keeping track of the PlayStations and games they sent home with students and worried that students were using the PlayStations for regular video games instead, said Kipp Bentley, Denver's director for education technology.
"They're pretty much sitting on a shelf in cabinets and not being used," he said. Schools "weren't getting a return on the investment."
Purchases of other companies' products also haven't met expectations, Bentley said, which has led the district to adopt a more discriminating approach. Principals are encouraged to confer with Bentley before making any purchases, and district educators share their experiences with products on a central list of recommendations.
Baltimore's experience
In Baltimore, administrators spent heavily on instructional software throughout the 1980s and early 1990s. In too many cases, the district now says, the software ended up gathering dust or going out of use when schools couldn't afford upgrades or maintenance.
"To tell the truth, in most schools those programs were abandoned," said Gregory J. Burkhardt, the district's director of classroom support systems. "They weren't performing as what principals saw was up to snuff."
The district now has an extremely cautious stance toward vendors -- caution hardened by recent budget problems.
At Samuel Coleridge-Taylor Elementary in West Baltimore, which used Lightspan games until a few years ago, Principal Michael L. Cheatham is using federal funds to pay for an extra teacher to fill in for absent teachers, so he doesn't have to rely on temporary substitutes, and to pay for extra staff to help two classes of emotionally disabled children.
Students visit a computer lab weekly to learn technology literacy, but no money is being spent on math and reading software such as Lightspan.
"The data will tell you that the primary factor that influences student performance is the quality of teaching in the classroom, and I want to make sure that classroom instruction is going well," Cheatham said.
There is little such wariness in Alabama, though, where schools that bought Lightspan are excited to be starting their first full school year with the video games.
Graham Elementary was one of several dozen schools invited by the state to apply for the "comprehensive school reform" grants, funding available to poor schools under No Child Left Behind for school-wide improvements beyond regular classroom activities.
Last year, at a two-day conference organized by the Alabama Department of Education, school officials learned how to apply for the grants, which pay about $100,000 annually for three years -- and are supposed to pay only for improvements backed by scientific research.
The conference served another purpose: to introduce the schools to software vendors that the state had invited to offer their products.
For Lightspan, the conference paid off in a big way. Seventeen of the 36 schools that qualified for funding decided to use most of their grant money to purchase PlayStations and Lightspan games. While most other schools went with other software programs, only two schools used the funds to hire staff to address gaps -- an instructional coach to better coordinate instruction, and part-time staff to encourage parental involvement.
In Talladega, 50 miles east of Birmingham, Principal Buck said the school will send the PlayStations and games home with students as a way to "extend the school day." The games will be given mostly to low-income students, who make up more than half the school population and live in the shotgun houses and housing projects surrounding Talladega's cluster of fine antebellum homes. On last year's test, its poor students failed to meet No Child Left Behind standards in math.
Meanwhile, Buck said, teachers would also use Lightspan during class as a way to occupy groups of students when, as is increasingly prevalent in schools today, teachers break the class into "centers," small groups each engaged in a different activity. While the teacher is instructing one group, he said, another group can use the games, which are differentiated by skill level.
Before buying the software, Buck said, he intended to call schools such as Lansdowne to check Lightspan's claims. But he didn't have the time.
"In the best of worlds I would have called them, but the turnaround time [for applying for the grant] is so short," he said.
This time, it will work
Promises of higher test scores also won over officials at Pike County Elementary School in Brundidge, in the farm country of southern Alabama. The school, a low-slung building that backs onto a mobile home park, has even greater needs than Graham Elementary: Virtually all of its 460 students are low-income, the school is on warning status for writing scores, low parent involvement is a "serious issue," and there is no art or music instruction, says Principal Donella Carter.
The school has not had the best experiences with education technology -- its computer lab for Compass Learning math and reading software has suffered repeated technical glitches. But after getting the Lightspan pitch at the state meeting, Pike County also decided to spend most of its grant on the games, in hopes they would raise scores.
Carter said state officials did not explain to her that the money could have been spent instead on staff positions such as parent liaisons. But while she would love to have some liaisons or art instructors, she says, she would have gone with Lightspan anyway.
Carter said she was well aware of the dubiousness of many software company claims. In Lightspan's case, she believed the promises.
"That's why you have to be careful. Everyone has something, and some of that research is not worth a hill of beans," she said. "You have to recognize the fluff."
Coming tomorrow: The new digital divide