Deconstructing “What Works” in Education Technology

Over the weekend, The New York Times published the second story in its series on “Grading the Digital School.” The first story in the series questioned the massive expenditures schools make on education technology, pointing to stagnant test scores as an indication that these investments might not be worth it. Last weekend’s story extends that skeptical eye to the outcomes from educational software, again questioning whether using these tools in the classroom makes much of a difference in terms of student achievement.

The story focuses on the claims made by Carnegie Learning that its math software provides “revolutionary results.” But Carnegie Learning is hardly alone in the claims it makes about the effectiveness of its products. Adjectives like “revolutionary,” “innovative,” “feature-rich,” “content-rich,” “engaging,” “standards-based” pepper the landscape. And much like the well-known advertising claim that “4 out of 5 dentists surveyed recommend Trident gum to their patients who chew sugarless gum,” educational software frequently touts claims about the satisfaction and success of the teachers, students or administrators that use the products.

The New York Times article interrogates these claims, noting that “amid a classroom-based software boom estimated at $2.2 billion a year, debate continues to rage over the effectiveness of technology on learning and how best to measure it.”

The article points to the Department of Education’s What Works Clearinghouse (WWC) as a resource for school districts to use to help make better procurement decisions. As the name suggests, the What Works Clearinghouse is meant to be a “central and trusted source of scientific evidence about ‘what works’ in education.”

The WWC covers a range of educational topics — math, literacy, science, special education, drop-out prevention, for example — as well as for multiple age groups and different populations, such as English-language learners. The Web site contains an interface that lets schools search for the “interventions” it wants (that is, for the educational programs, practices, or policies) — with options to search for specific topic areas, age groups, populations, delivery method and program types. It also lets users return search results by the “effectiveness” of the programs — in other words, positive effects, potentially positive effects, mixed results, negative effects or potentially negative effects.

The WWC does not conduct research itself but rather reviews the research of others. That’s a key piece here, as the WWC wants to ensure that what “counts” as effective educational software is really supported by rigorous research. According to its procedures and standards (PDF), “the WWC only reports impacts for studies for which it has high or moderate confidence that the effect can be attributed solely to the intervention rather than to the many other factors that are at play in schools and in the lives of students.” In other words, the WWC is supposed to offer a “Good Housekeeping Seal of Approval,” of sorts, for educational research as well as for the educational software and practices it examines.

But there are problems with this approach.

Even the Times piece points out that many schools do not know about and thus do not use the research there. According to a study last year by the General Accounting Office study, just 42% of school districts even know about the Web site; and of those that do, they say they only use it for a “small or moderate extent” in decision-making. That may be in part because the Web site doesn’t say anything about the costs of these programs, so schools might not feel they have all the information they need in order to make a purchasing decision (the software with “the best results” might not be within everyone’s budgets). And the research on the site often isn’t quite up-to-date (the “relevant time frame” for research covered by the WWC includes anything from the past 20 years).

Furthermore, the research actually only covers a limited number of software products. In part, of course, that’s because of the site’s stringent policies on “what counts” as research. But it’s also a reflection of the types of products that get scientific research done about them — that’s often the software created by the major companies. With that limited scope, the site doesn’t really help schools that want to use products from smaller companies or from app developers.

An article by University of California, Berkeley professor Alan Schoenfeld, a former “senior content adviser” for the WWC points out some of the other problems with the organization (PDF). Schoenfeld notes some of the very real complexities and challenges of conducting educational research and evaluating curricula. He also suggests that, despite its claims of scientific rigor, the WWC is often mired in the politics of education policy and ideology.

That brings us full circle, in some ways, to the Times series on “Grading the Digital School.” In both stories, it’s standardized test scores that are used as the measurement for what’s working and what’s not working in education technology. That is, after all, what the Department of Education counts as well.

But is that really the only or the best measure for the success of any educational effort, whether it’s digital or not? And if it’s not, how then can educators and parents identify educational software that’s worth using? How will we measure “what works” in ed-tech?

Audrey – thanks so much for this important post. Questioning and investigating how exactly the What Works Clearhouse decides what counts as good educational research is quite similar in many ways to questioning how the current mainstream educational policy sphere decides what counts as educational “success”.

In both cases, we have quantification fetishes abound. In WWC, it means only even considering research that uses certain highly quantitative methods (eg, randomized experiments that have control and experimental groups) as opposed to the breadth of educational and learning sciences research that utilizes case studies, iterative design based research and other qualitative or mixed methods. In deciding whether students are successful, it’s about measuring achievement on single-point in time multiple choice tests as opposed to performance-based assessments, portfolios, or even more radical, having learning environments that are themselves giving realtime assessment to learners as they engage in the learning process.

The reporting in the Times in this series has fallen into the same quantification fetish, and everyone I know who works in educational research and learning sciences has had plenty to critique about the two articles published so far. Hopefully they’ll improve the series after getting some constructive feedback.

Summer break presents the perfect opportunity for students to dig into games and build skills that’ll reap huge rewards when they return in the fall. Game making can be one of the best ways to get students thinking creatively while cultivating useful technical literacies, and there’s a ton of absorbing tools that students won’t tire of over the long break. Here are three options to choose from depending on the type of technology students have at home.

For educators who are interested in using games for learning — specifically towards developing skills as they relate to the Common Core State Standards — here are five games students can enjoy and that we’ve found sync with standards.

The success and popularity of Minecraft in and out of classrooms is no surprise. It’s one of the best examples of the potential of learning with games because it embraces exploration, discovery, creation, collaboration, and problem-solving while allowing teachers to shepherd play toward any subject area. But Minecraft is not the only game of this kind. Take a look at some of these.