The Reproducibility Project not only brought into question a lot of what we think we know, it also brought ethical concerns to light. Reproducibility is an important part of research. Results are only good if they can be consistently repeated–otherwise, we may be looking at errors in methodology and science that doesn’t do anyone much good. If researchers were regularly overstating their results, that would be a problem. If studies weren’t reproducible and no one had any idea, clearly we were far too quick to adopt new science.

All fair points to make, but it turns out that the Reproducibility Project has problems of its own.

Slate has a great article breaking down not only the criticisms of the project but the counterpoints to those criticisms. This isn’t a callout, after all–it’s the scientific method at work.

Upon investigating the study further, the researchers identified a second and more crucial problem. Basically, the OSC researchers did a terrible job replicating those 100 studies in the first place. As King put it: “You’d think that a test about replications would actually reproduce the original studies.” But no! Some of the methods used for the reproduced studies were utterly confounding—for instance, OSC researchers tried to reproduce an American study that dealt with Stanford University students’ attitudes toward affirmative action policies by using Dutch students at the University of Amsterdam. Others simply didn’t use enough subjects to be reliable.

The new analysis “completely repudiates” the idea that the OSC study provides evidence for a crisis in psychology, says King. Of course, that doesn’t mean we shouldn’t be concerned with reproducibility in science. “We should be obsessed with these questions,” says King. “They are incredibly important. But it isn’t true that all social psychologists are making stuff up.”

So the good news is that the underpinnings of modern psychology haven’t been dashed to pieces. The bad news is that we still don’t really know how reproducible some of these results are. But we might yet–if the Reproducibility Project does nothing else, it gives us good reason to look twice at the science we take for granted.

Help us give hope at events around the world. Support Take This on Patreon!

9805 NE 116th St
Suite 7411
Kirkland, WA 98034

Take This, Inc. 2019
EIN: 46-3882735

Take This is a mental health nonprofit decreasing stigma and increasing support for mental health in games.

We are a mental health organization providing comprehensive resources and support that is tailored for the unique needs of the game development community and embraces the diverse cultures and issues of the game community. We combine clinical best practice with a love of all things gamer, providing a safe, accepting space for gamers and developers. We work in a spirit of partnership with other organizations addressing these issues among game enthusiasts, streamers, and creators.