Study shows one way that scientific progress is broken

The classic model of scientific progress is that the field advances when new findings contradict or supersede old ones. But a new study reveals that this process isn't working today—at least, not in scientific journals, where most data is shared with colleagues. Indeed, the researchers found that "rebuttals scarcely alter scientific perceptions about the original papers."

For the study, a group of researchers looked at the citation rates on seven marine biology papers about fisheries. Citation rates are often used as a proxy for the "importance" of a scientific paper, with the notion that the more a paper is cited, the more influential it is. Each paper had been the subject of a rebuttal, also published in a scientific journal. The researchers wanted to know whether these rebuttals affected citation levels on the original papers—and, perhaps more importantly, whether they convinced people to question the interpretation of data in the original papers.

It turns out that rebuttals don't seem to affect the scientific community's understanding of the original papers in any way. "The original articles were cited 17 times more frequently than the rebuttals, an order of magnitude difference that overwhelms other factors," write the study authors in Ecosphere. "Our test score results emphasize that rebuttals have little influence: even the rare few authors who happened upon the rebuttals were influenced only enough to move from whole-hearted support of the original article (a score of five) to neutrality (a score of three), despite the fact that all of the rebuttals argue that the interpretations of data in the originals were incorrect. Astonishingly, 8 percent of the papers that cited a rebuttal actually suggested that the rebuttal supported the claims of the original article, an observation which may give pause to those contemplating writing a rebuttal in the future."

Obviously, the sample size here is modest. The authors of the study only looked at the effects of rebuttals on seven papers in the field of marine conservation biology. However, their findings are still fairly damning. They suggest that once a paper has been published and widely cited, it's almost impossible to contradict it with new evidence or data analysis.

The authors believe that one way to address this problem would be to link to rebuttals in the original papers so that readers understand that the paper is contested. This is especially important in fields like conservation biology, where highly cited papers are often used to make public policy. Scientific journals should make it easy for readers to find rebuttals in the interests of transparency and the advancement of scientific knowledge.

Annalee Newitz
Annalee Newitz is the Tech Culture Editor at Ars Technica. She is also the author of Scatter, Adapt, and Remember: How Humans Will Survive a Mass Extinction, and her first novel will be published in 2017. Emailannalee.newitz@arstechnica.com//Twitter@annaleen