Potential flaws in genomics paper scrutinized on Twitter

Article tools

A recent Twitter conversation that cast doubt on the conclusions of a genomics study has revived a debate about how best to publicly discuss possible errors in research. Yoav Gilad, a geneticist at the University of Chicago in Illinois, last month wrote on Twitter that fundamental errors in the design and data analysis of a December 2014 study2 led to an unfounded conclusion about the genetic similarities between mice and humans. Gilad and his co-author Orna Mizrahi-Man, a bioinformatics researcher at the University of Chicago, have since detailed their data reanalysis1 in the open-access journal F1000Research (in which articles are openly peer-reviewed after publication). Michael Snyder, a geneticist at Stanford University in California and co-author of the original paper, stands by his team’s study and its conclusions and says that Gilad broke the “social norms” of science by initially posting the critique on Twitter. Gilad says that he took to social media to highlight his work, which might otherwise have been overlooked.

The original paper, published in the Proceedings of the National Academy of Sciences (PNAS) by members of the Mouse ENCODE (Encyclopedia of DNA Elements) consortium, compared gene-expression profiles from selected tissues in mice and humans. The authors concluded that specific tissues in mice had more in common with other mouse tissues than with the same tissue in humans — a finding that suggested the genetic gulf between mice and humans was wider than many had previously thought. “That raised a lot of fur,” Snyder says. “A lot of people are invested in mice as biomedical models.”

Gilad was intrigued by the paper, because he says that its conclusions run counter to those of many other studies that have compared gene expression in human and mouse tissue. So he asked for the raw data from the Mouse ENCODE authors and took a closer look. In an April Twitter post that included graphs and a table, Gilad made the case that the team had not accounted for ‘batch effects’, a potential source of bias.

These effects can arise from variations in the way different batches of samples are handled, which can affect the data and be mistaken for biological effects. Gilad concluded that the analysis involved five different sequencing runs to analyse RNA. One of those runs included both mouse and human tissues, but the other four contained samples from only mouse or only human tissues. He argues that any differences in the runs themselves could have exaggerated the differences between the two species.

According to Gilad’s reanalysis, which has not yet been peer reviewed, removing possible batch effects from the data reverses the conclusion of the PNAS paper — such that the gene-expression data cluster by tissue and not by species. He concluded that mouse tissue is more similar to the same tissue in humans than to mouse tissue from other parts of the body.

Gilad’s post quickly drew a string of comments from researchers who were surprised that the PNAS paper had not explicitly addressed batch effects, which have gained increasing attention in many labs. Ran Blekhman, a geneticist at the University of Minnesota in Minneapolis, responded to Gilad’s report on Twitter:

Some called for the PNAS paper to be retracted, while others thought that the paper should be maintained but that researchers’ comments about it should be made visible.

In a response posted to Gilad’s paper on the F1000Research website, Snyder and his co-authors write that they spent two years assessing the data. One of their main priorities, they say, was to “address concerns pertaining to potential laboratory and batch effects”. Snyder said in an interview that the team found no signs that batch effects had altered the findings. All of the sequencing runs were conducted by the same person using the same reagents, lowering the risk of unintentional bias. Furthermore, he says, they used control samples to confirm that results stayed consistent even when samples were processed on different lanes of the sequencing machine. Snyder says that his team plans to publish a rebuttal in the near future, but he decided against defending the paper on Twitter. “I don’t think engaging in that medium would have helped us,” he says. He adds that he would have been happy to discuss the issue of batch effects with Gilad. “If someone has concerns, the normal route is to contact the journal or the authors,” says Snyder. “There was an avalanche of comments before we even knew about it.”

Gilad says that he posted the original critique on Twitter because it would have been difficult to draw much attention to his findings through the traditional channels. “Papers that challenge results from ENCODE or similar large consortium projects are never published in the same glamour journals as the original reports, and these papers are usually largely ignored,” he says.

Lior Pachter, a computational biologist at the University of California at Berkeley, who was not involved in either study, wrote on Twitter last month that he thought the PNAS article should be retracted. After subsequently reading Gilad’s full report, Pachter says that he still has the same response to the PNAS paper. “It was obvious that batch effects hadn’t been addressed in this paper,” he says.

In response to Gilad’s initial tweet in April, geneticist Ewan Birney at the European Bioinformatics Institute in Hinxton, UK, who led the human ENCODE analysis, also said on Twitter that the paper should be retracted if the bulk of the results end up being invalidated. However, Birney offered some context in his tweets: “Remember, we all have batch/unknown confounders, and it’s not uncommon that one’s experimental design is not optimal due to all sorts of things (logistics etc).”

Snyder and his team say in their comment posted to Gilad’s article that they are doing more sequencing experiments to address some of the concerns and will soon post their findings.

Error or no error, the online discussion shows how social media is changing the way research results are being discussed. Pachter defends Gilad’s decision to share his findings on social media. “The post drew attention to something that could have been overlooked,” he says. Thanks to Twitter and blogs, he says, “you can crowdsource discussion and analysis. I think that’s very healthy for science.”