cientists are forcefully challenging a recent study that claimed to identify a rare genetic mutation that sharply boosted the risk of multiple sclerosis. The critics cite calculation errors and say they have been unable to replicate the findings — and question why the original paper was ever published in a top journal.

The withering assault has dashed hopes that the study might quickly lead to new drugs for MS and also raised questions about how such critiques are handled: like more and more journals, the one that published the paper does not run letters to the editor, making it harder for scientists to see that the claim has been hotly disputed.

As soon as the MS claim was published in June, outside scientists expressed doubts to STAT that there was really a link between the newly discovered mutation and the disease. Since then, other experts have dug into the study in greater detail, tried to replicate its findings using a database of thousands of MS patients, and checked the original paper’s statistics. In each case they concluded there is no link between the rare mutation and MS.

advertisement

The study — which reported that the mutation raised the risk of MS by a whopping 70 percent — was led by Carles Vilariño-Güell of the University of British Columbia and published in the respected journal Neuron. It fingered a gene called NR1H3, finding that a rare mutation silences other genes involved in regulating inflammation or other aspects of immunity. As a result, inflammation runs wild, and the immune system destroys the protective myelin sheath on neurons, crippling their ability to carry signals needed for movement and other functions.

The purported mutation-MS link led to hopes that correcting the mutation or its downstream consequences might treat or even cure MS, which affects some 2.3 million people worldwide.

Daniel Weeks, a professor of human genetics and biostatistics at the University of Pittsburgh, said an “extremely straightforward” analysis showed that some crucial statistical calculations in the paper were wrong. Doing them correctly “dramatically changes the strength of their claim,” he told STAT. “It’s no longer statistically significant.” That is, the association between the mutation and MS had a good likelihood of being due to chance.

Chris Cotsapas of Yale University and the Broad Institute, who studies the genetics of immune-related diseases such as MS, went further, looked for the mutation reported in the Neuron study but in different MS patients and healthy controls. Using a database maintained by the International MS Genetics Consortium, which holds information on 13 times more people than Vilariño-Güell studied, Cotsapas identified 31 people who have the mutation and also MS (out of a total sample of 32,852 MS patients in the database) and 31 who have the mutation but not the disease (out of 36,538). That is, the mutation was hardly more common in MS patients than healthy people.

“That tells us there is no association” between the rare NR1H3 mutation “and MS risk,” Cotsapas said, echoing the conclusion of a paper he posted to the online preprint site BioRxiv. “We found no evidence to support their claim.”

Sign up to our Weekend Reads newsletter

The third strike against the paper’s conclusions came from DNA sequence data gathered by the Exome Aggregation Consortium (ExAC). The NR1H3 mutation is seen in 21 of the 60,706 genomes in this database, Eric Minikel and Daniel MacArthur of the Broad found. That is about as frequent as the mutation appears in the people analyzed for the Neuron paper. Only if a mutation is more common in people with a disease than in those without it does it make sense to conclude that the mutation causes the disease, the Broad scientists noted. The mutation “contributes extremely weakly, if at all, to MS risk,” they wrote in an online comment on the biomedical journal portal PubMed.

If the NR1H3 mutation turns out not to cause MS, it wouldn’t be the first time a claimed link between a gene and a disease crashed and burned. Last week, another analysis of ExAC data concluded that 43 genes once linked to heart disease were likely false positives, too. But the flaws in the Neuron paper were so apparent, said Cotsapas, that the decision to publish it represents “a failure of peer review.”

Vilariño-Güel stands by his conclusion, he told STAT. He doesn’t dispute that a mutation that raises the risk of some disease should be more prevalent in patients than in healthy people, the opposite of what the ExAC and MS consortium analyses found. But he argued that while such DNA data, from the general population, “can identify common genetic variants which increase the risk of disease,” they can miss links to very rare mutations like NR1H3. Studying only DNA from families of MS patients, as he did, can reveal such links, he said. He also argued that some apparently healthy people in the MS consortium’s database might eventually develop the disease.

“I do believe [the association between the mutation and MS] is correct, although it would be nice to have an independent replication,” Vilariño-Güell said. “In addition to the data in the Neuron article, we have unpublished results that provide additional support” for the original claim.

One way to resolve the disagreement is to breed lab mice carrying the mutation and see whether they develop MS. Vilariño-Güell expects to have a colony of such mice by early next year, letting his team run experiments to determine whether the mutation really does cause MS.

In the meantime, journals’ policies make it hard for interested scientists to know what’s going on. Fewer and fewer journals run letters to the editor, said Pittsburgh’s Weeks. As a result, when readers access the disputed paper through the Science Direct portal or read it on Neuron’s website, there is no indication of the criticism. “The concern is, how can you correct the scientific record in a way everyone will see,” Weeks said.

Neuron declined to comment on whether there was a failure of peer review in its decision to publish the original paper or on plans to run criticisms of it.

Scientists involved in the dispute told STAT that they expect their critiques to be published soon as “Matters Arising,” Neuron’s section for such responses. In a statement, the journal said that it is “open to submissions that challenge previously published papers” and that those submissions undergo peer review, which letters to the editor traditionally didn’t. “This process can take some time,” the statement said.

I have had ms since 1982, at 30 yrs old. I am now 65. Started with relaxing remitting. Now about 8 to 10 yrs ago went to secondary progressive.
Use walker. On now cops on and myfortic which seems to be doing ok.
I did speak to my dr. Ms specialist. About this drug.
Will discuss next visit.

We’ve been discussing the issues of reproducibility, statistical significance, and publishing bias towards significance in my quantitative research methods course. Stories like this are a great motivation for me to slow down and really understand the small details of what I’m learning. This research has given hope to so many families and the fact that other scientists are questioning it is a sign that the community is aware of their impact on the general public.

We’d like to clarify that readers can raise concerns and leave feedback on Cell.com for any Neuron article. For the paper in question, a comment was posted on Cell.com August 5 and is linked to the paper on the Neuron website: http://www.cell.com/neuron/comments/S0896-6273(16)30126-X. In addition, our “Matters Arising” format provides a peer-reviewed forum for readers to challenge a Neuron paper. We also want to reinforce that we value expert peer review as critically important for evaluating scientific content, including data that contests previously published work.

The world has unrealistic expectations of the peer review process. Mistakes in this process are inevitable and in my opinion it is counterproductive to focus so much energy on trying to improve an inherently flawed process. A much better approach is to focus on augmenting and simplifying the post publication, reader-based commentary process. There is a “wisdom-in-the-crowd” generally speaking. It is this concept that is embodied in a new generation journal, Cureus.com via its SIQ (scholarly imapact quotient) and commenting process.

This sounds like the self-correcting process of science working as expected, particularly if Neuron is willing to publish a Comment piece discussing the strengths and weaknesses of the original paper. Maybe the faults in the original paper should have been caught in peer review, but maybe it’s better that these results are in the public sphere and available for scrutiny by the broader MS community.

Experts. Their only answer is it takes time. They are as worthless as wea thermal. If any of them were actually experts there would be a cure by now. Can’t tell me they don’t get paid enough to do nothing.

The article contains multiple references to “the Journal” that published the disputed article — tho did not mention it by name. The Journal is Neuron and even though the reader could find that info by clicking one of the many links contained in the article, it’s also likely a reader would have missed it.

Seems to me that the Doctor’s and MS Sites that post this stuff are totally out of control .. If they can Isolate a Gene that “Causes MS” then why haven’t these same Doctor’s located a Gene to cure Cancer or Lupus or Diabetes ??

Obviously another Bogus Claim but wow the Research Money that they make is phenomenal .. Instead of wasting all those Dollars to create BS the MS Society should put that money to Good Use and help those who already have been long suffering some more Assistance ??

Seems to me that they (NMSS) is making money even as a supposed Non-Profit and pays way to much on Administration and gets Paid by the Pharmaceutical Companies to Falsely Advertise all this nonsense .. There are many who could use help with Doctors , Bills at home and more Medical Equipment since THAT was the Original Purpose of this “Society” ..