I am a science writer and a former Registered Massage Therapist with a decade of experience treating tough pain cases. I was the Assistant Editor of ScienceBasedMedicine.org for several years. I’ve written hundreds of articles and several books, and I’m known for readable but heavily referenced analysis, with a touch of sass. I am a runner and ultimate player. • more about me • more about PainScience.com

In 2005, PLoS Medicine published a now-famous paper with the attention-grabbing but exasperating title, “Why Most Published Research Findings Are False”.1 It was written by John Ioannidis, MD, PhD, an American doctor from Greece, a distinguished author of about 400 peer-reviewed papers, 40 books and book chapters or so, and much more. Ioannidis is a giant, and I am really nobody to criticize.

One of the best examples of why a lot of research findings turn out to be false is that a lot of research makes evidence seem more “significant” than it really is. There are two common and serious problems with statistical significance, and they both tend to plague research about manual therapies and pain problems — which is why I actually wrote an article about statistics (eeek) for this website. See Statistical Significance Abuse. Comic strips and funny videos included.

I like the content of the paper very much, but the title is a misleading generalization that, unfortunately, plays right into the hands of quacks and charlatans, who like nothing better than to cite any expert who seems to be saying that science is so flawed — that “science is wrong” — that it can’t be used to debunk their nonsense. This is their sole interest in the paper, and indeed it has been extensively cited co-opted for this very purpose. A substantial portion of its popularity is probably due to how effectively its title can be used to undermine the credibility of science.

Ironically, while this work should serve to improve the quality of scientific medicine, it is being used by some cranks to attack the scientific basis of medicine.

I don’t believe that it’s an option for scientists to be oblivious to these issues — it’s naïve to think that a title like that won’t be abused, and it’s naïve to think it doesn’t matter.

It will be abused, and it does matter.

Science is not wrong

Attention: the paper does not say that “science is wrong”!

What Ioannidis really says is much less ominous: he argues that it should take rather a lot of good quality and convergent scientific evidence before we can be reasonably sure of a “scientific fact,” and he presents good (scientific!) evidence that a lot of so-called conclusions are premature, not as ready for prime time as we would hope.2 Any supposedly large treatment effect in medical research is probably exaggerated, and “when additional trials are performed, the effect sizes become typically much smaller,” so “well-validated large effects are uncommon.”3 Extremely uncommon, in fact.

It’s okay for science to work like this. Ioannidis did not mean that science is broken or deeply flawed.

In fact, this is actually how science should work, and it’s not the least bit surprising to good scientists, who never claimed in the first place that their results are infallible or that their conclusions are “true.” When most scientists publish, they’re just saying, “Hey, here’s what we did, here’s what we saw — what about you guys?” And then the research community responds, hopefully, with thoughtful criticism, and more research.

“Scientific investigation is the noblest pursuit,” Ioannidis writes.4 “I think we can improve the respect of the public for researchers by showing how difficult success is.”

Sadly, not all scientists are good at their jobs.

A medical Madoff

Credit for that excellent heading goes to Scientific American, which reported on “a medical Madoff,” a doctor who published several influential but entirely fraudulent scientific papers:

Over the past 12 years, anesthesiologist Scott Reuben revolutionized the way physicians provide pain relief to patients undergoing orthopedic surgery for everything from torn ligaments to worn-out hips. Now, the profession is in shambles after an investigation revealed that at least 21 of Reuben's papers were pure fiction, and that the pain drugs he touted in them may have slowed postoperative healing.

The story of Scott Reuben’s fraud will also be co-opted to make the point that “mainstream” medical science is rotten. But it wasn’t “science” that was bad, it was Scott Reuben. The “science” wasn’t misleading, Scott Reuben was misleading. Scott Reuben was a liar-liar-pants-on-fire — and thank goodness science is self-correcting by nature, or who knows how much longer he would have gotten away with it.

Kurt Donsbach is going down … finally. It was extraordinary that Scott Reuben managed to get away with publishing fabricated science for a decade — scientific fraud is usually shut down more quickly. On the other hand, alternative medicine doesn’t condemn its own, and con-artists can thrive for decades as long as they distance themselves from real medicine and claim that science does not apply to them.

Kurt Donsbach, a naturopath and a chiropractor, practiced with impunity for half a century before he finally faced eleven felony counts, including “unlawfully dispensing drugs as a cure for cancer” and “falsely representing a cure for cancer” — but he’s one of the few of his kind who even get charged. Most get away with it, largely because they claim to be above scientific criticism.

Reuben’s bogus results were more or less doomed to be debunked. Science usually exposes scientific frauds quite quickly, and this is a much more important characteristic of science than the fact that frauds occur in the first place. Like a bad organ transplant, the scientific community will reject frauds the moment they are detected. This is strikingly different than the world of alternative medicine, where quacks can get away with selling non-medicine for decades, because you cannot discredit someone who had no credit to begin with — someone who never belonged to any respectable community or adhered to any standards in the first place, someone who may indeed define themselves in opposition to professional communities and standards. You can’t boot someone out of a club they never joined, and so they are “regulated” only by popular opinion and market forces — and our culture basically gives them a pass on proving anything, and if doctors and scientists criticize them it’s just a “conspiracy.”

This is why prosecutions of con-artists in health care are remarkably rare. And it’s why anything that undermines the credibility of science in public opinion is an aid and comfort to quacks.

Freedman’s Atlantic article about Ioannidis

In 2010, David Freedman authored an article about Ioannidis for The Atlantic, Lies, Damned Lies, and Medical Science, which got a great deal of attention at the time and continues to be cited widely. It is basically a translation of Ioannidis’ key concepts for a general readership. Unfortunately, it sensationalizes and emphasizes the “science is wrong” message … but on a much larger scale than ever before.

Freedman’s slant is subtle. He is not wearing an anti-scientific agenda on his sleeve here. (Update: he is now. In the July/August 2011 Freedman published in The Atlantic again, this time as a much more blatant apologist for alternative medicine.) He gets a lot of things exactly right, and concludes with lovely quotes from Ioannidis that go a long way to balancing the article. I don't doubt that many readers actually got the right idea, and that is great.

But don’t underestimate the intensity or importance of the “war” between mainstream and alternative medicine — a much hotter war than most people are aware of — and the degree to which irrational, unethical quacks will make hay from any shred of an argument against science. Unfortunately, Freedman's treatment of Ioannidis gives great ammunition to anyone who has a chip on their shoulder about mainstream medicine. Citing Freedman gives them a new way to miss Ioanndis’ real point with near impunity, and sound good doing it. I've already seen it cited extensively in that way.

Unfortunately, we also know from some of Freedman's other publications that he has does indeed have a clear anti-scientific bias. For example, he tries to make the case that Wakefield's dangerous vaccine-autism link fraud is a meaningfully representative example of a general problem with medical science. You really have to be banging an anti-science drum hard to get that out the extreme Wakefield example! Every statement Freedman makes in that post has truth in it, but he’s bending over backwards to make science look as untrustworthy and corrupt as he possibly can — as if two out of every three scientists is a Wakefield!

Once you know this about Freedman, it's hard not to see how he used his Atlantic article as an opportunity to sensationalize the wrongness of science. And the proof is in the way the quacks have been gleefully citing it every chance they get. One quack will post it on Facebook, say, and get a long string of "amens" about the evils of science and "mainstream" medicine.

Human overconfidence and fallibility

Fortunately, outright scientific fraud is actually quite rare — and it was not the subject or point of Ioannidis’ paper. His point was that a lot of science is done amateurishly to “prove” pet theories, and this is tends to produce incorrect results, surprise surprise. Dr. Gorski phrased it a more precisely:

What Ioannidis’ research shows is that clinical trials examining highly improbable hypotheses are far more likely to produce false positive results than clinical trials examining hypotheses with a stronger basis in science.

Science is a human enterprise, and people are imperfect (you may have noticed). Honest, well-intentioned, competent scientists can have strong egos and biases, and even the best can make logical mistakes and become locked into their prejudices.5 Many good scientists play into the hands of critics with overconfident “conclusions” that are all-too easy to point to as examples of scientific inconsistency and imperfection.

And that’s the good scientists.

Overconfidence and premature conclusions are what Ioannidis is correctly warning against. This is why I have criticized so many scientific papers. Here’s a good example, with the cockiness built right into the title: “Radial extracorporeal shock wave therapy (ESWT) is safe and effective in the treatment of chronic recalcitrant plantar fasciitis.”6 The evidence in that paper was nowhere near enough to justify that title. It wasn’t fraudulent, just overconfident.

So it’s ironic that Ioannidis’ title was also brash and irresponsible.

Piling irony on irony, it’s possible Ioannidis may not have even written his own title — an PLoS Medicine editor may have done it. Science writer and broadcaster Flora Graham points out she doesn’t write her own headlines:

The subeditor does that! Writing a good headline is much harder than it looks. I have felt that my carefully measured story has been given an inflammatory or inaccurate headline — and given readers the wrong idea. What if Ioannidis’ editor wrote his headline? His paper cautioning against over-interpretation of evidence ends up with a title that over-interprets his evidence!

Ooooh, that rotten “media”!

Usually it is everybody’s favourite scapegoat, the media — not scientists themselves — who unwisely portray research findings as “conclusions.”

Little joke:If you lined up all the economists in the world from end to end, they would still never reach a conclusion.

Unfortunately, thanks to “the media,” the public has the same perception of scientists.

Sure, sure, almost every summary (abstract) of a scientific paper has a “conclusion.” But good scientists know full well that all knowledge is tentative, provisional. They know that even the best evidence is just a piece of an evolving (and fascinating) puzzle. But reporters want to make it look like it’s all a done deal! This exacerbates the public perception that science is unreliable and inconsistent and that scientists, like economists, can never agree on anything, the never-ending battle of the experts. And this is good, rich soil for evangelicals and quacks to sow doubt and dissent about evolution and scientific medicine. The average person is much more prone to getting suckered into buying quackery when scientific medicine is constantly portrayed as chasing its own tail.

So it really troubles me when scientists themselves, even incredibly smart guys like Ioannidis — or his editor? — hand the enemy an intellectual bazooka on a silver platter.

Notes

But even Ioannidis’ research findings about the wrongness of research findings may be wrong! And precisely this point has been raised by some critics. In 2007, Goodman and Greenlander wrote: “we agree with the paper’s conclusions and recommendations … but calculating the unreliability of the medical research literature, in whole or in part, requires more empirical evidence and different inferential models than were used. The claim that ‘most research findings are false for most research designs and for most fields’ must be considered as yet unproven.” Ioannidis replied substantively, and the debate goes on … as it should. BACK TO TEXT

None more famously than Albert Einstein, who produced more good ideas in his 20s than most universities can produce in a decade, but still couldn’t accept the implication of quantum mechanics that “God” does indeed “play dice”. BACK TO TEXT

This overconfidently titled paper essentially declares that there is no longer any controversy about ESWT for plantar fasciitis. However, my confidence in their conclusions is suppressed by the fact that the researchers are on the payroll of a company that makes ESWT devices, and the entire study was funded by that company. As always, conflicts of interest are not necessarily a deal-breaker, but they can be, and this one seems particularly strong.