Healthy Skepticism

Could a third of medical research be wrong?

The latest medical research is wrong about one-third of the time, that is… according to the latest medical research. A survey of 49 highly cited medical studies by epidemiologist John Ioannidis found the results of 14 studies were contradicted or downplayed by later research. Ioannidis' survey raises some hard questions. Is there a fundamental flaw in medical research, or is this just part of scientific progress?

Problems occurred most often in studies that did not use randomized samples – five out of six were contradicted. A group that includes two high-profile cases of preventive therapies for coronary-artery disease. One study recommended hormone replacement therapy for post-menopausal women -- a treatment that some doctors now believe may increase chances of developing the disease. The other therapy used high doses of vitamin E to keep the coronary arteries healthy, a treatment that was later shown to be ineffective in randomized trials.

In spite of the conflicting research, Nutritional Epidemiologist Eric Rimm is standing by his work. Rimm's study showed vitamin E reduced the risk of developing coronary-artery disease in healthy men ages 40 to 75. "I think what we originally reported hasn't really been re-tested," he said. The follow-up study cited by Ioannidis tested whether vitamin E prevented heart attacks and strokes in men and women over the age of 55 who already had cardiovascular disease or diabetes. According to Rimm, the health benefits of antioxidants like vitamin E provide is still a lively debate. "I thought our findings would be more generalizable," Rimm said, "But I think our results stand up, it just doesn't protect people with existing heart disease."

Over generalizing research results is one way that Ioannidis sees medical studies being misused by doctors. "There are many issues that are not finalized with a single study," he said, "issues like trade-offs between benefits and harms, side-effects, and generalizability." If Ioannidis' work can be said to have a moral it is – don't put too much faith in one study.

Solving the problem is not as simple as sticking to randomized experiments or requiring results to be duplicated. Observational studies, like Rimm's vitamin E research, are not randomized, but they can provide a foundation for future research. Likewise, duplicating research results can be unethical, and that may be the case for the 11 studies in his survey that have not been followed-up. One case is a clinical trial of the drug Zidovudine, a medication that was 75 percent effective in preventing HIV positive mothers from transmitting the disease to their unborn children. Re-testing Zidovudine would require exposing some unborn children to an increased risk of HIV infection. So, how should patients deal with the confusion?

"We should switch our mode of thinking about a statistically significant result to what I would call a credible result," said Ioannidis. He proposes a system of rating published research based on the rigor of its experimental design, sample size and amount of supporting research. "There is nothing wrong about acknowledging that all of the research published in medical journals is not one-hundred percent credible," he said. "There is no perfect research." Ioannidis advises patients to protect themselves by taking a more critical approach to their doctor's advice. "Ask not just 'is it good for me?' but 'what is the uncertainty?"