Art Diamond's web log

Many Important Medical Articles Cannot Be Replicated

The standard scientific method is more fallible, and less logically rigorous, than is generally admitted. One implication is to strengthen the case for allowing patients considerable freedom in choosing their own treatments.

(p. D1) It has been jarring to learn in recent years that a reproducible result may actually be the rarest of birds. Replication, the ability of another lab to reproduce a finding, is the gold standard of science, reassurance that you have discovered something true. But that is getting harder all the time. With the most accessible truths already discovered, what remains are often subtle effects, some so delicate that they can be conjured up only under ideal circumstances, using highly specialized techniques.

Fears that this is resulting in some questionable findings began to emerge in 2005, when Dr. John P. A. Ioannidis, a kind of meta-scientist who researches research, wrote a paper pointedly titled "Why Most Published Research Findings Are False."

. . .

. . . he published another blockbuster, examining more than a decade's worth of highly regarded papers -- the effect of a daily aspirin on cardiac disease, for example, or the risks of hormone replacement therapy for older women. He found that a large proportion of the conclusions were undermined or contradicted by later studies.

His work was just the beginning. Concern about the problem has reached the point that the journal Nature has assembled an archive, filled with reports and analyses, called Challenges in Irreproducible Research.

Among them is a paper in which C. Glenn Begley, who is chief scientific officer at TetraLogic Pharmaceuticals, described an experience he had while at Amgen, another drug company. He and his colleagues could not replicate 47 of 53 landmark papers about cancer. Some of the results could not be reproduced even with the help of the original scientists working in their own labs.