Wednesday, 11 January 2012

The Journal of Irreproducible
Results, a science-humor magazine, is, sadly, no longer the only
publication that can lay claim to its title. More and more
published scientific studies are difficult or impossible to
repeat.
It’s not that the experiments themselves are so flawed they
can’t be redone to the same effect -- though this happens more
than scientists would like. It’s that the data upon which the
work is based, as well as the methods employed, are too often
not published, leaving the science hidden.

Too Little Transparency

Consider, for example, a recent notorious incident in
biomedical science. In 2006, researchers at Duke University
seemed to have discovered relationships between lung cancer
patients’ personal genetic signatures and their responsiveness
to certain drugs. The scientists published their results in
respected journals (the New England Journal of Medicine and
Nature Medicine), but only part of the genetic signature data
used in the studies was publicly available, and the computer
codes used to generate the findings were never revealed. This is
unfortunately typical for scientific publications.
The Duke research was considered such a breakthrough that
other scientists quickly became interested in replicating it,
but because so much information was unavailable, it took three
years for them to uncover and publicize a number of very serious
errors in the published reports. Eventually, those reports were
retracted, and clinical trials based on the flawed results were
canceled.
In response to this incident, the Institute of Medicine
convened a committee to review what data should appropriately be
revealed from genomics research that leads to clinical trials.
This committee is due to release its report early this year.
Unfortunately, the research community rarely addresses the
problem of reproducibility so directly. Inadequate sharing is
common to all scientific domains that use computers in their
research today (most of science), and it hampers transparency.
By making the underlying data and computer code
conveniently available, scientists could open a new era of
innovation and growth. In October, the White House released a
memorandum titled “Accelerating Technology Transfer and
Commercialization of Federal Research in Support of High-Growth
Businesses,” which outlines ways for federal funding agencies to
improve the rate of technology transfer from government-financed
laboratories to the private business sector.

As Jon Claerbout, a professor emeritus of geophysics at
Stanford University, has noted, scientific publication isn’t
scholarship itself, but only the advertising of scholarship. The
actual work -- the steps needed to reproduce the scientific
finding -- must be shared.