NEW YORK (Reuters) - After years in which a handful of
scientists raised alarms about biomedical studies that cannot be
independently confirmed, raising the possibility that much of
the research literature has been compromised by widespread
incompetence or even deception, top U.S. science officials are
acknowledging the crisis of "irreproducibility."

"We have to take this seriously," Dr. Francis Collins,
director of the National Institutes of Health (NIH), the
country's largest funder of basic biomedical research, told
Reuters in an interview, citing the waste of time and money
spent trying to build on studies whose results are a mirage or
whose methods are too opaque for others to follow.

In an essay in the journal Nature published on Monday,
Collins warned that "the checks and balances that once ensured
scientific fidelity have been hobbled," and outlined steps NIH
will take to combat the "non-replication" problem.

On Friday, President Barack Obama's Council of Advisors on
Science and Technology (PCAST) will hold a meeting on
reproducibility. The council is scheduled to hear from
researchers who have investigated the crisis and from journal
editors. Collins said high-impact journals are "are looking for
papers that are groundbreaking and surprising" - the kind most
likely to report results that turn out to be phantoms.

"We wanted to learn more about the issues," said geneticist
Eric Lander of the Broad Institute and PCAST co-chairman. As to
whether the advisers might make any recommendations, "it's too
early to know whether it would make sense for PCAST to weigh
in."

Also scheduled to address PCAST is Secretary of Commerce
Penny Pritzker. Pharmaceutical, biotech and medical diagnostic
companies have become increasingly concerned about scientific
papers whose claims hold the promise of leading to new
medications or diagnostic tests - but that they can't reproduce.

Scientists at biotech pioneer Amgen, for instance, reported
in 2011 that they could confirm only six of 53 landmark studies
in cancer biology. Researchers at pharma giant Bayer announced
in 2012 that only 14 of 67 attempts to confirm claims in
oncology, women's health and cardiovascular disease succeeded.
Officials at Novartis and AstraZeneca told a recent cancer
meeting that they encountered the same problem.

"Whether in diagnostics or pharma, you hear the same story,"
said Mary Lou Gantzer, chief executive officer of closely held
startup BioCore Diagnostics: "'We saw something interesting in a
paper, and we couldn't reproduce it.'"

SECRET SAUCE

Replication attempts fail for one of two reasons: The
original claim is wrong, or it is correct but the scientists
trying to confirm it goofed.

To address the second category, NIH is encouraging journals
to devote more space for scientists to detail the methodology of
their experiment. That way, there is no excuse for omitting
mention of a crucial step or "secret sauce."

"I wish more replication failures could be blamed on this"
habit of keeping methods partly hidden, Collins said. "But we
have to recognize that many results are not replicating because
they're wrong," usually because of flawed methodology.

In an attempt to remedy that, NIH is developing mandatory
training in experimental design for in-house scientists this
year. It is also testing whether its funding decisions should be
based in part on a detailed analysis of the proposed
experimental design, such as whether it includes statistical and
other practices that reduce the chance of false results.

Including those best practices would make research using
animal models or lab-grown cells - where most replication
failures occur - more like the gold standard of clinical trials
with humans.

Scientists who have led the charge against irreproducibility
varied in their reactions to the NIH proposals.

"It's great that they're acknowledging this is a problem,"
said geneticist Elizabeth Iorns, founder and CEO of Science
Exchange, an online portal that matches scientists with
providers of technical services for their experiments. "But
their solutions are likely to have minimal impact," since there
is no rigorous research showing, for instance, that the kind of
training in experimental design that NIH will require reduces
poor science.

NIH could have a greater impact if it funded studies of how
widespread irreproducibility is in various fields to "get a grip
on which have the worst problem," Iorns said, and if it funded
attempts to replicate important results. "You wouldn't need a
lot of money to do this."

Last year, Iorns' "Reproducibility Initiative" began to
check 50 recent high-impact papers in cancer cell biology. She
expects that the verdicts on all 50 will be in this year, at a
cost of $1.3 million (underwritten by the Laura and John Arnold
Foundation). NIH's 2014 budget is $30 billion.

C. Glenn Begley, who led the work at Amgen attempting to
confirm the cancer studies, welcomed NIH's recognition of the
replication crisis and the idea of making animal research adhere
to the high standards of human clinical trials. "If we could do
only one thing, it would be insist that all of our studies were
blinded," said Begley, chief scientific officer of closely held
TetraLogic Pharmaceuticals.

He also thought training young scientists in proper
methodology would help. "Most investigators aren't out to dupe
anyone, but they don't know what's acceptable and what's not,"
he said. Some scientists have told him it's fine to report the
one experiment that finds a drug candidate has an effect, for
instance, and not the dozens that fail to find an effect. "They
get an outlier result and present it as the result," he said.

Reforms at the institutional level can go only so far,
however.

"The real problem is that scientists are reluctant to speak
up about studies that won't replicate because there is so much
to lose," Begley said. "If I criticize you, and you review my
next grant application, you might (take revenge). That's why
people are afraid to say the reason they couldn't replicate a
study is that it was just plain wrong."