Medical research exaggerates success of studies

Published 4:18 pm, Tuesday, July 23, 2013

Researchers seeking new medical treatments need to be much stricter in their animal studies and make their results - and especially their failures - more accessible if they are to improve the chances of finding drugs for Alzheimer's disease and other conditions that have been notoriously stubborn to treat, according to a new paper from Stanford.

An analysis of nearly 4,500 animal studies found almost twice as many reports of positive results - meaning, the treatment being studied showed statistical significance of being effective - than one would expect given the parameters of the study designs, said Dr. John Ioannidis, a professor of medicine at Stanford who was lead author of the paper.

In other words, based on statistics alone, it's clear that many animal studies are biased to produce positive results, which then are used to push treatments into human clinical trials, and in some cases put drugs on the market.

That bias - which is usually subtle and probably unintentional, Ioannidis said - helps explain why drugs to treat stroke and Alzheimer's and a host of other challenging conditions often seem so promising in mice and other animal studies, only to fail in human experiments.

LATEST NEWS VIDEOS

Newly Found, Smiley-Faced Spider Species Named After Bernie SandersGeoBeats

Barack Obama Likens Dropping Daughter Off At College To ‘Open Heart Surgery’GeoBeats

Fire retardant is dropped on Oakland hills grass firesfgate

President Trump: 'I Grew Up in New York, I know Many Puerto Ricans'Buzz60

Firefighters battle an Oakland Hills fire from the airsfgate

Fire trucks arrive on scene of Oakland hills firesfgate

Fire burns in Oakland hillssfgate

"For so many diseases we have dozens of interventions that seem to work in animals, or at least this is what reading the published literature would tell you," said Ioannidis, who is an expert in clinical trial design and statistical analysis.

But that supposed success in animal studies hasn't translated into drugs to treat Alzheimer's, for example. "For stroke, we have maybe one intervention that works. For Parkinson's disease we have a few," Ioannidis said. "There's a lot of discrepancy."

A critical function

One basic flaw of animal studies is that on a purely biological basis, what works in a rat doesn't always work in a human being. Still, animal studies are critical to medical research, said Ioannidis. Studying rodents and other mammals is a key step in the process of moving a potential drug treatment from the laboratory into human subjects, to make sure therapies are safe and have at least some chance of working.

But professional pressure to produce positive results means that many scientists, perhaps without even being aware of it, will only report data that support their work. The problem is that once scientists then try to move their research into human experiments, the results don't hold up.

Human clinical trials operate under much higher standards and many more guidelines than animal studies, mostly to promote the safety of patients, but also to prevent accidental or intentional bias.

The "gold standard" for human research is the randomized, double-blinded, placebo-controlled trial. That means research subjects are randomly divided into a group that gets the treatment and a control group that gets a placebo. Both the subjects and the scientists are blinded, so that no one knows who's getting the treatment and it's nearly impossible to let bias enter into the results.

Also, human clinical trials must be registered with the U.S. National Institutes of Health, which means that the goals and the parameters of the study are made public before the trial even begins.

Opening for bias

But no such universal registration exists for animal studies.

Bias can enter into the research in several ways, Ioannidis pointed out in his paper. A scientist might run dozens of statistical analyses on his data, but only publish the results that are positive and ignore the results that are neutral. Or he might change the parameters of a study after the research has been completed to put a more positive spin on the results.

Scientists are not necessarily trying to quash negative results, and they may not even be aware of their subtle bias, Ioannidis said. For example, if a drug therapy seems to be effective in a mouse after 12 hours, but not after 16 hours, it's understandable the scientist would be more interested in reporting - and further studying - the 12-hour effect.

"There is flexibility in the way data can be analyzed," Ioannidis said. "There may be 50 different outcomes being studied and five different statistical models that could be applied. One of those outcomes may be statistically significant, but if that's all that gets reported, and everything else is silenced, the message may be very misleading.

"In scientific studies there should be a clear understanding up front in what's going to happen with the data," he said.

Not helping matters is that medical journals often are reluctant to publish negative study results. So what ends up published is almost never an accurate representation of the actual research being done, said John Huguenard, a neuroscientist who was not part of Ioannidis' group.

"There's a bias in the publication industry against negative results. It's not exciting to publish them," Huguenard said.

Reviewing results

The new paper, which was published last week in the journal PLOS Biology, analyzed 160 previously published meta-analyses of potential treatments for six neurological disorders that were tested in animals. Meta-analysis is a type of research that pools together the results of multiple studies to look for trends or patterns across a wide field.

Only eight of those 160 treatments showed promise of being effective in the animal studies, and with no hint of bias under his group's scrutiny. In contrast, 108 treatments had been published and reported as having positive results.

In all, Ioannidis looked at 4,445 animal studies that were part of the meta-analyses. Based on statistical modeling, he would expect that 919 of those studies would produce positive results, but in fact, 1,719 studies were published with a positive spin.

"Across a very wide spectrum of diseases where animal studies have been performed, there's substantial evidence of bias," Ioannidis said.

Ioannidis is not the only one to criticize the over-representation of positive animal studies in medical research. Last year, a national group of neuroscientists called for researchers to be more open both in reporting negative results and in releasing details about the structure of their animal studies.

Looking for change

Huguenard, who was part of that team, said he's optimistic that the research community is open to changing the way animal studies are conducted and making results more accessible.

"As scientists we think we're above reproach in terms of our integrity," Huguenard said, but he added that it's human nature to lean toward the positive rather than the negative - especially in a competitive field, under intense career pressure.

"We have to admit that there are unconscious biases that can influence how we see the data," he said. "The idea is to minimize bias, knowing that it's going to exist."

One idea is to create a registry of animal studies, which would serve several purposes. It would encourage scientists to share certain details of their work - their unfiltered results, for example, or important descriptions of the animals in their study. At the same time, it would force scientists to commit to study designs and endpoints, giving them less opportunity to spin the results to suit their needs.

Results database

Perhaps most important, a registry could serve as a de facto database of both the positive and the negative results. If a study produced negative results and never was published, at least the scientific community would be aware that it exists.

Negative results may not be sexy or particularly helpful for career advancement, but they are a necessary part of scientific discovery. The fact that they're being largely ignored now isn't helping anyone, said Dr. Arnold Kriegstein, director of the Broad Center of Regeneration Medicine and Stem Cell Research at UCSF.

He said stem cell scientists are especially vulnerable to bias in animal studies, in part because there is so much competition.

"There's a big clamor in the stem cell field to push treatments into clinic very quickly," Kriegstein said. "My concern is many of these expensive human trials are being organized based on very sketchy or inadequate animal information.

"In the end, they turn out to be really disappointing, and the hopes of patients are dashed, and a lot of resources are gone," he said. "These are becoming really big issues."

In his paper, Ioannidis notes that even the most scientifically rigid animal studies that suggest a treatment may work in people are more likely than not to fail in human experiments.

Of the eight treatments that met his group's high standards for showing significant promise without any sign of bias, only two later showed some success in human clinical trials. But that's far better than the field of animal research as a whole - only about 1 percent of animal successes translate into treatments that work in humans, he said.

"It's never going to be 100 percent," Ioannidis said. "But if we can manage to get animal studies to a 25-percent success rate, that's pretty decent."

Latest from the SFGATE homepage:

Click below for the top news from around the Bay Area and beyond. Sign up for our newsletters to be the first to learn about breaking news and more. Go to 'Sign In' and 'Manage Profile' at the top of the page.