Ben Goldacre

An award-winning writer and broadcaster, Ben Goldacre was trained in medicine at Oxford and London and now works full time as a physician. Since 2003, he has written the weekly Bad Science column in The Guardian, where he debunks any and all unfounded scientific claims made by parties from government to journalists and pharmaceutical companies. His book Bad Science (4th Estate), published in 25 countries, has sold over 500,000 copies worldwide, topping the paperback non-fiction charts. Among Goldacre’s many awards are the Royal Statistical Society’s first Award For Statistical Excellence in Journalism, the Faculty of Public Health DARE Prize Lecture, an honorary doctorate from Herriott-Watt University, “Best Freelancer” at the Medical Journalists Awards 2006, the Healthwatch Award in 2006, and “Best Feature” at the British Science Writers Awards (twice). He was also shortlisted in the Samuel Johnson and Royal Society literary prizes in 2009. Goldacre is a frequent guest on numerous British television and radio programs and speaks regularly at conferences and events. He made The Rise of the Lifestyle Nutritionists, as well as the two-part documentary The Placebo Effect, for BBC Radio 4. He is currently producing a three-part documentary series for the BBC World Service.

2012

Where’s the rest of the data iceberg? (Ben Goldacre)

Science Kit for Ben Goldacre

Missing data.

Trials are the best tool we have in medicine to find out if a treatment works, or doesn’t. They are simple, fair tests: take a group of people with a condition; divide them into two groups; give one treatment to one group, and another to the other; then measure how many people get better in each group. Medicine relies on this evidence.

Unfortunately, when people conduct trials, they often leave the results unpublished, if they don’t like them. This means that we only see half of the evidence, which can be completely misleading. If I tossed a coin 100 times, but only told you the outcome from half the tosses, I could make it seem as if I had a two-headed coin: you would be a fool to let me get away with that. And yet this is exactly what we tolerate in medicine.

Fluke positive finding gets published. Refutations don’t.

A psychologist published research showing evidence of precognition in the Journal of Personality and Social Psychology. It’s possible that this was a fluke finding. Others tried to replicate the same study, and found no evidence of precognition. The journal declined to publish their study, saying they don’t publish replications as a matter of principle. This is a serious problem for research dissemination, as it means freak positive findings are more widely disseminated.

This is not limited to precognition research! A group of researchers report in the March issue of Nature how they tried to replicate 53 early, laboratory, preclinical studies of potential targets for cancer treatments. 47 out of 53 could not be replicated. They recommend that there should be more opportunity for researchers to publish negative data, and shifts in the incentives for researchers to do so.

In the 1980s over 100,000 people died through the practice of prescribing anti-arrhytmic drugs to patients who’d had a heart attack. We could have had earlier warning of the dangers of this, if we had seen the results of an earlier trial on a similar anti-arrhythmic drug that killed patients. Because the drug was abandoned, and it never came to open market, the findings of this trial were never published. The researchers published their results a decade later, explaining that they were now concerned their findings could have helped to avert disaster. This story is well told by the researchers themselves.

“When we carried out our study in 1980, we thought that the increased death rate that occurred in the lorcainide group was an effect of chance.”

“The development of lorcainide was abandoned for commercial reasons, and this study was therefore never published; it is now a good example of ‘publication bias’. The results described here

might have provided an early warning of trouble ahead.”

Cowley AJ, Skene A, Stainer K, Hampton JR. The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction: an example of publication bias. International journal of cardiology. 1993;40(2):161–6.

This story is also well covered in the excellent book “Testing Treatments”, available free for download online (declaration: I wrote the foreword!).

And it’s not the distant past….

Reboxetine7 trials were conducted comparing reboxetine against placebo. Six trials showed the drug was no better than a dummy sugar pill. One trial showed it was better than a dummy pill. Only that positive trial was published. Similarly, three trials showed reboxetine was as good as any other antidepressant, but data from three times as many patients showed it was worse. The unflattering data wasn’t published.

That’s an extreme example. How commonly does this happen? Very. If you want to document how often trials go missing, one good option is to find a list of trials you know have been conducted and completed, and then go to see if those have been published. For example, you can often get information on trials conducted before a drug was licensed, from medicines regulators. This isn’t all the trial data on a drug, but it’s more than enough to explore the phenomenon of unflattering data that goes missing in action before being published in the academic journals that doctors read. These researchers went and found all the trials on all antidepressants.

37 of the positive trials - all but one - were published in full. The trials with negative results had a different fate. Only three were published. 22 were never published. The remaining eleven trials - which had negative results, in the FDA summaries - did appear in the academic literature, but when they did, they were written up as if the drug was a success. In reality, we have 38 positive trials, and 37 negative ones; in the academic literature, we have 48 positive trials, and 3 negative ones.

A systematic review published in 2010 summarises all the studies ever conducted looking at the problem of trial results that go missing in action. They found that overall, positive results are about twice as likely to be published as negative results. This is an excellent and thorough overview of the literature on how often trials go missing, and why. It’s not a pop science read but it’s thorough!

The company refused to hand over the results in a form that the Cochrane reviewers were able to use. They have since got access to some of the full Clinical Study Reports, which has exposed possible discrepancies. This is a story to watch closely.

The most current discussion on this story appears in the April 2012 PloS, in which they discuss the company’s explanation of why they won’t share the data, and whether they find these reasons credible:

People have made various grand claims to have fixed this problem: this has been harmful, since it has left the wider academic and medical community with the false impression that there is no longer a problem.

This paper is a good account of the history of how people came to realise that there was a problem to fix, and how the fixes failed to materialise.

Chalmers, I., 2006. From optimism to disillusion about commitment to transparency in the medico-industrial complex. JRSM 99, 337 –341.

The ICMJE, the international medical journal editors association, said that they would force people to register all trials, by saying they would only publish pre-registered trials. Sadly, they didn’t hold the line, and continued to publish unregistered trials.

Even if this fix had been implemented, it would not be enough, because it only requires publication of trials starting from now. We don’t practice medicine using only the results of trials starting from now! We need the results of all trials, for all drugs in current use, without exception, to reduce the risk of us inflicting avoidable harm on patients.

This isn’t about stifling innovation, or “attacking industry”: it’s about ensuring that we innovate, with our eyes wide open. Doctors and patients should be allowed to see all the evidence on what works.

Further reading:

This is an excellent academic overview of the need for all trials to be published, from an open access journal.

This is an excellent book on evidence based medicine, and the problems in it. It is written by eminent scientists and patients working together, and can be downloaded free online (declaration: I wrote the foreword!).

Tags

Keywords

Newsletter Sign-up

Thanks! Your email has been added you to our newsletter mailing list.

TED trademark and logo used under license from TED Conferences LLC. TEDMED, LLC is a nonprofit that focuses on the intersection of health and medicine, while the TED Conference (which is also a non-profit and operates the website www.TED.com) has a broader focus on Ideas Worth Spreading. TEDMED, LLC is wholly owned by The TEDMED Foundation, a 501(c)(3) Public Charity. | 2 High Ridge Park | Stamford, CT | 06905