WATCH: What Doctors Don't Know About the Drugs They Prescribe

TEDandThe Huffington Postare excited to bring you TEDWeekends, a curated weekend program that introduces a powerful "idea worth spreading" every Friday, anchored in an exceptional TEDTalk. This week's TEDTalk is accompanied by an original blog post from the featured speaker, along with new op-eds, thoughts and responses from the HuffPost community. Watch the talk above, read the blog post and tell us your thoughts below. Become part of the conversation!

__________________________________________

TEDTalks can sometimes portray science in triumphalist tones, with fabulous innovations that are changing the world forever. But the real action in science is often around dirty, messy, angry problems, and my TEDTalk is about the dirtiest I've seen yet.

Doctors need the results of clinical trials to make informed choices, with their patients, about which treatment to use. But the best currently available evidence estimates that half of all clinical trials, for the treatments we use today, have never been published. This problem is the same for industry-sponsored trials and independent academic studies, across all fields of medicine from surgery to oncology, and it represents an enormous hidden hole for everything we do. Doctors can't make informed decisions, when half the evidence is missing.

Most people react to this situation with incredulity, because it's so obviously absurd. How can medics, academics, and legislators have permitted such a huge problem to persist? The answer is simple. This territory has been policed -- and aggressively -- by the pharmaceutical industry. They have worked hard to shut down public discussion on the topic, for several decades, with great success.

They say, for example, that the problem is modest, and that critics have cherrypicked the evidence: but this is a lie. The best evidence comes from the most current review of all the literature, published in 2010. It estimates that half of all completed trials are left unpublished, and that trials with negative results are about twice as likely to be buried.

Then they pretend that the problem is in the past, and that everything has been fixed. But in reality, none of these supposed fixes were subjected to any kind of routine public audit, and all have now been well-documented as failures. What's more, they all shared one simple loophole: they only demanded information about new trials, and this is hopeless. Anything that only gets us the results of studies completing after 2008 does nothing to fix medicine today, because more than 80% of all treatments prescribed this year came to the market more than ten years ago. We need the results of clinical trials from 2007, 2003, 1999, and 1993, to make informed decisions about the medicines we use today. This isn't about catching companies out for past misdemeanors, it's a simple practical matter of making medicine optimally safe and effective.

The arguments go on, with ever more red herrings: industry spokespeople pretend that information about trials -- such as Clinical Study Reports -- can't be released without breaching the confidentiality of individual patient participants. But in reality, the EU Ombudsman has already forced the European Medicines Agency (EMA) to release hundreds of these exact same documents. He stated clearly that the administrative burden of removing any individual patient information is minimal. Next, they claim the cost of sharing trial information is prohibitive: but both the EMA and GSK have committed to releasing all the Clinical Study Reports that they have, and the EMA has already shared millions of pages of documents, quite happily. Sometimes industry people even claim -- in hugely patronizing tones -- that it's better for only regulators to see trial results, behind closed doors, because the public would panic if exposed to dissenting views.

There's more. Sometimes they pretend that the academic journals are the bad guys, for rejecting papers with negative results, when the evidence shows this was barely ever an issue, and, in any case, there are now endless open access journals, specifically designed to accept negative results. Then there are the hole-pickers: people who pay lip service to the problem, with a brief claim that they are "on your side," then expend all their worldly effort trying to pour cold water on the problem, pretending that things aren't so bad after all.

Nassim Taleb, author of The Black Swan, once laughingly told me that every page of my book Bad Pharma was infused with paranoia: that he could tell I'd been made furious by having the same argument about missing data, over and again, with chancers; that I had resorted to obsessively covering every exit in the argument with snipers, to hem the enemy in, across hundreds of pages. I admit I am obsessed, and I hope you'll share those details: if not by reading the book, then at least by inhaling the eight page briefing we have produced for our campaign, at alltrials.net.

Because this is where I have to confront my inner grouch, and cave in to the optimism of TED. I have to tell you about the successful campaign for trials transparency that has grown since this talk first went up.

In January, the science writer Simon Singh grew frustrated at my whining that things weren't moving fast enough, and gave a generous donation to the campaigning charity Sense About Science. This kickstarted some more traditional lobbying. We teamed up with the great and good -- journal editors, academics, and a knight -- to form the alltrials.net campaign.

AllTrials asks people to sign up to two simple things: firstly, recognize that this is an ongoing problem; and secondly, call for the release of basic information (including brief summary results, and the long "Clinical Study Report" where available) for all trials, on all treatments currently in use, whether past or future.

This has grown almost faster than we can. Over 40,000 individuals have signed up, and over 200 organizations. More than a hundred patient groups have committed their support, recognizing that it's their members who participate in trials, and take medicines based on their results. We are now supported by almost all medical and academic professional bodies in the UK -- the Royal Colleges, the Societies and Faculties -- not to mention the National Physicians Alliance from the U.S., and the European Public Health Association. The outsider position in the UK is now not supporting the AllTrials campaign.

In February, GlaxoSmithKline, one of the biggest drug companies in the world, signed up, and committed to publicly release all their Clinical Study Reports, going back to the beginning of the company. More companies are discussing signing, and if these promises are delivered, they could finally add an interesting new element to the game: for the first time ever, I believe there will be commercial incentives to transparency. Imagine that there are two treatments, both with apparently equal benefits. One is made by a company that shares all its results, but the other is made by a company that publicly mocks and derides the very notion that doctors and patients should dare to ask for such a thing. Which treatment do you want?

I think the answer is clear, but we need more than this. We need public engagement on the issue, and wider recognition among policy makers, to get on with concrete solutions. That's why I hope you'll sign up to alltrials.net, and if you're a member of a professional body anywhere in the world, then please get them to sign too. If you're from a company, then we are very happy to talk informally -- and confidentially -- about any anxieties you may have, but please, save your reputations, and sign.

The current state of affairs is absurd. We can spend tens of millions of dollars on just one trial, hoping to ensure it is free from bias, and trying to accurately detect tiny differences between one treatment and another. And yet we let all those biases right back in, by allowing people to bury half the data. Future generations will look back at our tolerating this bizarre situation in astonishment, the same way that we look back on medieval bloodletting. The time is now, to fix it forever, easily.

Ideas are not set in stone. When exposed to thoughtful people, they morph and adapt into their most potent form. TEDWeekends will highlight some of today's most intriguing ideas and allow them to develop in real time through your voice! Tweet #TEDWeekends to share your perspective or email tedweekends@hufﬁngtonpost.com to learn about future weekend's ideas to contribute as a writer.