Medical research we never hear about: the problem of unpublished studies

Every day, there is another medical study in the news. There’s another newspaper or TV story telling us that X can cure depression or make you thinner or cause autism or whatever. And since it’s a medical study, we usually think that it’s true. Why wouldn’t it be?

But what most people don’t realize, let alone really think about, is that there might be other studies that show that X does none of those things—and that some of those studies might never have been published.

Just this week, the journal Pediatrics released an article that perfectly demonstrates this problem. There have been a number of studies that have shown that a certain type of medication, serotonin receptor inhibitors (SRI’s), can help stop the repetitive behaviors of autism, like hand-flapping or head-banging. If you were to do a search of the medical literature, as doctors and parents and patients often do, you’d think that using SRI’s is a good idea. But when researchers dug deeper, they found that there were just as many unpublished studies that showed that SRI’s didn’t help. If they had all been published (they were all good enough to be published), that same search of the medical literature would have shown that using SRI’s isn’t a good idea.

This is bad. We rely on studies to guide our decisions. What is going on?

The journals that publish articles certainly play a role. After all, it’s cooler to publish a study that has a grabby headline and promises an answer or a cure. That’s much more likely to get readers than a study that says that something doesn’t do anything at all. But it turns out that the researchers themselves play a bigger role.

Some researchers don’t even write up their studies or try to publish them. You can’t blame them, to some extent. If you set out to show that Wonderdrug cures Bad Disease, and it turns out that Wonderdrug does squat, that hardly seems interesting or important. Or if you start studying Wonderdrug and then have to stop the study because of side effects or other problems with giving it, you might not write that up or try to publish it either—after all, it wasn’t finished. So the study goes into the trash bin, and you move on to the next idea. No big deal, right?

Well, if other studies are being published that say that Wonderdrug does cure Bad Disease, and don’t mention any side effects or problems, it is a big deal—because a whole other side of the story would be missing.

A companion study in Pediatrics looked at clinical research involving children and found that results weren’t available for more than half the studies involving children—because they weren’t completed or weren’t published.

As a pediatrician, this worries me. I’m making medical decisions for my patients based on less than half of the information out there?

There is a database, ClinicalTrials.gov, where researchers are supposed to “register” their studies before they even start. That way, there is a record of it—and even if they stop it, or if it never gets published, there is a way to at least know it existed and find out more about it. Unfortunately, not all studies get registered. Many journals, including Pediatrics, won’t publish a study unless it was registered—if all journals would do that, maybe all researchers would register their studies.

This doesn’t mean that you can’t trust medical research. Medical research has brought us antibiotics and heart surgery and drugs that cure many cancers; medical research saves and improves countless lives every single day. But it does mean you have to be a savvy and skeptical consumer of health information. It means that you need to ask questions, learn about the sources of the information, read widely and—always—talk to your doctor before you make a health decision based on a study.

It means that you always have to remember that there just might be another side to the story.