MCDONALD: Negative data is still data

I am exceedingly good at failing. Even though I’m getting my Ph.D. in one of the best immunology departments in the world, the common thread running through my graduate school career has not been success, but failure. I’m not an anomaly, however. I am simply a scientist.

For those of you who haven’t tried research, let me explain. Most experiments don’t work. There are the experiments where you’ve been up for 16 hours straight, and you accidentally forget to add a critical enzyme. And there are those where you spend six months developing a new assay, only to discover that the background is too high, and it’s completely worthless. Then there are experiments that test a perfectly reasonable hypothesis in a perfectly effective way. It’s just that it turns out your hypothesis is wrong. The latter situation is the most frustrating, because negative data like this is often still ‘good science’ — meaning it’s carefully executed, and well controlled, so you can draw solid conclusions from it. But for the most part, it’s not going to make the cut for publication, especially in high-profile journals like Science or Nature — the ones every biologist strives for.

As a result, negative data generally doesn’t reach other scientists who could use it. Another research group might think of the same hypothesis a year after you’ve already tested it, and they decide to pursue it, wasting time and money in uninformed redundancy. (In clinical trials of drugs, negative data is an even thornier problem involving questions of fraud, which I will not go into here.) Another consequence of publication bias is an overinflated perception of science as successful, both by the public, and often by the struggling, increasingly stressed graduate students doing the work.

Indeed, take a look at the research featured on the covers of the first 2012 issues in these journals. Science describes work that may allow the next touchscreens to be free of pesky fingerprints; in Nature astronomers report that a gas cloud is on its way to being consumed by a black hole an amazing “four million times the mass of the Sun.” These are all valuable discoveries and deserve to be shared with the scientific community, but they don’t provide an accurate picture of most of the science that’s happening.

To fill some of the void, several small, often online-only journals have sprouted up, including the Journal of Negative Results in Biomedicine, The All Results Journals and the most recent arrival to the hidden part of science, The Journal of Errology (JoE). Not yet publishing, JoE is a self-described “experimental online research repository that enables sharing and discussions on those unpublished futile hypothesis [sic], errors, iterations, negative results, false starts and other original stumbles that are part of a larger successful research in biological sciences.” (Clearly, The Journal of Errology perhaps takes the idea of error as acceptable too far. In addition to that mistake, on their own website they also refer to themselves as The Journal of Errorology.) The emphasis here is not so much on the paper in the ‘journal,’ but on providing details you might only get from close scientific peers.

While I appreciate the effort of these journals, I don’t think they will solve the problem of under-reported negative data until scientists themselves value all data more highly, and institutional changes alleviate the fear of being scooped with a refreshing bolus of cooperation. But they’re a step in the right direction.

Until then, don’t be fooled by the flagship journals that will only rarely publish negative findings. To be sure, whichever way it flops, the Large Hadron Collidor’s search for the Higgs boson (and other particles) will make Science. But this is an exception to the way we’ve structured the establishment of science to operate. Everyone interested in scientific progress must keep pushing to find a way for science to be as visible as possible — and include our successes as well as our failures.

Jessica McDonald is a sixth-year graduate student in the Department of Immunobiology. Contact her at jessica.mcdonald@yale.edu.

Comments

Lamplighter

Hello, headline writers! The word “data” is a plural noun and the correct verb form is “are,” not “is.”

Catherine08

Thank you, Jessica. As the story goes, it took Edison thousands of failures before he discovered a working design for the light bulb.

yalengineer

F1000 recently announced a preprint database. That along with Nature Precedings does open up the door to publishing negative data.

It sounds that the bigger issue is how does negative data tie into building one’s academic portfolio. At the moment, it’s not much and heavily relies on the number of Nature/Science/Cell papers and H-Index. But it is possible.