Last month, Ivan met David Vaux at the 3rd World Conference on Research Integrity in Montreal. David mentioned a retraction he published in Nature, and we thought it would be a great guest post on what it’s like to retract one of your own papers in an attempt to clean up the literature.

David Vaux

In September 1995 Nature asked me to review a manuscript by Bellgrau and co-workers, which subsequently appeared. I was very excited by this paper, as it showed that expression of CD95L on Sertoli cells in allogeneic mismatched testes tissue transplanted under the kidney capsule was able to induce apoptosis of invading cytotoxic T cells, thereby preventing rejection. As I wrote in a News and Views piece, the implications of these findings were enormous – grafts engineered to express CD95L would be able to prevent rejection without generalized immunosuppression.

In fact, I was so taken by these findings that we started generation of transgenic mice that expressed CD95L on their islet beta cells to see if it would allow islet cell grafts to avoid rejection and provide a cure for diabetes in mismatched recipients.

Little did we know that instead of providing an answer to transplant rejection, these experiments would teach us a great deal about editorial practices and the difficulty of correcting errors once they appear in the literature.

What we found was that unfortunately, these grafts were not protected, and indeed CD95L-expressing grafts seemed to provoke more, not less, of an inflammatory response. Puzzled by this, we decided to repeat the experiments by Bellgrau et al., but unlike them, we found that allogeneic mismatched testes grafts were rejected. A subsequent more thorough reading of the literature revealed that similar mismatched testes tissue grafts had been performed previously, both in the mouse and therat, and their results were the same as ours, and opposite those of Bellgrau et al., i.e. the unmatched testes tissue was rejected.

Knowing that Nature had an explicit editorial policy to publish, in some form, work which refutes an important conclusion of any paper which appears in its pages, we submitted our findings describing the transgenic mice and our failure to replicate the work from Bellgrau et al. to Nature. We received two very positive reviews, but based on a third, very negative one, from Bellgrau et al., the editors decided not to publish our findings as a letter or as correspondence.

In 1996, we submitted our manuscript to Nature Medicine, but it was rejected without review, with the comment from the editor in chief, Adrian Ivinson, that he did “not think formal submission to Nature Medicine would be appropriate”. We then sent the manuscript to PNAS, where it has attracted 305 citations. Subsequently, another paper appeared describing transplants of beta cells from CD95L transgenic mice, and their findings were the same as ours, i.e. graft CD95L did not confer protection, but if anything, provoked inflammation. To our surprise, this paper appeared in Nature Medicine, accompanied by a News and Views by Lau and Stoeckert emphasizing the importance of the findings.

I was becoming increasingly frustrated by Nature’s refusal to abide by its own ethical policies to publish rebuttals, and Nature Medicine’s decisions apparently based on papers’ sources rather than their contents, when I had a flash of inspiration – I had published a News and Views extolling the virtues of Bellgrau et al.’s paper – now I could retract it!

I wrote to Phillip Campbell at Nature saying that I wished to retract my News and Views piece because I no longer had confidence in the findings on which it was based. My reasons for doubt were:

We were unable to reproduce Bellgrau et al.’s findings;

Three earlier groups who had published similar experiments had also come to the opposite conclusion;

The failure of transgenic CD95L to protect allogeneic islet cells was contrary to the model they proposed.

I added “I regret having to take this course, but as Nature refuses to abide by its own ethical policy, namely to “publish refutations of any important conclusion that appears in its pages,” I am left with no other option.

Thankfully, Nature did agree to publish the retraction, but, perhaps unsurprisingly, they were unhappy with the wording. The retraction included just two sentences:

I wish to point out that I no longer stand by the views reported in my News and Views article “Immunology: Ways around rejection” (Nature 377, 576–577; 1995), which dealt with a paper in the same issue (“A role for CD95 ligand in preventing graft rejection” by D. Bellgrau et al. — Nature 377, 630–635; 1995). My colleagues and I have been unable to reproduce some of the results of Bellgrau et al., as reported by J. Allison et al. (Proc. Natl Acad. Sci. USA94, 3943-3947; 1997).

This was accompanied by:

D. Bellgrau et al. consider that their results are reproducible and stand by them. They note, however, that the magnification in Figure1g of their paper should be 113 times, not 45 times as printed. Both groups believe that other published data support their views, and interested readers can contact them directly for further details. — Editor, Nature.

Note that they did not say that the results were reproducible, or that they had actually reproduced them, they just considered them to be reproducible. Indeed, no one, including Bellgrau et al., have subsequently reported reproducing these results. Furthermore, it turned out that Sertoli cells do not even express CD95L.

The retraction was published in 1998, and has attracted 16 citations of its own. However, of the 976 citations of the Bellgrau et al. paper, about 700 were subsequent to publication of the retraction, so it’s clear many remain unaware that its findings are questionable. Clearly, the processes that allow the scientific record to self-correct can be improved, not least by Nature.

Want to keep up with all things Retraction Watch? You can follow us on Twitter, like us on Facebook, add us to your RSS reader, and sign up on our homepage for an email every time there’s a new post.

Thank you so much. Dr. Vaux, for telling your story. I am going to share it with the young scientists in my lab. Perhaps they will find themselves in a similar situation as yours someday, and will know what to do.

Boutique journals attract flawed papers, purportedly of great importance. If a significant fraction of papers published in boutique journals were to be contested openly in those journals, then those journals would quickly climb down the ladder of “prestige”. Clearly then, the editors of boutique journals have to be highly resistant to contested findings. [Amazingly, the editors of many boutique journals have not even been prominent researchers.] Thank you, David, for providing an account of your close encounter with Nature and Nature Medicine.

Thanks for the story.
There is a detail I don’t quite get, it may be my bad – pardon my French. Do I understand correctly that the authors or the original Nature paper were reviewer on the rebuttal proposed to the same journal (“…but based on a third, very negative one, from Bellgrau et al.,…”)? Shouldn’t they have been excluded from this process for (huge) conflict of interest? And how can they be identified as reviewer should remain anonymous?

In my opinion it is actually good that the original group is included as reviewer of a comment. Who knows, there may be a discrepancy in testing methods that may explain the difference, or even just plain incompetence. It prevents some major embarrassment for both the submitter of the critical comment and the journal.

However, to use it as the sole basis of rejecting the comment, against the two likely much-less-biased reviewers, is just plain wrong. I would, if I were editor, have asked the other two reviewers to also assess the reviewer comments of the original group. If they disagree and can explain why, I’d say: publish!

Regarding being identified: some reviewers expose themselves for ethical reasons, sometimes by accident (“we did not do this wrong!” Oh, oops…!), and sometimes it just is immediately obvious. I myself once found that the “anonymous” reviewer had copy-pasted the exact same comments I had received more than a year earlier from the corresponding author. He also commented on something that was present in my initial comment I had sent to the corresponding author, but was not present in the comment I submitted to the journal. Now, I wonder how that anonymous reviewer could have known that…………….

I think Nature and similarly known journals just need to go for the big money – they sometimes can not publish really good science as long as there is no “main stream” for it. And much of the perhaps 5% accepted submissions have to deal with highly exposed, already reputated scientists – usually also having major competitors… I think there are many examples that such journals failed totally.

I noticed Dr Voux retraction in 1998 when I was doing research on co-transplantation of islets with testicluar Sertoli cells. I spent over 3 years to reproduce the previous studies showing immunoprotective effects of Sertoli cells but finally I failed to do so.

Such stories underline the importance of aggregating community-provided post-publication peer reviews and ratings. The target paper may be reviewed and rated on Epistemio at http://www.epistemio.com/p/Jr3gS9VW .

Citing of incorrect research findings is a fact of human scientific endeavour – people even cite retracted papers – and 900 citations is particularly depressing. Scopus’ citation ‘curve’ for this paper looks quite steep: there were many citations within 5 years of publication and there have been very few in the last 5 years – others from the same edition of Nature have flatter trajectories indicative of lasting value. This suggests that the paper has not really become embedded in scientific understanding.

The 1988 paper reports an effective T cell reaction to the surface of adenovirus E1A transfected cells. No doubt these viral protein harbouring cells fully deserve their punishment. All reported indications are that E1A itself is the cell surface culprit that allows the T cells to go about their business.

Nevertheless, while E1A has no doubt been one of the most instructive viral proteins ever to have infected mammalian cells, isn’t its fame amongst the research community entirely due to shedding light on nuclear regulatory processes affecting cell cycle? (Tuning/blocking cell cycle in favour of generating more viral output.) Perhaps E1A experts could comment on its cell surface moonlighting activities and how Darwinian selection isn’t relevant to immuno-compromised adenovirus infected cells?

For me at least, it is difficult to reconcile a story about Nature ruling out discussion of irreproducible work published in their august journal, when they clearly are – very rightfully – willing to report on scientific fraud such as the above example.

Surely no one here would go so far as to suggest that Nature news and blogging items treat bad science published in Nature any differently to bad science published in non NPG group publications?

I feel Vaux’s pain. I spent two years and four submissions before Nature was willing to publish a piece of correspondence highlighting major technical problems in a Nature paper. As both us and the original authors agreed on the technical problems (but differed in how they affect the interpretation of the experiment), this should have been a much less painful experience. Nature simply was not interested in setting the record straight. Shameful.

Here we have the integrity of science being upheld by David Vaux whom I applaud, and yet an alleged stem cell self confessed liar published a series of further articles where the companies he names as collaborators don’t appear to exist!! One of his named co-authors also does not appear to exist.

Should all journals have a mandatory responsibility to check retractions from anyone submitting articles for publication? Should all authors be required to declare any retractions at submission? Following on any reseacrher with a retraction should then go through a series of further regulation, such as providing all raw data for scrutiny.

But wait, there is more! Surely David Vaux is in a unique position to comment on his experience of retracting Nature publications, for this Retraction Watch article apparently only tells half his story?

PubMed informs us that before his Nature News and Views retraction, Vaux already had a 1992 Nature retraction (PMID:1448156). This appears to be a retraction of a paper reporting his own experimental work?

Well, for better or worse, you may count me amongst the more avid followers of Retraction Watch. Nevertheless, for me this seems unique? Surely there is no comparable example of a researcher so willing to retract their Nature publications, whilst leaving the rest of their oeuvre undisturbed?

Just for the sake of clarity, I am David L Vaux, the author of the 1992 Nature retraction (PMID:1448156) was David JT Vaux. I have never met him, but I’ve followed his career with interest, not least because people often confuse us.

While I am here commenting, Retraction Watch readers might also enjoy reading this piece, published in Nature, that talks about similar problems in another journal:

Grrrr… this kind of thing makes me so angry! It’s insane how difficult it is to publish perfectly good scientific work simply because someone else previously published the opposite conclusion. I’m glad that there’s people like David Vaux out there.

I am in the middle of a mild version of this. I should note that it is mild because I left that part of the field, and am not looking back.

Anyway, as part of a larger project, I wanted to replicate the results of a group who showed that a protein we had been working on for many years could be shown to have a particular role in cultured cells (as published in a Cell Press journal). It was a bit puzzling, because another leading lab in the field had seen no such effect when they did the same thing, but what the heck, it would be straightforward and move us ahead. Well, after months and months of experiments, no effect. Even went to their lab to see the Magic Grad Student do the experiment, bought the Magic Lot of Serum, and looked cross-eyed at it. Basically,the effect they reported appeared to be an endogenous property of the cultured cells and independent of our protein. Did I mention that it was not technically difficult?

Meantime, the story evolved with each publication. Whereas all you needed to do in the first paper was express gene A, in subsequent papers (including Nature) one needed to co-express a few others and add a signaling molecule. Magic Student is now faculty in China, and no other human has replicated the result in 6 years.

Now I see another respected group has published in a journal that is Definitely Not Nature that they have independently replicated my results (it must be independent, because we have never spoken) showing that the Cell/Nature result is an artifact. That, along with one other publication and at least two other unpublished studies (not including mine) weigh the scales toward the effect not existing.

Allright, after all that exposition, here is the quandary: This gene is now known to the field on the basis of a function that it probably does not have. What does one do? My career will not be affected one way or another, but it does grate at me to think that many of us have wasted hundreds of hours trying to move forward with their observations, and many more will be wasted if it stays out there?

I suppose this is a bit of a hijack, so feel free to click “dislike,” or the mods can make it go away.

The protein is an ion channel, and the authors claim it allows sodium to leak through the membrane. I am being cagey because I am not sure I want to open that can of worms and accuse someone of fraud. What if the majority of us are really wrong? When I step back and look at it, there are two possible interpretations. 1) The rest of the people who have looked at the channel, all of whom are very experienced electrophysiologists, did not get conditions right. 2) The student who did the work engineered an elaborate fraud to arrange his data in order to make an endogenous leak current look like it was carried by our channel. It simply can’t be the result of sloppy work if they are wrong.

So far, it looks like people are saying “get it out there,” but arguing on the basis of negative results is difficult. My gut is telling me to wait until a good lab finds the “real” function of this protein, and there are a couple that are working hard at it right now.

You are taking this to much to heart.
My first job as a research technician was to replicate a paradigm shifting experiment in the USA (all red-blooded Americans I’m afraid, no Indians or Chinese to point the finger at here) that showed a certain protein had a growth inhibitory effect on cancer cells. I started off doing a dose response curve, no effect, I tried smothering the cells with it, no effect, I tried sneaking up on them late at night and dosing them by surprise, no effect.
After about 8 months the head of my institute grudging accepted that I wasn’t going to get any results, did this mean we published these negative results and correct the record? Not a bit! I was set to work showing the intracellular pathway of this (non-existent) inhibitory effect, while another technician was set to show that if you over-expressed the protein it would be inhibitory. I eventually, under supervision of a post-doc, got some results that might indicate intra-cellular signalling – although personally I never thought they were more that artefactual – in any case the effect was a very rare event. After 2 years I happily left for another position.
Meanwhile this group and the original group in the US happily published paper after paper, each bouncing off the other, proving the undoubted inhibitory effect of the protein, each citing the other’s work. Until after 12 years the head of one group found another position, just leaving the American group to keep churning them out.

In summary, what you describe is not rare or unusual in the slightest, it is what many excellent and productive careers are built on. Don’t upset the apple cart.

It is not my own observation, but one of a colleague, but I believe it is so apposite that it is worth sharing. It seems like high impact Journals today are in the same position as the catholic church in recent years with regard to sexual abuse. Otherwise reasonable decent people believe their institution’s “name” needs to be protected and thereby commit crimes vs decency and propriety. Science, unlike the church, never has anything to fear from complete honesty. Why can’t these journals get that and start respecting their target audience; rational, highly educated and articulate adults who know that it cannot all be true.

Now, let’s do the next step and try not be amazed by papers appearing in Nature more than by papers in other high-profile journals. And this step should include not just browsing the journals’ names in an applicant’s cv – will we manage?

There’s a very simple fact that is often ignored by publishers: scientific papers receive much more scrutiny after they are published than before. The recent somatic cell nuclear transfer human ES cell paper in Cell highlights this brilliantly–problems with the images were flagged almost immediately after the paper appeared on line, yet apparently these were not detected during the review process. We could call this sloppy review, but we are humans after all, and a few thousand pairs of eyes looking at something is bound to produce better results than the three or so pairs that reviewed the manuscript in the first place.

In cases like Vaux’s example, it took years of experimentation to get to the heart of the problem. This is not something that reviewers of the original paper could have picked up on. Unless we are going to require that data be independently reproduced before publication, it is inevitable that wrong conclusions will be published.

In other words, the published literature is fallible. Putting fraud aside, mistakes will be made, controls omitted, variables unaccounted for, and incorrect conclusions will be drawn. It is inevitable. Journals, in my opinion, need to adjust their posture to account for the fact that all papers are a work in progress, and be more receptive to publishing corrections, correspondence, and (worse case scenario) retractions.

But there is one crucial factor that scientists need to take to heart as we complain about the current situation and press for change. Journals like Cell, Science, and Nature are money-making entities there to serve the best interests of their publishers. They are not impartial keepers of the scientific record nor are they the scientific police. It’s much easier and more profitable for them to publish cool stories that may be incorrect and move on than it is to set the record straight. And we, the scientists, support this strategy by valuing publishing in these journals more highly than we do, say, society journals. The value of a well-timed Nature paper for a promotion or grant decision, for example, cannot be underestimated. We are the ones that make these journals sexy and are thus complicit in supporting the notion that, in terms of scientific success, it’s better to be first than to be correct.

I believe that the sort of change many of us recognize as being sorely needed begins with us. We need to change how we value different journals and how we measure scientific success. I am very pleased to see that almost 10,000 of us have signed the San Francisco Declaration on Research Assessment (http://am.ascb.org/dora/) which calls for a total revision of how the output of scientific research is evaluated and offers a very thoughtful set of guidelines for how this can be achieved. I encourage everyone to visit the site, sign on if so inclined, and begin implementing change.

Usually, scientists use the citations of a paper as a good source for quality; but sometimes, one has to cite just wrong results and conclusions to show the knowledge about other scientists findings and to be able to discuss this. Therefore, even scientifically questionable papers, or publications which turn out to be wrong or not as accurate they appeared to be at the time published can get high citation rates.
Unfortunately, sometimes scientists knowingly publish their different results due to the pressure to get funding or a carreer boost. I remember 20 years ago I was looking carefully for CD44 splice variants (CD44v) in glioblastomas – and found them with any method, i.e. reverse PCR, as well as immunohistochemistry with polyclonal antibodies against several different variant epitopes (Eibl RH et al. 1995), and could confirm that also with monoclonal antibodies (unpubl.). Although I presented the results in lab meetings a member of the group who turned out to collaborate with another group then published the lack of CD44v in glioblastomas (Cancer Res. 1993 Nov 15;53(22):5345-9. Variant CD44 adhesion molecules are expressed in human brain metastases but not in glioblastomas.). Of course, both groups did not investigate the same material, but it appeared to be unlikely to see just the opposite outcome. Of course the difference in detection capabilities could explain the outcome – and in my results the level of CD44v was usually very low, but not absent. Since Cancer Research is quite a good journal (I also published there three times) one can not say it was wrong accepting that paper, but it is clear that my findings published in a less important journal won’t be cited as much as the other paper. That is science.

I can only add that I found the major conclusion of a Nat Genetics paper wrong on the basis of the published data, and the author acknowledged that in a conversation with me. Nat Genetics however refused to print a retraction. I had a similar experience with a Commentary in Nature. I think this must be a fairly frequent occurrence. Nature seems to believe it either cannot afford too many retractions or simply refuses to accept its own fallibilty.

It has been my personal experience that the editors of Nature Medicine play favorites. They allow certain big names to influence the review process of others in a very unfair manner.

At some point in time, we will all need to stop citing all nature journals out of pure protest. The only way to bring fairness back to the system is by the scientific community standing together and hitting them where it hurts the most, their impact factor, because that is all they care about.

Double-blind peer review, in which neither the editor who decides whether a paper is sent for review, nor the referees, are told the authors’ names or affiliations, would reduce bias and conflicts of interest in publications. Imagine how science would be if papers were published according to the quality of their scientific content, rather than who the authors are or where they come from.
The adoption of double-blind clinical trials was one of the greatest advances in medical research, and no-one today would think of abandoning it. In 50 years time (hopefully less), when double-blind peer review is universal, we will look back and wonder why we put up with the current system, with its cliques, back-scratching, back-stabbing, xenophobia, and corruption, for so long. Double-blind peer review is not perfect, but at least it will increase the likelihood that editors and reviewers actually read submitted manuscripts before they make their decisions.

Thanks for a really interesting post and to all for insightful comments. These highlight the scale of the problem. At one level, NPG is not going to forego the 900 citations, they sell on IF, as do many journals. So here we are the solution, because we lazily reach for Ifs as a proxy to actually reading a bunch of papers. We ignore the fact that IFs are the result of a highly skewed distribution and using these as a Gaussian is laughable at our peril.
Back to the “problem paper”. We really do have a major issue, because, as many have noted, it is very difficult to figure out that a paper has a problem. I believe that in the days of papers in journals read in the library, when a paper was retracted, the librarian would get a stamp out and stamp “retracted” on each and every page of the article. In the days of electronic papers, this is very easy, but does not happen. This is again a symptom of publishers protecting a perceived brand quality to keep ahead of the market.
For those in need of a dose of irony, there is a recent editorial in Nature on enhancing data quality in the papers they publish. Read this and then re-read the above post.

Readers, I applaud the efforts of Dr David Vaux to clean up the scientific record by retracting his own mistaken published commentary and, in the process, highlighting the weaknesses of editorial processes at scientific journals, including the prestigious journal Nature.

Over the past 18 months, I have been arguing near and far for the correction or retraction of the University of Sydney’s spectacularly faulty and self-published – yet “peer reviewed” – Australian Paradox paper, a ham-fisted attempt by the University’s leading food-industry service providers to exonerate sugar as a menace to public health: http://www.australianparadox.com

I encourage the University of Sydney’s Professor Jennie Brand-Miller and Dr Alan Barclay – and its senior management – to give priority to the integrity of the scientific record and retract their faulty paper with its obviously false “finding” of “an inverse relationship” between sugar consumption and obesity.

Nature is a scam. Many years ago, I sent a manuscript for the business section of Nature. It was about the power of collective wisdom. The editor at that time expressed interest and after 2-3 months, an editorial staff of Nature published it under his name

When I contacted the editor of the business ection, he told me that they were working on this in parallel. If this was the case, they could have informed me when I submitted. Dont expect high standards from the editors!

As a scientist with a deep interest in bioethics, I was deeply concerned to find what look like serious anomalies in a number of manuscripts by David Vaux. This is an individual that has clear ties with the industry creating severe conflicts of interests. As a self-proclaimed science police who attacks scientific publications as a vigilante, he is not in a position to author papers with figure manipulations. A picture is worth a thousand words, have a look at these 3 examples. There is more to come. Note that John Silke and David Vaux are common authors on these 3 papers.

1. Figure 2A (PMID: 11604410). A square in upper right corner appears when image is color-inverted and contrasted. A western blot image does NOT naturally contains a distinct square with different contrast and pixel pattern. The membrane, the film or the image appear to have been altered.

The phrase you might be seeking is “People who live in glass houses shouldn’t throw stones”. However, might I point out that nobody is in a position to author papers with image manipulations, not just those who complain about fraud when they come across it from others. Who amongst those of us that have co-authored multi-author papers can be sure that we are immune from being hit by fraud in our own publications? How would we behave if it happens? Let us see how David Vaux responds to your reporting before playing the hypocrisy card.

And yes, it is clear that these images are very troubling. It will be very interesting to see what else you have.

Regarding the comment from Pubpeer 1 January 25, 2014, it is good to be skeptical of any and all published research. Nothing should be closed to scrutiny. Similarly, challenges to published research should be answered by openness, and provision of original data when it is possible.

I have only one tie with industry, but I am glad it is clear. I do try to always declare any conflicts of interest I have. I first acted as a consultant for Gentara (now TetraLogic Inc) in 2005, and I am currently on their Scientific Advisory Board. They do not fund any research in my laboratory, but have supplied chemicals, such as smac-mimetic compounds. The total remuneration I have received from them over the last 10 years is less than $20,000 USD. In any relevant publications I declare my relationship with them.

To publish in NSC journals (you should know what NSC means), you have to be a member of an ‘elite club’. If not, you are just wasting your time sending articles to them for publication. How could your competitors for the next grant favour your chances of becoming a better opposition? Just think about that and you would understand why you need to be a member of that ‘elite club’. Public or perish, good bluck.