To Retract or Not to Retract… That’s the Question

The decision of the editors was based on the failure of at least 10 other studies to confirm these findings and on growing support that the results were caused by contamination. When the authors refused to retract their paper, Science issued an Expression of Concern [2].

In my opinion retraction is premature. Science should at least await the results of two multi-center studies, that were designed to confirm or disprove the results. These studies will continue anyway… The budget is already allocated.

Furthermore, I can’t suppress the idea that Science asked for a retraction to exonerate themselves for the bad peer review (the paper had serious flaws) and their eagerness to swiftly publish the possibly groundbreaking study.

And what about the other studies linking the XMRV to ME/CFS or other diseases: will these also be retracted?
And what happens in the improbable case that the multi-center studies confirm the 2009 paper? Would Science republish the retracted paper?

Thus in my opinion, it is up to other scientists to confirm or disprove findings published. Remember that falsifiability was Karl Popper’s basic scientific principle. My conclusion was that “fraud is a reason to retract a paper and doubt is not”.

This is my opinion, but is this opinion shared by others?

When should editors retract a paper? Is fraud the only reason? When should editors issue a letter of concern? Are there guidelines?

Schekman considers it “an unusual situation to retract a paper even if the original findings in a paper don’t hold up: it’s part of the scientific process for different groups to publish findings, for other groups to try to replicate them, and for researchers to debate conflicting results.”

I don’t have any hard numbers on how often journals ask scientists to retract a paper, only my sense that it is very rare. Author retractions are more frequent, but I’m only aware of a handful of those in a year. I can recall a few other cases in which the authors were asked to retract a paper, but in those cases scientific fraud was involved. That’s not the case here. I don’t believe there is a standard policy that enumerates how such decisions are made; if they exist they are not public.

However, there is a Guideline for editors, the Guidance from the Committee on Publication Ethics (COPE) (PDF) [5]

Ivanoranski, of the great blog Retraction Watch, linked to it when we discussed reasons for retraction.

With regard to retraction the COPE-guidelines state that journal editors should consider retracting a publication if:

they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error)

the findings have previously been published elsewhere without proper crossreferencing, permission or justification (i.e. cases of redundant publication)

it constitutes plagiarism

it reports unethical research

According to the same guidelines journal editors should consider issuing an expression of concern if:

they receive inconclusive evidence of research or publication misconduct by the authors

there is evidence that the findings are unreliable but the authors’ institution will not investigate the case

they believe that an investigation into alleged misconduct related to the publication either has not been, or would not be, fair and impartial or conclusive

an investigation is underway but a judgement will not be available for a considerable time

Thus in the case of the Science XMRV/CSF paper an expression of concern certainly applies (all 4 points) and one might even consider a retraction, because the results seem unreliable (point 1). But it is not 100% established that the findings are false. There is only serious doubt……

The guidelines seem to leave room for separate decisions. To retract a paper in case of plain fraud is not under discussion. But when is an error sufficiently established ànd important to warrant retraction?

Apparently retractions are on the rise. Although still rare (0.02% of all publications by the late 2000s) there has been a tenfold increase in retractions compared to the early 1980s (see review at Scholarly Kitchen [6] about two papers: [7] and [8]). However it is unclear whether increasing rates of retraction reflect more fraudulent or erroneous papers or a better diligence. The first paper [7] also highlights that, out of fear of litigation, editors are generally hesitant to retract an article without the author’s permission.

At the blog Nerd Alert they give a nice overview [9] (based on Retraction Watch, but then summarized in one post ;) ) . They clarify that papers are retracted for “less dastardly reasons then those cases that hit the national headlines and involve purposeful falsification of data”, such as the fraudulent papers of Andrew Wakefield (autism caused by vaccination). Besides the mistaken publication of the same paper twice, data over-interpretation, plagiarism and the like, the reason can also be more trivial: ordering the wrong mice or using an incorrectly labeled bottle.

Still, scientist don’t unanimously agree that such errors should lead to retraction.

“I don’t give a fig what any journals might wish to enact as a policy to overcompensate for their failures of the past.In my view, a correction suffices” (provided that search engines like Google and PubMed make clear that the paper was in fact corrected).

Drug Monkey has a point there. A clear watermark should suffice.

However, we should note that most papers are retracted by authors, not the editors/journals, and that the majority of “retracted papers” remain available. Just 13.2% are deleted from the journal’s website. And 31% are not clearly labelled as such.

Summary of how the naïve reader is alerted to paper retraction (from Table 2 in [7], see: Scholarly Kitchen [6])

Watermark on PDF (41.1%)

Journal website (33.4%)

Not noted anywhere (31.8%)

Note appended to PDF (17.3%)

PDF deleted from website (13.2%)

My conclusion?

Of course fraudulent papers should be retracted. Also papers with obvious errors that invalidate the conclusions.

However, we should be extremely hesitant to retract papers that can’t be reproduced, if there is no undisputed evidence of error.

All retracted papers (and papers with major deficiencies and shortcomings) should be clearly labeled as such (as Drugmonkey proposed, not only at the PDF and at the Journal website, but also by search engines and biomedical databases).

One day the scientific community will trade the static print-type approach of publishing for a dynamic, adaptive model of communication. Imagine a manuscript as a living document, one perhaps where all raw data would be available, others could post their attempts to reproduce data, authors could integrate corrections or addenda….

Derek Lowe at In the Pipeline came to much the same conclusion that you did:

“There will be also be some people who ask whether Science did the world a favor by publishing the original paper in the first place. But on balance, I’d rather have things like this get published than not, although in hindsight it’s always easy to say that more experiments should have been done.

The same applies to the arsenic-bacteria paper, another one of Science’s recent bombshells. I’m not believing that one, either, at this point – not until I see a lot more supporting data – but in the end, I’m not sad that it was published, either.

I think we’re better off erring a bit on the wild-ideas end of the scale than clamping down too hard.

That said, you do have to wonder if Science in particular is pushing things a bit too hard, itself. While I think that these ideas deserve a hearing, it doesn’t necessarily have to be there.”

I’m not against wild hypotheses, if presented in opinion papers. On the contrary. But original scientific papers, should refrain from wild theories, if not founded upon sound data. The Science papers was flawed for reasons mentioned in the previous post.

Terrific post (as usual!). And thanks for leaving the nice comment on my Science 3.0 blog. I must say that the Editors requesting a retraction seems to me to be a cowardly way of worming out bad press over poor peer reviewing. Hmm… I hope Richard Smith gets all down and dirty with more evidence to support his claim that peer review is flawed!

The study maybe poorly designed, but that should not be a criteria for retracting it. From what I have read, it appears to be more of a case of contaminants polluting a study rather than falsification of results. While errors are pardonable, forgery is not. So, the request for a retraction does seem a little over kill (and the cowardly way out thingy as well). At least to me!

I don’t know if I understood you well and if you agree with me or not. I meant that if the study had been better designed (good controls) or better checked it would likely not have been accepted as such. (and thus no need for retraction now). When contaminants are to blame for the positive results, the hypothesis (XMRV being associated with CFS) is being falsified, isn’t it?
But as said, retraction now does seem premature.

As guess as someone suggested, I AM surprised who agrees with me. Actually, I don’t have the technical background to argue too much; it’s me who’s learned some things from this and another column, but I’m grateful for the less inflamed discussion here.

When concerns finally were raised about Wakefield’s 1998 paper, 10 of 12 authors backed away from it. That’s far from the case here, ‘though it seems only Derek Lowe wants to remind the world there’s still an XMRV debate going on somewhere. I likewise told Science: caution is fine, but unless you’ve proven fraud, let other researchers and history render the verdicts. I think we’ve seen both sides overplay the “autism card” here. Start pulling papers at the whim of whomever, and it hurts all scientific research, which has enough problems these days already.

You reasoning makes perfect sense. It is clear that WPI adheres to their original experiments. I don’t know whether this proofs anything except that the authors actually believe in XMRV as a causal phenomenon themselves, and this belief is not affected by the negative studies and the demonstration of contamination as a likely cause. It is a pity the debate has never been open from the start. Therefore it has always been “Them against US” (and vice versa), including theories of conspiracies (also vice versa). Not good.
Pulling papers without clear proof of fraud or serious error is doesn’t solve anything. Sooner or later we will definitively find out the real facts behind all this. If necessary, the paper can always be retracted than.

I reviewed this in my previous posts. This post was more about the process of retracting, with XMRV as an examples (so I left the technical details out this time).

My critic mainly concerned the PCR:

In short. No blinding, patient samples on one gel, the negative controls on another. Strong PCR bands after ONE round (while the PCR appeared suboptimal)… No good contamination controls, no sensitivity given, homogeneity of the sequences (yes already determined from the start).

After the paper was published, WPI suddenly stated that culturing blood cells was necessary for PCR. However, this was not mentioned in the Science paper. (this couldn’t be noticed by the reviewer perhaps).

Saying that the peer review failed is probably exaggerated, but lets say that an expert in PCR (and perhaps some virologists aren’t) should have seen this and ask for a better design and better checks. If a better design still gave the same results and a greater homogeneity would have been demonstrated, the paper would have been more robust (only some extra time lost). If not, it would not have been published (at least in Science) and we could have prevented a lot of fire.

The details clearly stated the need to activate the PMBCs. The single round PCR (copied by Knox but made less sensitive) only produced GAG sequences in 7% of patients examined. The reverse transcriptase nested PCR which detected GAG sequences in 67% of patients was based on the PCR of Urisman et al., who optimised their assay to detect GAG sequences in cDNA extracted from patient VP-35. They ammended the protocol by reducing the annealing times of the assays. The samples were, contrary to popular reporting, blinded. Dr’s Lombardi and Ruscetti then attempted to refine the assay further by culturing the activated PMBCs using a very specific protocol. This built up the titre sufficiently to enable the PCR to detect previously undetectable levels of GAG sequences. This information was made public in early december 2009 and latterly by an addendum published in Virulence. Other laboratories responded to this information by using a single round PCR assay using different reagent and cycling conditions on whole blood, thus dramatically reducing the proviral titre, if the provirus was present. They then declared that they were sure that they would have found it if present.

The Lombardi study was not based on PCR, but also involved the activation and culture of PMBCs and T and B cells prior to serology, once again to raise the tire of viral antigens. One of the monoclonal antibodies used produced a reaction which is specific to the SFFV env antigen. No one has been able to explain this finding using their contamination hypothesis and no one has been willing to use this antibody or the serology methods used in any way to challenge this finding

The transmission EM images showed both immature and mature virions, which once again cannot be explained by contamination. The presence of immature virions demonstrate post release modification of capsid proteins.

The study design of Lombardi et al. should be familiar enough. It was the same protocol which successfully discovered another human retrovirus namely HTLV-1. This is hardly surprising, as the same person was largely responsible for designing both studies. That person was Frank Ruscetti. Many people using other methods had failed to isolate HTLV-1 and now people using their own pet methods are failing to detect human gammaretroviruses. In short history is repeating itself.

It must be remembered that a survey in 1996, some 13 years after the discovery of HIV, showed that less than 40% of labs could reliably detect the virus.

The theory that HIV was a contaminant, which entered the human population via monkey cell cultures used in vaccine preparation, was supported by many self opinionated retrovirologists, who had declared that infection of humans by retroviruses was an impossibility.

Imagine where the human race would be now if this speculation had been allowed to crush future research into HIV!

He links to http://www.retrovirology.com/content/8/S1/A234 (Detection of MLV-like gag sequences in blood samples from a New York state CFS cohort) and he states that his pronouncement of the subject as dead may have been premature.

These new findings (and it is only an abstract at a conference, so I can’t check the technical details) don’t change my view, but confirm that retraction is premature. (thus Derek, you and I agree at this point)

Whether this new study really confirms the WPI findings and whether they are good, remains to be seen.

this paper does not address the contamination in the Lo/Alter paper. However, I believe the Lo/Alter contamination will be addressed in an upcoming publication. Pure speculation here: XMRV is the result of two previously undescribed ERVs (PreXMRV-1 and-2). Lo/Alters sequences are such obvious ERVs that I doubt there is any virus involved– Im assuming its straight mouse DNA contamination from reagents, as opposed to XMRV which can be an exogenous virus that is contaminating cell lines

You should indicate which paper you refer to. This comes out of the blue. We were discussing other sources.

17062011

Charlie(12:47:03) :

I did, Paprotka et al.

9062011

Jack(23:45:22) :

Many thanks for information relating to COPE and your post in general.

WPI have always claimed that they have investigated contamination and even gone so far as ruling out contamination (even if that claim later proves incorrect), therefore this point cannot surely apply at this point, unless proof of investigation is not presented or considered adequate, I suppose:

‘2. there is evidence that the findings are unreliable but the authors’ institution will not investigate the case’

If the stalemate (exampled most visibly at the State of Knowledge conference perhaps), is being taken out of the hands of WPI, i.e. other studies and opinions are increasingly saying contamination is there and that WPI just have not looked hard enough; then what happens next?

Hi Jack, IF one of these two points is met editors should write an issue of concern. The Science editors have already done that. Which is -popularly said- a way to say: we take our hands off. This is according to the guidleines, imo. But I don’t think a request for retraction was justified at this stage.

If contamination is a fact (and the latest studies are very convincing), then we can assume the data in the paper are incorrect and retract it (or stamp it as wrong).

The story of Human Gamma Retroviruses (including XMRV) and other replication competent retroviruses (RCR’s) is much larger than the Science paper. This is the “tipping point” IMHO.

While many will trash the 2009 Science paper, there are substantial errors in other published negative papers. Have retractions been suggested for these papers?

There is a public health risk. Ego’s are at stake. New competition for research funding is not welcome if you happen to be studying something else . There is money to be made and prizes (Lasker/Nobel) in the minds of people studying this area.

This will eventually come out into public knowledge but much like HIV, there is denial, controversy, scientists behaving badly, etc.

If others elect NOT to truly replicate a study but choose to vary multiple variables and they find nothing, it is not a replication study.

Major corporations are heavily invested. Not everyone will show their hand given the high stakes.

To discredit the WPI alone is ridiculous. Peer review should be peer review.

Some of these “scientific” publications violated fundamentals of scientific practice. What is sad is that real people/patients are suffering while politics, greed, egos, patents, etc are being played out.

16062011

Charlie(23:48:03) :

You mean the WPI, Cleveland clinic and NCI.

18062011

JT(04:13:29) :

@ Charlie and others… just watch what unfolds.

While WPI, ,Cleveland Clinica, and the NCI may receive negative biased press, the bottom line is that it will not last forever. Should be interesting to see how long it will last.

I have minimal time to respond to this blog so you can all have fun stating whatever you want..If you are in good health now, relish it and protect it. That may not be the case in the future.

We were discussing the issues surrounding a retraction, not, I understood, any one or several organisations.
Even if you take the retraction in question, aimed at the Lombardi paper, it is not directed at WPI as an institution, but the authors of that paper and the claims that were made.
And JT – if you are the same JT who has been claiming ‘watch this space’ for eons now, well I am still watching, and so far the authors of Lombardi have done diddly to defend themselves as far as I am concerned – other than Dr Mikovits and her supporters, delving again into emotive issues that should form no part of this.
This discussion was surrounding the reasons for the asked for retraction, and those surrounding retraction in general. I dare say that ‘Science’ will await the results of Lipkin and the BWG before taking this further – or maybe the ‘letter’ will prove sufficient?
At the very least it will flag the Lombardi paper in future. This publication has raised the whole contamination debate to a new level. Scientists need now to be even more aware of the risks and to further investigate – perhaps – what it is these strains of virus are actually capable of, though for many – I suspect – and in relation to ‘XMRV’ the issue is settled.
Not though for others with a vested interest and who see the ‘fight’ as far from over, whether that is from an inherent and genuine belief that ‘they are right, when others are wrong’, or because they have dug too deep a hole for themselves and will not revisit and recheck ‘old’ ground. We shall see.
I for one, find the debate about retraction quite interesting, and am able to consider it without recourse to ‘conspiracy’, which I feel is the ‘easy way out’ of anything one cannot or will not comprehend.

@Jack… not sure what you’re referring to regarding “Watch this space”. I suspect there must be another JT.

Regarding retraction, I still believe evidence will support the WPI’s findings for a new group of retroviruses being associated with and or causing disease. WPI may not get credit for the discovery since there is so much at stake but their work has prompted many in the scientific community to look again at retroviruses as causative agents and to review their own labs/cell lines/practices.

As you mentioned, every scientist needs to follow safe laboratory practices. Working everyday with materials that are hazardous/infectious can result in less than perfect adherence to known safe laboratory practices. I have been guilty of it myself.

@JT Thanks.
Lombardi touted a ‘link’ with CFS and were not discoverers of a retrovirus ‘XMRV’ let alone anything that could prove causation. They would like to have done – but didn’t.
What the future holds…? Who knows, but the paper, Lombardi et al 2009, is what is at issue here and not, for that matter, the WPI – though granted it has become a ‘one-horse-race’ for them. Kind of a big mistake if you ask me – especially now the ‘clinic’ is purportedly opening – again. We could talk about the WPI and its’ ‘failings’ in my eyes, but this was and should be about Lombardi – and more particularly, here, the asked for retraction.
Would the same have been asked of any other paper? Yes I think it would, given the circumstances, and those prostate cancer papers, will inevitably – surely – face a similar critique. BUT this was about the establishment of a ‘link’ with specifically a cohort of ‘CFS’ patients, whose blood (not anything else), may not now even have carried ‘XMRV’ – ergo the paper was wrong to suggest such a thing.
Had the scientists involved been able to prove and not use rhetoric, but prove themselves and their findings to be above even the claims of contamination – well they have had plenty of time to do so – more than sufficient time. And even subsequent to the paper, the ‘evidence’ (widely touted as being conducive to proof) for an infectious agent in humans did not live up to scrutiny or was not produced – heck those samples sent to Genbank didn’t even warrant detailed scrutiny.
Yet this retraction business is interesting, as without this paper, the light would not have been shone on contaminants in the way that it has – correct me if I am wrong here do as I am no scientist – but continue to find the whole issue interesting, particularly as someone with the most important personal stake in this I suppose: that of being a patient.

I have to say – apart from it being always a useful and productive exercise to re-read previous articles – that having again digested your article:https://laikaspoetnik.wordpress.com/2010/04/27/three-studies-now-refute-the-presence-of-xmrv-in-chronic-fatigue-syndrome-cfs/
– I am left in even less doubt that calls for ‘replication’ are ill-advised and unfounded. How to replicate a paper that is ‘flawed’ to the extent that you (and others), have reported?
It just does not make sense and never has done – like ‘consipracies’, calls for ‘true replication studies’ are a ‘cop-out’ – in regard to the Lombardi paper as it was published in 2009.
Sorry I seem to have occupied enough of your blog – will now bow-out :)