Post-publication peer review in public: poison or progress?

What a difference a few days make. The much-ballyhooed story of arsenic-utilizing bacteria stemming from NASA’s embargoed press conference and paper in Science (somewhat ballyhooed here by yours truly), is accumulating criticism from microbiologists calling into question the degree of rigor applied to some of the paper’s experiments.

[Update: For an excellent, unexplored take on the analytical chemistry used in the paper, see this upcoming C&EN article by my colleague here, Dr. Carmen Drahl.]

On Sunday, I added to my post the first chink in the armor: a detailed technical critique by University of British Columbia microbial geneticist, Dr. Rosemary (Rosie) Redfield. Dr. Redfield has long found value in blogs and has encouraged her trainees to have their own blogs to openly discuss their science.

I’ve done some of the techniques (awhile ago) that Redfield discusses from the paper but it even took me some time to go through her critique. The central theme of her criticisms is that the experimental results leading to the conclusion that the Mono Lake GFAJ-1 bacterium can grow using arsenic instead of phosphorus may be an artifactual: the detection of arsenic in the bacterial DNA could have been due to insufficient clean-up of the DNA prior to ICP-MS analysis and that trace amounts of phosphate in the media and from dead cells could have provided the remaining cells with enough phosphate to survive in 40 mM arsenate. For example, the former artifact might result because the authors used phenol-chloroform to extract bacterial DNA from agarose gel slices which might have carried over some non-specifically bound arsenic that might not have occurred had a Gene-Clean type glass beads purification been done instead.

Here’s an excerpt of her conclusions:

Bottom line: Lots of flim-flam, but very little reliable information. The mass spec measurements may be very well done (I lack expertise here), but their value is severely compromised by the poor quality of the inputs. If this data was presented by a PhD student at their committee meeting, I’d send them back to the bench to do more cleanup and controls.

There’s a difference between controls done to genuinely test your hypothesis and those done when you just want to show that your hypothesis is true. The authors have done some of the latter, but not the former.

As painful as her analysis might have been to read for the authors, NASA, and editors of Science, Redfield’s critique represents the future of post-publication peer-review. Already the present at journals such as those from PLoS where readers can comment on papers at the journal website, open public discussion of science immediately upon release of a paper will help science progress more quickly. No longer confined to the archaic correspondence to the journal that takes months to view, the internet and the science blogosphere is facilitating open discussion of peer-reviewed publications within hours and days of release.

And this is where I follow-up on my longstanding position that all scientists should have a blog of some sort that, at the very least, can capture the content and discussion of journal clubs. Of course, the frank discussion at journal clubs requires that one also be willing to open up one’s own work to similarly rigorous criticism.

So, it’s still somewhat unusual for the legacy scientific publishing establishment to encounter such criticism from a senior scientist such as Redfield. But it was the response of NASA and the authors that has provoked much head-shaking among the scientific community.

When NASA spokesman Dwayne Brown was asked about public criticisms of the paper in the blogosphere, he noted that the article was peer-reviewed and published in one of the most prestigious scientific journals. He added that Wolfe-Simon will not be responding to individual criticisms, as the agency doesn’t feel it is appropriate to debate the science using the media and bloggers. Instead, it believes that should be done in scientific publications.

It’s difficult to get more hypocritical than this – the story was hyped for days and promoted by press release and press conference. Yes, the paper was subjected to peer review by one of the world’s leading journals but – but dear god, people – grow a thicker skin if you expect to hump a research finding publicly but not expect that it will be subjected to rigorous post-review analysis by scientists who know how to use the internet and set up a Blogger or WordPress blog.

David Dobbs was more erudite than I in his post yesterday at his Neuron Culture blog at Wired, “The Wrong Stuff: NASA Dismisses Arsenic Critique Because Critical Priest Not Standing on Altar”:

This is a call to pre-Enlightenment thinking. Brown is telling us to judge utterances not by their content, not even by the integrity, reputation, and experience of the individuals who deliver them, but by whether they’re delivered from the proper place in the proper building — in pre-Enlightenment days, the Church of Rome; in Brown’s post-arsenic days, the Church of the Peer-reviewed Journal.

It’s an extraordinary dismissal. Rosie Redfield is a full-bore member of the academy and a researcher in the field under question. She is — to extend the metaphor — a priest. But though Redfield wears the proper robes, Brown wants to dismiss her because she’s not standing on the proper altar.

Earlier this morning via Twitter, Ivan Oransky of Embargo Watch pointed out that comments of this sort accumulating at his blog post sound similar to the editorial of Royce Murray’s we covered a few weeks ago.

“We cannot indiscriminately wade into a media forum for debate at this time,” declared senior author Ronald Oremland of the U.S. Geological Survey. “If we are wrong, then other scientists should be motivated to reproduce our findings. If we are right (and I am strongly convinced that we are) our competitors will agree and help to advance our understanding of this phenomenon. I am eager for them to do so.”

“Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated,” wrote Felisa Wolfe-Simon of the NASA Astrobiology Institute. “The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner.”

Zimmer is not at the altar – he’s just one of the most prominent and decorated science writers of our time. In fact, Oransky called him, “the dean of my generation of science writers.” But don’t you see, Carl? Asking for comment “do[es] not represent the proper way to engage in a scientific discourse.”

Again, from Zimmer’s Slate article:

While Redfield considers Wolfe-Simon’s research “flim-flam,” she thinks it’s fine for the NASA scientists to hold off responding to their critics. She is working on a formal letter to Science detailing her objections. But Jonathan Eisen of UC-Davis doesn’t let the scientists off so easily. “If they say they will not address the responses except in journals, that is absurd,” he said. “They carried out science by press release and press conference. Whether they were right or not in their claims, they are now hypocritical if they say that the only response should be in the scientific literature.”

Let it be then. The former water cooler, conference room, and pub critiques of scientific work are now taking place in the widely-accessible and, yes, messy venue of online communication. If authors don’t wish to engage and defer discussion for a few weeks and let letters to be sent to Science and published, that’s their prerogative. Perhaps Dr. Redfield should chisel her letter on a stone tablet. I’m sure that a telegraph line might still run from Vancouver to Washington, DC.

In defense of Science, Oransky opened his post yesterday by quoting a much more reasoned and thoughtful comment from AAAS director of public programs Ginger Pinholster who responded to his post on December 3rd as to how the story took on a life of its own.

I don’t envy the authors – my observation has been that NASA put them in a difficult situation by how they hyped the December 2 press conference as revealing, “an astrobiology finding that will impact the search for evidence of extraterrestrial life.”

Me? I can assure you of one thing: We’re going to go back and triple-check some of our own data before sending in our next revised manuscript.

And no press conference.

UPDATE: Literally minutes after I scheduled this post to go up, Dr. Felissa Wolfe-Simon put up this post at her professional website:

Wolfe-Simon et al Comment: 08 December 2010
My research team and I are aware that our peer-reviewed Science article has generated some technical questions and challenges from within the scientific community. Questions raised so far have been consistent with the range of issues outlined by journalist Elizabeth Pennisi in her Science news article, which was published along with our research. For instance, other scientists have asked whether the bacteria had truly incorporated arsenic into their DNA, and whether the microbes had completely stopped consuming phosphorus. Our manuscript was thoroughly reviewed and accepted for publication by Science; we presented our data and results and drew our conclusions based on what we showed. But we welcome lively debate since we recognize that scholarly discourse moves science forward. We’ve been concerned that some conclusions have been drawn based on claims not made in our paper. In response, it’s our understanding that Science is in the process of making our article freely available to the public for the next two weeks to ensure that all researchers have full access to the findings. We invite others to read the paper and submit any responses to Science for review so that we can officially respond. Meanwhile, we are preparing a list of “frequently asked questions” to help promote general understanding of our work.

Although much of the attention has been on Rosie Redfield’s initial critique, Alex Bradley’s guest post at We, Beasties on ScienceBlogs appeared the next day and added to the discussion of the hydrolytic lability of arsenic esters in putative As-containing DNA.

I find this article by Martin Robbins and this one by Ed Yong to be two of the most thoughtful post-mortems on the story thus far.

Share this:

9 Comments

The former water cooler, conference room, and pub critiques of scientific work are now taking place in the widely-accessible and, yes, messy venue of online communication. If authors don’t wish to engage and defer discussion for a few weeks and let letters to be sent to Science and published, that’s their prerogative.

Almost exactly my thoughts from my (a bit too early!) take on it. (My view is more clumsily expressed, too.)

The way I read some others writing on this subject is there is a case that you have to carry on as you started, i.e. having chosen to start with a press release/conference approach (I’m assuming the authors had a say in the material offered), then are they obligated to keep using that approach-? (I’m chickening out and not saying either way on this for the moment!)

If the peer reviewed journal ‘shrine’ is so infallible how did this article get published in the first place? Media must take some responsibility, concur with your first paragraph, like white pointers amid a school of whiting, we were guilty of publishing ad verbatim.

David, I can’t agree with you more about the hypocritical aspect of this story: hyping the discovery with a press conference, then criticizing the blogosphere for analyzing it.

And to the commenter above, FWS is the starred author. She’s the responsible party who should be standing up to the criticism.

Noah Gray

9 December 2010

Hi David. I completely agree with you and we seem to be watching scientific discourse transform right before our eyes. And this is a good thing.

Just one note, for quite some time now, Cell Press has had commenting enabled, while Nature-branded titles have added such functionality recently. But as all publishers note, such features are still slow to be used by the public. Such will remain the same too, until scientists feel they are getting some “credit” for sharing their intellectual property in a comment thread.

Gaythia

9 December 2010

There needs to be a line between “held responsible” and “held solely responsible”. And support for attempting to do the right thing. In this case, I think FWS does seem to be attempting to do the right thing.
There are many situations in which the in-house, in-lab scientist is on the line. That is as it should be. That scientist may need to correctly draw a limit, and do such things as delay or refuse a press conference, withhold publication pending further data, or stop production. In so doing they need the full backing of their peers. (for the purposes of this blog, the ACS comes immediately to mind).
Also, it needs to be possible to publish preliminary results, and, even if not fun, risk being wrong. Historically, papers were frequently reviewed by and then read by a fairly incestuous group of interlinked scientists. Turning this into a world wide free for all is a fruitful way to get to the bottom of things. But if it causes scientists to be too hesitant to come forward with ideas, and wait for perfection, then science loses.

Faraday

9 December 2010

It must be hard for FWS. She seems like a good scientist with very interesting results. I’d be willing to bet the real culprit of this madness are the not-so-science-educated higher-ups at NASA eager to push the story.

Noah, thank you for the reminder about Cell Press and Nature. I’d been aware of this but understand that the features have not been terribly popular, in part, for the reasons you notes.

Gaythia, I agree with most of your statements and, like Faraday, have compassion for FWS (although she is the corresponding author and, as Paul noted, bears more responsibility for the work than the more senior colleagues on the paper).

I don’t think this episode will necessarily cause scientists to hesitate to come forward with ideas and wait for perfection. Instead, this may cause funding agencies and journals to be more cautious in holding such highly-hyped press conferences

Gaythia

11 December 2010

In a David Dobbs (Neuron Culture blog) list of links I found the following: http://michaelnielsen.org/blog/the-mismeasurement-of-science/ which I think is worthy of further contemplation as it relates to the above and also such things as the discussions elsewhere on CENtral Science on the future of jobs in Chemistry.

As David Dobbs put it: http://www.wired.com/wiredscience/2010/12/aglitter-in-the-net-2/ “The issue of what scientists get credit for — publication, being famous, reviewing papers, whatnot — has a huge influence on what scientists do. They react to incentives just as everyone else does. And right now, almost all the incentives favor flashy publications in certain journals, even as they discourage data-sharing and the evaluation of others’ work and fail to necessarily reward the biggest ideas.”

We could add highly-hyped press conferences to the above list, as well as look at what incentivises PhD chemist production and fields of study.

About the Safety Zone

The Safety Zone covers chemical safety issues in academic and industrial research labs and in manufacturing. It is intended to be a forum for exchange and discussion of lab and plant safety and accident information without the fanfare of a news article.