Time to jump into the arXiv?

As part of the Miller Institute here at Berkeley, I’ve had the wonderful opportunity to meet weekly with scientists from across disciplines (including chemistry, physics, astronomy, biology, and mathematics). One topic that comes up a lot at our discussions is how to address the challenges, while maintaining the benefits, of peer-review.

In science we design experiments, test hypotheses, draw conclusions, write up manuscripts interpreting the results, and then submit our research findings for peer-review. This process takes different forms in different disciplines. In Biology, generally, peer-review takes the form of submitting a manuscript to a journal, the editors take a cursory read through the research and decide if it fits the scope of the journal. If it does, the editors seek out experts in the field of study and request that these researchers read the article and write a summary of their professional opinions of the strengths and weaknesses of the article, specifically assessing whether the methods and analyses are sound, and should be published, or whether more/different work should be done. These reviewers are, typically, anonymous, meaning that the authors will not know who has submitted suggestions regarding the submitted research. Often it takes a couple rounds of comments from the reviewers/editors, revisions and resubmissions, and then – yay! – the paper is published. The time from submission to publication can take as little as a few weeks, or upwards of 12 months.

In my opinion, the purpose of peer review is to strengthen Scientific research, specifically identifying and addressing concerns about methodology and interpretations that might lead to faulty conclusions. Peer-review can also catch cases of academic dishonesty (including plagiarism, and falsifying data). It isn’t always perfect (see examples of falsifying caught after publication, here, here, here, and here), but generally peer-review seems to be a good tool for facilitating the self-correcting nature of Science.

A benefit often touted of anonymous peer-review is that reviewers (especially new scientists) need not worry about being retaliated against if they bring up concerns regarding a manuscript. A major concern about the traditional peer-review, is that the time between submission and publication can drag out the sharing of research-findings, and hinder scientific advances.

It wasn’t until I spoke with scientists from other fields, specifically astrophysics and astronomy, that I learned about a completely different way to achieve peer-review, using a pre-print server called the arXiv (arxiv.org).

I haven’t used the arXiv, but my understanding of it, as explained by my colleagues, is that researchers upload their draft manuscripts to the arXiv as soon as the paper is in a complete or near-complete form (for example, at the same time as I would submit a manuscript for peer-review to a journal). Others might even submit abstracts or preliminary data, to stake a claim on a particular research topic. The arXiv is then published daily, completely open-access, so anyone can immediately review the most-recent research, add their own comments to the site, and the authors can upload revised versions, as they choose to address comments from the community. While some authors choose to also submit to peer-reviewed journals, it is not required of authors in all fields; submitting to the arXiv is sufficient (for academic success) in some disciplines.

In addition to getting scientific research out immediately, another benefit of the arXiv are that the reviewers are not anonymous, so discussions can ensue about particular concerns. However, because there is no filter as to what can be uploaded to the arXiv, it is entirely possible for manuscripts with no scientific merit to get wide-distribution and, perhaps to those not familiar with the format of the arXiv, the facade of scientific approval.

There is one other drawback to the arXiv that makes me, as a potential submitter, very nervous: being scooped.

A paper is “scooped” if someone else publishes the same (or very similar) concept before you get a chance to publish yours. But, wait, if it is on the arXiv, isn’t that documentation that I had the idea first? Well, yes, but… the arXiv isn’t commonly used in Biology yet, so it isn’t clear how important or how much priority will be given to authors who publish there before “traditional” peer review. This is especially concerning if the novelty of the paper is the idea (which is easy to reproduce with the same or different data) versus a method (which is more difficult to replicate). Maybe this isn’t a valid concern, because anonymous reviewers could, one might argue, just as easily “scoop” ideas from a manuscript they have reviewed. Furthermore, perhaps posting ideas/research early might facilitate more collaborations instead of competitions between research groups.

All said, I think that submitting to pre-print servers can be a very valuable tool for facilitating scientific discourse and advances. Will I start submitting there? We will have to wait and see.

11 Responses to “Time to jump into the arXiv?”

Nice post. I wrote about some of the benefits of the arXiv here: http://gcbias.org/2012/07/22/arxiving-our-papers/ (there’s also links there to others). My view is that you are far more likely to be scooped due to being hung up in review for a year, than by posting on the Arxiv. Having your article on a preprint server gives others a chance to cite it.

Graham, it’s nice to see all the positive things about arXiv. I especially liked the link to journal policies regarding archiving. There seems to be a lot of forward momentum in Quantitative Biology, so perhaps arXiv.org will open up new sections.

Posting to arXiv is one thing – but reading papers on the arXiv another. It is a bit like reading a blog – perhaps the writing has some merits – and perhaps it is completely wrong. In the Mathematical sciences, that works OK, because the difference between a correct equation and a wrong equation usually is pretty clear cut. However, in the biological sciences, if peer review does not precede publication, there is really not the same strong encouragement for researchers to make a careful experiment that thoroughly eliminates alternative explanations. If you are first with an experiment or a hypotheses – you are first – and that is what will be remembered. Science publication through arXiv does not provide the same checks and balances that the review process is supposed to provide when it works well. With demands for teaching, writing grant applications, doing outreach and helping with administration – in addition to increased demands for scientific production – 24 hours in a day is just not enough for modern researchers. I don’t have time to read papers that could be completely without any merit. There are enough good and solid peer reviewed papers that I don’t find the time to read. So the last thing I want to spend my time on is reading papers on arXiv. If the research is solid, it should be able to pass peer review and will soon enough be out in a peer reviewed journal.

You write, “I don’t have time to read papers that could be completely without any merit.”

Many published, peer-reviewed papers are either uninteresting or wrong. I generally filter these out with a quick look at the title, authors, and abstract. I’m sure you do the same. The average quality of popgen on arXiv is low simply because very few serious people post there. If you were to post your papers there, I’m sure many would read them (I also hope it’s safe to assume you would be as rigorous in a preprint as you would be in a “real” paper!).

My main worry is really the long term implications of moving towards publishing without peer review. I agree that there are also problems with the peer review process, but I would rather see that process fixed than moving towards a model in which absolutely anything can published – whether it has any merit or not. I think that would be damaging to the entire scientific process. It would exacerbate some of the problems I see with the current scientific process in which focus often is on the most flashy results rather than on the most rigorous research, and in which there are way too many papers published. It would also lead to an increased amount of opportunity for climate change deniers and creationists to pretend to be involved in the scientific process. So I see some dangers here, that perhaps are bigger for biological research than for mathematically oriented research.

I am not sure how this will develop – but in the meanwhile I will out of laziness focus my time on reading the papers that I know at least a couple of other researchers have gone through before me to see if they could catch any errors.

Self-archiving provides another chance of exchanging ideal, in addition to traditional scholarly publishing. In context of paperless publishing, the first wave was concerned with “Open Access” and equal opportunities. And now, second wave seems to be concerned with free publishing rather than open accessing. Actually, there were worries about early stage of Open-access journals, including PLoS. For instance, at the beginning years of PLoS ONE, we just finished a manuscript on a small project. And, I told my advisor to submit the manuscript to PLoS ONE. He disapproved, and, said: they accepts all the manuscripts they receive. Now, the journal demands high quality research papers based on scientific method, including high degree of animal ethics. After PLoS series lunched as an open access journal successfully, subscription-based journals provided under open access conditions on the internet. And, the next wave is, as the blogger wrote, to come. There is a famous episodes involving this. One might remember Gregory Perelman who uploaded his work on the proof of Poincaré conjecture to arXiv. This episodes gives us a glimpse of a possibility for self-archiving that it can be a personal-publishing as long as one having a high degree of scientific standard. Because, the second wave is concerned with free publishing rather than open accessing….