The Rogue Neuroscientist on a Mission to Hack Peer Review

Share

The Rogue Neuroscientist on a Mission to Hack Peer Review

Dominic Trevett/Getty Images

Sam Nastase was taking a break from his lab work to peruse Twitter when he saw a tweet about his manuscript. A PhD in cognitive neuroscience at Dartmouth College, Nastase had sent his research out for review at a journal, and hadn’t yet heard back from the scientists who would read the paper and—normally—provide anonymous comments. But here, in this tweet, was a link to a review of his paper.

“I was like, ‘Oh that's my paper, OK.’ So that was a little bit nerve-wracking,” says Nastase. A few weeks later, he received the same review as part of a response from the journal, “copied and pasted, basically."

So much for secret, anonymous peer review. The tweet linked to the blog of a neuroscientist named Niko Kriegeskorte, a cognitive neuroscientist at the Medical Research Council in the UK who, since December 2015, has performed all of his peer review openly. That means he publishes his reviews as he finishes them on his personal blog—sharing on Twitter and Facebook, too—before a paper is even accepted.

Scientists traditionally keep reviews of their papers to themselves. The reviewers are anonymous, and publishers protect their reviewers’ identities fastidiously, all in the name of honest, uncensored appraisal of scientific work. But for many, the negatives of this system have started to outweigh the positives. So scientists like Kriegeskorte, and even the journals themselves, are starting to experiment.

Kriegeskorte's posting policy has made a lot of people uncomfortable. He's faced resistance from journal staff, scientific editors, and even one scientist who anonymously reviewed a paper that he reviewed openly. “People in the publishing business, my feeling is that they feel that this is deeply illicit,” Kriegeskorte says, “but they don't know exactly which rule it breaks.” Still, after more than a year of this experiment with exclusively writing reviews on his blog—he’s done 12 now—Kriegeskorte says he’ll never write a secret review again.

Wide Open

Kriegeskorte had been struggling with the publication process since he was a post-doc. The long, drawn-out process of getting a paper published, only to have it locked behind a paywall, seemed criminal. Instead of limiting the dissemination of scientific knowledge—a holdover from the pre-web days of science—he envisioned a system that provides perfectly open access and evaluation. “Once the plan had formed in my mind, it became irresistible to try it out,” Kriegeskorte says.

So he devised a procedure: If Kriegeskorte is invited by a journal to write a review, first he decides whether he’s interested enough to review it. If so, he checks whether there’s a preprint available—basically a final draft of the manuscript posted publicly online on one of several preprint servers like arxiv and biorxiv. This is crucial. Writing about a manuscript that he’s received in confidence from a journal editor would break confidentiality—talking about a paper before the authors are ready. If there’s a preprint, great. He reviews the paper, posts to his blog, and also sends the review to the journal editor.

If there isn’t a preprint, things get a little unorthodox. Kriegeskorte emails the authors of the paper and tells them that he’s been invited to review it. This act, of identifying himself to the authors before he’s even reviewed the paper, is normally a no-no. In at least one case, that by itself got him thrown out as a reviewer. He contacts the authors to ask them if they would post their paper on biorxiv or another preprint server, and tells them that he’ll only review the paper if they post it.

So far, almost all of the scientists he’s asked have complied.

Nastase is one of those scientists. He actually wasn’t caught completely off-guard when he saw Kriegeskorte’s review on Twitter; earlier that year, Kriegeskorte had approached him when his lab submitted its work to a different journal. Because there was no preprint, Kriegeskorte asked them if they would be willing to post one. While Nastase's lab supports open science ideas—like sharing code on their website—they had never posted to a preprint server. “This was a good kick in the butt to actually do it." Now he regularly posts his papers as preprints.

The process has some surprising benefits for scientists. Chris Baldassano, a neuroscience post-doc at Princeton, saw an uptick in his Twitter followers when Kriegeskorte posted a review of his paper. The review was helpful, and overall positive. But, he says, “if it had been more critical, I would have been more uneasy about it.” An open review that questions the importance of a young scientist’s research focus could affect their career.

At least one paper that Kriegeskorte has reviewed—by Andrew Haun, a neuroscientist at University of Wisconsin-Madison—was rejected by the journal. Haun has a thick skin about it: One upside, he says, is that he'll be able to write a reply to Kriegeskorte—something that normally doesn’t happen when a paper is rejected. Other recipients might not have such a cheery outlook; Kriegeskorte thinks he’s probably upset some authors he's reviewed. But to him, those are just the growing pains for a new culture of unshackled information.

The Edward Snowden of Peer Review

The creep toward open peer review is well on its way. This month, the European Journal of Neuroscience announced that it would start attaching full, bylined reviews to papers. Open reviews, the argument goes, are more thorough and constructive—and the rich scientific debates they reveal can be a valuable educational tool. “We believe this is the future," the editors-in-chief wrote in an editorial about the decision.

The journal did include a mechanism for negative reviews to remain anonymous, “to protect the research community from potential harassment or reprisal.” And Kriegeskorte agrees that having a mechanism for occasional anonymity is important to protect scientists who need to be critical. “However," he says, "I think that it will be important to stand by our judgments in general."

Several other journals have taken steps to support that philosophy, informally allow reviewers to open up by simply signing their reviews. Eve Marder, a neurobiologist at Brandeis University and a deputy editor at eLife, says that around one third of reviewers under her purview sign their reviews. But even when they don't, eLife reviewers have to talk to one another and synthesize their reviews, so at least one or two colleagues know a reviewer's identity—a little social pressure that puts people on their best behavior. Regardless of whether they're signed, eLife prints the reviews along with the paper if it's published.

All of these reforms are in the name of escaping the bad habits and troll-isms that scientists experience in the peer review process. At highly selective journals like Cell, Science, and Nature, professional editors are often looking for reasons to reject a paper rather than accept it—so unscrupulous, biased reviewers can weight the scales. "It is very easy to kill any paper without ever having to bend the facts," says Kriegeskorte. "All you have to do is look at the strengths and then look at the weaknesses and then focus attention on the weaknesses." Review reforms may also rein in the requests for additional experiments that can add years to a PhD student's life, Marder says.

Beyond that, open review might ultimately be better for the reviewers, who under the old system get no credit for the hours of work they put into each review. And being involved in a group decision process, whether behind closed doors like at eLife or in a public forum, will reduce the chances that reviewers make errors. “I think it's a really complicated story about where people feel the power dynamic is. And what the danger is,” says Marder—the danger of how an author will take a reviewer’s criticisms, and of how critical a reviewer will be.

Kriegeskorte thinks that the mostly positive response to his experiment indicates that the danger is low. “If someone hacked into all these journal servers and just opened up all the reviews and all the authors, you know, the Edward Snowden of peer review, what would actually happen?" he asks. "Would this be a terrible catastrophe? Would people hate each other? Or would it be a kind of catharsis for the community?" Kriegseskorte hopes that, after everyone gets over the initial shock of knowing their reviewers, they'll get more comfortable with the devil they know.