About this Author

College chemistry, 1983

The 2002 Model

After 10 years of blogging. . .

Derek Lowe, an Arkansan by birth, got his BA from Hendrix College and his PhD in organic chemistry from Duke before spending time in Germany on a Humboldt Fellowship on his post-doc. He's worked for several major pharmaceutical companies since 1989 on drug discovery projects against schizophrenia, Alzheimer's, diabetes, osteoporosis and other diseases.
To contact Derek email him directly: derekb.lowe@gmail.com
Twitter: Dereklowe

December 20, 2010

Putting Some Numbers on Peer Review

Posted by Derek

Since we've been talking about peer review on and off around here, this paper in PLoS One is timely. The authors are putting some numbers on a problem that journal editors have long had to deal with: widely varying reviews from different referees for the same exact paper.

It's a meta-analysis of 52 studies of the problem reported over the last few decades. It confirms that yes, inter-reviewer reliability is low. The studies that report otherwise turn out to have smaller sample sizes and other signs of lower reliability. The question now is: to what extent is this a problem?

One of the studies they quote maintains that too high a level of agreement would also be the sign of a problem (that some of the reviewers are redundant, and that the pool of referees might have been poorly chosen). I'm willing to think that total agreement is probably not a good thing, and that total disagreement is also trouble. So what level of gentlemanly disagreement is optimal? And are most journals above it or below?

FIguring that out won't be easy. Some journals would really have to open their books for a detailed look at all the comments that come in. I assume that there are editors who look over their reviewers, looking for those that tend to be outliers in the process. (Um, there are some editors that do this, right?) But that takes us back to the same question - do you value those people for the perspective they provide, or do you wonder if they're just flakes? Without a close reading of what everyone had to say about the crop of submissions, it's hard to say. Actually, it might not be easy, even then. . .

I believe the reviews should be published alongside the paper. It would provide transparency into what issues were raised (and what were not) by the reviewers and also protects the authors from political dirty tricks.

We don't know what the basis of disagreement is on any review. Even significant disagreement might actually be good. It depends what it is.
For example, if reviewers differ very widely in what problems are identified you could argue that they were the wrong reviewers or that they didn't give proper diligence.

If on the other hand all raise the same issues but they differ widely in personal interpretations then we have a very different story.

It seems that we have several different cases, most assuming editors are overloaded:

1. Reviewers were capable, but had no time and were cursory in their review, especially when reviewing a well known lab. I sort of hope the nevirapine optical activity story lies here.

2. Reviewers were thoughtful, but not from a relevant experience set. The arsenate/phosphate case might fit here.

3. Reviewers were used because someone said you have to, not because they want to actually check anything. I wonder if some of the retracted high profile papers in places like Science / Nature fit in here? Or, would that be #1?

4. Reviewers were incompetent and editors had no time to really check

5. Reviewers were incompetent, editors were absent and it is basically a vanity press revenue stream for the publisher.

Unfortunately there are too many of #5. And as long as they count toward promotion and tenure, or funding, the system will only get worse.

You could argue that #1-#4 are not much better.

I have a suggestion though. What about a primary / secondary publication model for each journal, not A-list journals vs the rest? Publish one article but divide it into the single most important discovery for the primary part and everything else goes into secondary material. In essence, push much more into supplementary information.

If there is a follow on article it would simply get tacked onto the original unless there was a new main point at least as important as the first.

It still needs review but the effort could get pushed into the main point first, put that out and follow up with review of the other points later. The 'secondary' review could even be done in a social network context, including deciding to split follow up papers out.

I echo here part of the sentiment of posting reviews. Unfortunately those ivory towers are full of sharp horns...

On some aspects of a submission, it isn't a question of too much agreement or too much disagreement, it's a question of getting it right.

There should be 100% agreement on most matters of data, it's analysis, interpretation and presentation with relevant precedents. How many of you have had to point out that the "new peak" in the NMR was not proof of a breakthrough discovery but residual solvent?

Whether an article, once vetted, is suited to a particular venue is more subjective.

Which reminds me of so many "fun" journals in the past history of chemistry. Chimia used to allow occasional chemical "joke" articles ... on purpose, that is. I think Nakanishi used to joke with, I think Nozoe, about publishing their mss's in the Journal of Nozoe Chemistry. (I think it was Nozoe.)
Journal of

I like the idea of publishing the reviewers comments along with the article itself. I would not want open comments - we already have blogs for that. But it would be interesting to see what each reviewer (anonymously of course) thought about the manuscript; as well as the authors rebuttal.

I remember reviewing an article for BMC and gave it a "major revision" label only to have an email not 2 days later saying it was accepted with no revisions. I emailed the editor and was told the author and he spoke about the article and that the editor agreed with the author. Times like this it would be interesting to see the reviewer comments.

What if we have a more social publishing forum, e.g. reddit model? Yes, there may be some bad work published, but that's what the group moderation is for... keeping each other on our toes. We'd have to learn to be polite and provide constructive criticism.
Funding to host the articles (incl. maintenance) might be an issue; wonder if advertising from suppliers on the page might generate enough revenue?

I would ague not only for open reviewing and posting of reviews, but for a comment thread on publication. However, I would want commenters to be vetted. If users are verified as being genuine members of an industrial firm, little startup, academic institution, or govt lab, and they post under their own names, you will get far less "Your MOM! Oh, YEAH?!? YOUR Mom!" commentary, and a lot more of the sorts of "see how smart I am" useful suggestions like "have you considered this experiment?". Now, it won't ensure perfect civility (even at meetings, where everyone knows who you are, some people act like jerks), but it helps. Sure, it's added work to have people register in such a way that they can be verified, but not that much work. Our community of inorganic chemistry educators verifies "faculty users" for certain privliges on the site (non-faculty members can do most things on the site, but things like solution keys are faculty only), and it's doable, especially for an institution with a subscription system already set up.

The "open" review process breaks down very quickly, when one is forced to write a comment such as this to the editors...

"While this paper is interesting, the authors have been banging away at this same experimental model for 20 years, and my gut reaction is "boring". Yes, it's technically correct. No, there's nothing wrong with the science. It's just more of the same from the same group and it's an easy paper for them. They're capable of more than this same old drivel".

Such a comment might be misconstrued by an author as "politics" being the reason their paper got rejected, when in fact all of the above is a perfectly legitimate reason to reject a paper. I don't think I'd be happy reading such a comment, either as an author or a casual viewer.

Whatever you say about the current system, it works 99.9% of the time. Now if we could just deal with the other 0.1% that slip through the cracks....

I have given anonymous submission some thought. A paper by "big name in chemistry" will be accepted much easier, because of the trustworthiness of his name. Although we all look at the origin of the paper while we referee it to asses its value, this knife cuts two ways. Often, a "big name" will publish similar results in a higher level journal than, lets say, a start-up lab from India.
Obviously, the name and origin of the authors can have a large effect on peer-review, both positive as negative. I have noticed that more and more journals are beginning to ask for "not preferred as referees", so that authors can avoid competition issues.

I have no clear-cut answer for this, but still, its hard to judge only the pure scientific results, when we all look at the names in the manuscript.

One thing should be noted regarding reviewing.
Many professor do referee work for more than 10 journals. That means 1-2 paper a day. That is a lot of work besdide your normal work.
I do not have suggestion about it but I think it is important to include it in the discussion.

Published reviewer comments.

I think reviewer comments should be pubblished:
- I would not publish the comments on novelty and importance of the article. We could have a non-disclosed section of referee repost.
Also things like "poorly written", "sloppy work" and "why do you send me this b..." could go in that section.
- Only "scientific comments" could be published. Key suggestions, missing experimental data, motivated concerns on mechanisms etc.

I also vote for keeping anonymous the reviewer:
- many professor love to read themselves and would add 50 comments to show how good they are and/or avoid their review performances to be later questionated by the readers.
- at the same time, knowing the name of the person rejecting your papers, on the long run, might start some really nasty "accademic secret war".

@Vrgl "While this paper is interesting, the authors have been banging away at this same experimental model for 20 years, and my gut reaction is "boring". Yes, it's technically correct. No, there's nothing wrong with the science. It's just more of the same from the same group and it's an easy paper for them. They're capable of more than this same old drivel".

This is NOT a legitimate reason to reject a paper. The science needs to stand alone, regardless of who the authors are. If the same work would be publishable if from a group who is new to the field, it should be equivalently publishable from a senior group that is cranking the wheel. The fundamental question is whether the science itself is worthy of publication, not whether it was easy or hard for any particular group. This is just the kind of comment/opinion that an open review process would ferret out in a useful way.

I don't know that you could necessarily apply this model to reviewers for a journal, but when you propose for telescope time, the proposal comes back with, in addition to a yes/no, anonymized reviewer comments. These comments are usually just a few sentences about the proposal ("Valuable result if detected" or "Unclear how the amount of time to request was chosen" or whatever), but they are accompanied by a numerical rating and, interestingly, a mean and standard deviation for all that reviewer's ratings. So if a reviewer rates everything one out of five except for a few good ones that get two, you can tell. And as I understand it the TAC normalizes all the scores before combining them to make their decision. So to some extent reviewers' habits are controlled for.

#5: There is Category 6: reviewer is defending their own field of work by trashing your paper (and similar intra-field politics) I stumbled into that by accident, when I didn't realize that a lab was trying to corner the market on a particular line of research. Both prof and grad student "independently" reviewed the work, and both trashed it.

As for reviews, I'm not fond of publishing them. Some of the best reviews I've received (or given) for that matter, were ones that improved the paper. Publishing a list of paragraphs clarified and typos caught is not useful. Nor is publishing a "good paper, publish it" style review.

@15 revisited
I should have added that I agree review comments should only be published anonymously, to avoid *academic wars* (#17; usually not so secret!)
And yes, only scientific appraisal, not technicalities. Reviews would probably need to be drastically edited to eliminate the casual put-downs, etc. mentioned in the comments above.
Perhaps the review form could be designed to have a 'scientific appraisal overview' section?
I would be happy for most my comments as a reviewer (pre-electronic; revealing my age here!) to have been made *available*.