A Modest Proposal to Improve the Peer Review Process

The following is a guest post by political scientist Scott Gehlbach (@sgehlbach) of the University of Wisconsin–Madison.

*****

The peer-review process, if not broken, is seriously under strain. Editors are forced to make hasty decisions based on imperfect signals from referees. Referees, in turn, are overburdened with review requests. And authors are at the mercy of referees who are not always qualified to evaluate all parts of a submission.

These three problems have a common cause: as a discipline we are asking referees to do too much. The typical review request takes the form: “Please evaluate this submission as a possible contribution to…” The referee process could be improved by adding a sentence that says, “As an expert in X, your thoughts on Y would be especially valuable,” where Y could be research design, a formal model, country context, framing, or any other element of the paper. In principle, the request could specify Y1, Y2, etc., though the point would be for the list to be less than exhaustive.

As an analogy, think of dissertation-committee members, who typically concentrate on parts of the dissertation where they have particular expertise. We don’t expect the Africanist to offer extensive comments on the model, or the formal theorist to advise on the ethnography, unless these specialists happen to be one and the same person.

Focusing reviewer effort in this way will allow editors to make better decisions, as it will be easier to extract the signal from the noise in referee reports. It will take a bit more work from editors at the front end of the review process, but I expect that it will save time at the back end. And I’m not convinced that it will take that much more time ex ante: editors already choose referees based on expertise (so, in the formulation above, they know X), and the increased use of desk rejection means that editors give manuscripts at least a cursory read on initial submission (so it shouldn’t be too difficult to fill in Y).

There are also benefits to other stakeholders. Referees will appreciate the sanction to direct effort to parts of a manuscript where they have the most expertise. And to the extent that all of this produces better and speedier decisions and more focused referee reports, authors (and ultimately readers) should profit.

A final note: There is a decentralized version of this reform that can provide many of the same benefits. Even if editors choose not to suggest that referees focus on particular elements of a submission, reviewers can still choose to restrict their comments in a way that reflects their substantive or methodological expertise. Indeed, I suspect that some referees do this already, but the key is to make it explicit: “In reviewing this manuscript, I primarily restrict my attention to Y.” Such a statement clarifies to editors and authors what the referee has, and has not, taken responsibility for.

I understand that there might be strategic considerations at play here. I leave those as an exercise for the comments section.

I’d say this happens fairly organically anyway. Most reviewers will spend more time on areas of a paper where they feel qualified to say something. Certainly when I review a paper I focus on the areas where I have expertise rather than trying to say something superficial about areas I’m not an expert on. I assume that the editor has chosen me because I can say something useful about the parts of the paper I have expertise in and has presumably selected other reviewers who have more expertise on other parts.

I also worry that this approach might stop the overlapping criticisms heuristic from working. If two reviewers make the same critique (even better if it is from two angles) then it helps to signal to the editor that the problem is urgent.

I enthusiastically endorse your suggestion that referees explicitly confine their comments to what they know about. I have always felt uncomfortable – and therefore occasionally defensive – about having to pass comment on the entirety of a paper when only a fraction was within my comfort zone. This will force editors to make calls on what we all have to live with: partial information.

I’ve been practicing this for years. A large percentage of my reviews say something like, “I assume the editors have other reviewers more competent in evaluating ____”.

And I’ve had some editors emphasize certain reviewer points by saying “Reviewer 1’s opinion on this matter carries especially strong weight”, which is particularly useful where reviewers contradict one another but one of them might just happen to be recognized as more expert in the given matter.

As an editor, I do this routinely. Sometimes it goes unsaid, but it is obvious that, when I sent an article full of fancy statistics to a philosopher for review (which sometimes happen), I don’t expect comments on the statistics. (Sometimes I’m surprised.)

When I was an editor, we would often do this when we wrote to referees – especially ones we knew were particularly busy and/or overcommitted. This was one benefit to not having computer generated bumpf, but actually taking the time to write an email (even if most of it was cut and paste and/or sent by an editorial assistant).

About

The mission of this blog is described in our inaugural post. And, technically, an orangutan is an ape, not a monkey.