What to Keep in Mind about Refereeing Work

This post is motivated by some of my concerns relating to suggestions about unblinding the referees (as discussed here). While I agree that there are problems with the current state of the peer review system, I tend to disagree strongly with some proposed resolutions.

During my last year as a graduate student at USC, I was managing editor of the Pacific Philosophical Quarterly. I’ve submitted my own work to journals, and, in addition, I’ve served as a referee for a few different journals.

I learned a number of things from being “behind the scenes” at PPQ. For instance, I learned how much of a manuscript’s progress through the review system depends on factors extrinsic to the manuscript. If there are two equally appropriate people to ask to referee a given paper, who differ on their threshold for “reject” vs. “revise and resubmit”, that could make the difference for what happens to your paper. It isn’t personal. I didn’t have a scorecard for how “easy” different referees were. It was just the luck of the draw.

I also learned that some of the biggest issues for the peer review system come from the fact that it is treated like a volunteer gig.

Why are so many journals so slow about getting decisions on papers? There are several possible bottleneck points for journals, but the two biggest, in my experience are: 1) getting potential referees to agree to review a paper, and 2) getting referees who have agreed to review a paper to meet deadlines.

While getting bad comments is frustrating, and can often make a rejection feel particularly unfair, I honestly think that such issues would be vastly less important if all manuscripts were issued decisions promptly (sometimes I joke that it would be fine for most journals to simply provide an efficient distribution of injustice).

So, why do we get such bottlenecks in securing referees and in getting feedback from referees? Here, I think the answer is blindingly obvious: It is hard to get referees, and hard to get them to prioritize refereeing work, because the profession does not treat such work as valuable. Combine this with the fact that we all have more work to do than we have time to do it, and that refereeing a paper can often be more tedious than it is exciting, and it should not be a mystery why it is hard to get quality refereeing done in a timely manner.

You know how I can tell the profession doesn’t treat refereeing work as valuable? There are no real incentives to do such work. Neither do we entice people to do it with rewards, nor do we punish people for failure to do it. Since one can only referee as much as one is asked to referee, I’d be inclined to favor positive incentives for people who do referee, rather than penalties for people who don’t, but the basic point is just this: If refereeing work really is an invaluable service to the profession, why aren’t we putting any professional value on its performance?

I’d be curious to know whether any university treats this sort of service to the profession as a serious factor in tenure or promotion. Where by “serious” I mean something more than listing it in the official description of tenure factors. What I mean is: Does it genuinely count in your favor for T&P to be a good professional citizen? Does it genuinely count against you if you are not?

Maybe we can’t unilaterally change the way university administrations value things for Tenure and Promotion. I don’t want to throw in the towel on that yet, but it isn’t the only option.

Journals (as an extension of the Publishing Houses) are also dropping the ball on this. What does someone get for putting in the time and effort to do good review work? Basically: pride in a job well done. I know that some publishers compensate people who referee book manuscripts. This seems like good practice, since refereeing book manuscripts is at least an order of magnitude more of a burden than refereeing a journal article. But, and this is the important thing: refereeing journal articles is still a burden!

Why on earth are we running things in a way where journal editors are put in the position of sending messages that might as well say this:

Dear Very Busy Academic,

I have a bit of time-consuming work that I need someone to do, and I think you have the right knowledge and skill-set to perform it. While I know we are all very busy with teaching, researching, mentoring students (and if we have any time or energy left over, perhaps some of it should be reserved for a life outside of work), I am hopeful that you recognize how the peer review journal process is held together with duct-tape and dreams, and will voluntarily pitch in to help out. In thanks when you finish, I will even send you a form e-mail that says “thanks” (what more gratitude could you hope to receive?)

If you don’t have the time, can you recommend a few other people who are more easily motivated to provide valuable services for no compensation?

Best,

Editor

PS: If you do a good job, you will no doubt become the first person I think of for any similar requests in the future. Yay!

Share this:

Like this:

LikeLoading...

Related

This entry was posted on Thursday, February 23rd, 2012 at 9:36 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.

Post navigation

11 Responses to What to Keep in Mind about Refereeing Work

What if journals kept statistics on (1) how long a referee takes to produce comments, and (2) the quality of the comments (rated on a scale of 1-10 by the editor), and these statistics were provided to tenure committees? Simply knowing the tenure committee was looking at it might make a difference in attitudes, even if there wasn’t an official policy change.

Lewis, this is a great post. I’m curious to know how else the profession might reward referees for doing good and timely work. (I’m also curious to know how other disciplines handle these issues.)

I can see referee work counting toward T&P in a department that both houses a journal and draws on its faculty for purposes of refereeing. But how do we get departments generally to care about this type of contribution to the profession–and care enough about it for this kind of work to count positively toward T&P?

What do you make of the suggestion, which seems to come up in these types of discussions, that journals should pay its referees?

I don’t know that it is true that journals “should” pay referees, but that is certainly an avenue worth exploring.

One positive incentive I’d been thinking about was some way for a person who refereed well for a journal to get priority in the journal’s submission queue. I am not sure if this would cause other problems, but the basic idea would be that if you are a good referee, your submission gets to jump the line so that you can expect shorter time-to-decision.

I think journals might want to experiment with a variety of ways of rewarding referees, and see which ways are most successful. They might have to get inventive given some financial constraints.

I want to be clear: I don’t think I know what the detailed solution is. I only know that, if you want people to put time and energy into refereeing, and you want them to treat it like an important task, the profession needs to overall act like it is approximately as valuable of a service as it really is.

The only thing that I want to add to what I just said about this is: if paying referees would require authors to pay money to submit to the journal, then I would be wary of the suggestion, given that it is not a good idea to be increasing financial burdens on the faculty with the most pressing needs to publish. It might wind up being the best strategy; but we’d want to think carefully before increasing the burden on them.

Let’s make this issue concrete. Suppose I get 200 submissions a year. That’s 400 referees each year. Let’s say I give each referee $5 on Amazon.com. That’s $2,000 each year! Certainly the for-profit publishers could afford that, though the big name journals probably get more than 200 submissions each year.

Failing that, what can I do as an editor? I could send a handwritten thank you note. But while that’s thoughtful, and no doubt appreciated by some, it’s not going to address the issue of motivation.

It seems only systemic changes in valuation by the profession would help.

I actually think some sort of queue jumping arrangement might be sufficient incentive. Not sure if it would work well in practice, and it might need to be organized across a set of journals, rather than getting a coupon that only works for the specific journal you refereed for. But, imagine if being a dilligent referee who turned in a thorough report in a timely fashion meant your next submission would jump to the front of the queue at a good journal. Given that review times are what they are, that might be a pretty strong incentive. You might even get people volunteering to referee when they are between projects, e-mailing a journal editor to say, “please keep me in mind if you get any papers in area X”.

That is part of why I was suggesting it might need to be something that a collection of journals jointly offer. There are obviously other incentives one could theoretically offer, but many of them would require you to undermine the evaluation process (e.g. anything that increased the likelihood of publication, rather than simply making it a shorter process would be problematic).

At any rate, I think that financial incentive is going to be problematic anyways, since it would probably be impossible to offer large enough financial incentives to really impact people’s willingness to referee. As you noted, even a modest incentive would be extremely expensive at the scale we are talking about.

I’ll keep thinking trying to think of other options, though. In the end, I think someone is going to have to be pretty inventive and clever to figure out a good solution to this problem, but it is a problem worthy of attention.

The difficulty is that universities don’t have much incentive to give weight to refereeing in evaluation and T&P decisions because the referee work in anonymous and does not improve the reputation of the university.