Refereeing behavior and the determinants of altruism

The results of an innovative study carried out by the editors of the Journal of Public Economics

In 2010, the editors of the Journal of Public Economics conducted an innovative experiment, with approximately 1,000 referees, into how we can motivate pro-social behavior.

Study overview

We chose to collaborate with Elsevier on a study into how we can motivate pro-social behavior. Timely and useful refereeing work is usually understood to be a favor to colleagues, an unpaid but important task for the profession. As such, it is a good candidate for comparing financial incentives with other more inexpensive alternatives, in situations where the work benefits the public more than it rewards the worker.

The study has highlighted some key factors:

Deadlines are extremely effective

Moderate financial incentives invoke a large response

Public comparison to peers has a moderate effect

There are different effects when looking at younger referees in comparison to established tenured ones

There is no backlash when financial rewards are missing or phased out.

Conducting the experiment

In 2010, we randomly assigned approximately 1,000 referees to one of four groups:

A control group with the usual 6-week due date

A short deadline group with a 4-week due date

A cash incentive group where a $100 payment was made for reports submitted by the shorter deadline

A social incentive group where referees' turnaround times were publicly posted on the journal's website.

Always following the same process the majority of referees received multiple invitations throughout the study, which ended in November 2011. Using data retrieved from the Elsevier Editorial System (EES), the team studied refereeing times both before and after the experiment, as well as comparing times to other Elsevier journals. The differing approaches had small but significant impacts on whether referees agree to review a paper, but do not appear to have generated differential selection. The shorter deadline, of a 4-week due date, reduced turnaround times by an average of 10 days. The cash offer doubled this effect, further reducing referee times by an additional 10 days and the social incentive treatment reduced turnaround times by approximately 5 days. Less surprising in hindsight is that the study showed that tenured professors are most responsive to social pressure, while untenured referees are most responsive to deadlines and cash offers.

The study also showed that a common misgiving about financial incentives -- that they may crowd out intrinsic (or altruistic) motivation -- does not appear to apply here. Referees are no slower at concurrent or later unpaid jobs. Finally, all the approaches show modest impacts on the quality (length) of reports, the recommendations of the referees and/or the final decision of the editors.

Presenting the study

This study was presented at the National Tax Association Annual Conference in November 2011 and will be part of a session on journals and academic publishing at the Summer Institute of the National Bureau of Economic Research in July 2012. The session will be attended by editors of leading journals, who may decide to re-evaluate their own journal's policies based on the evidence from this innovative experiment conducted by the research team, along with Elsevier.

About the authors

Raj Chetty is Professor of Economics and founder of the Lab for Economic Applications and Policy at Harvard University. He received his PhD in Economics from Harvard in 2003. He is the co-director of the Public Economics program of the National Bureau of Economic Research.

Emmanuel Saez is the E. Morris Cox Professor of Economics and Director of the Center for Equitable Growth at the University of California Berkeley. He received his PhD in Economics from MIT in 1999. He was awarded the John Bates Clark medal of the American Economic Association in 2009 and a MacArthur Fellowship in 2010.