Winners announced for Peer Review Challenge

Top entries focus on recognizing reviewers and making the process more efficient

By Clare Lehane Posted on 1 November 2012

Share story:

The Author

Dr. Clare Lehane graduated from University College Cork in Ireland with a PhD in marine ecology in 2004 and has worked in various aspects of publishing since then. She joined Elsevier in 2006 and is a publisher on the Energy and Planetary Sciences portfolio, where she has responsibility for 11 journals in the areas of nuclear energy, solar materials, bioenergy and greenhouse gas, along with three planetary sciences journals.

A version of this article was published in the October 2012 issue of Editors Update.[divider]

Since the 1660s, peer review has proved essential for judging the difference between science and speculation.

Over the years, however, pressure on the system has continued to grow as the global academic community expands and manuscript submissions rise.

It can be tough for editors to not only find appropriate reviewers willing to review for their journals but to motivate them to keep reviewing time and time again. At Elsevier, we are keen to work with the research community to explore ways to ease that pressure, and we are currently working on a number of peer-review pilots and innovations. One avenue we recently explored was a global competition seeking researchers’ thoughts on the future of peer review. This was inspired by the success of similar competitions such as the Elsevier Grand Challenge and the Executable Paper Grand Challenge.

In an issue of Reviewers’ Update earlier this year, we threw down the gauntlet to readers, asking them to submit ideas that could improve the current peer-review system. We also invited entries that explored how publishers and editors can help early career researchers become reviewers, or how reviewers can be recognized by either their institutes or publishers.

More than 800 readers took up the challenge, and entries embraced all aspects of peer review, from improvements to the actual “mechanics” of the process to how to reward reviewers. It was interesting to note that a great many entries detailed suggestions for rewarding reviewers in ways that can be noted on CVs. In addition, there was an intriguing dichotomy between entries that were firmly in favor of the extension of double blind type peer-review systems and those who advocated completely open peer review. The judges chose a list of finalists, whose ideas were published on the Peer Review Challenge website with an invitation to the academic community to post comments.

More than 300 people responded, and those comments were taken into account by the judges when making their final decision. Many teleconference calls and emails later, the final three winners have been chosen and their winning entries can be found below.

Philippe Terheggen As a judge,Philippe Terheggen, Senior VP of Physical Sciences II at Elsevier, said he was was impressed by the quality of the entries. “We were really pleased by the number of entries, as well as the thought and initiative invested in them,” he said. “The next stage is to work with our winners to develop their ideas. We need to determine whether they can be integrated into existing systems, like the peer review annotation idea, or develop the framework necessary to manage and deliver the reviewer points system in a logical way.

“This Challenge has shown that the scientific community is keen to publicly and systematically acknowledge reviewers’ work, as well as to utilize platforms that make reviewing easier and more transparent for all players. It is our task as publishers to work with the community …”

If you would like to get involved in any of the pilots in this article, contact Dr. Clare Lehane at c.lehane@elsevier.com.[divider]

Overall winner

Dr. Gosling suggested introducing a standardized way to recognize the cumulative effort of a particular reviewer. He explains: “Peer reviews are an important service to the academic community and fundamental to academic publishing. I entered the Challenge because I feel strongly that reviewers should be recognized and rewarded in some way for the time and effort they invest – often on top of a very busy schedule – in preparing article reviews for journal editors. I see the future of peer review as a positive feedback cycle whereby reviewers (especially early career researchers) – encouraged by an opportunity to enhance their reputation as a reliable and good reviewer and to receive journal and/or book discounts – prepare high-quality helpful reviews that in turn help to make the job of journal editors more straightforward. This way, authors, editors and reviewers all benefit from the peer-review process. To date, reviewers have received little benefit, let alone anything tangible. I envisage the acknowledgment of reviewers’ time and efforts through a widely recognized tangible accrediting system that would be well-known and citeable on a CV, which I have termed Elsevier Reviewer Badges.”

Runner-up

Muthukrishna’s idea also concentrates on reviewer rewards and develops the idea of a public points system for reviewers, inspired by websites such as Reddit and Slashdot: “As a researcher and a technologist, I was thrilled to hear about Elsevier’s Peer Review Challenge. Technology affords new and often better ways of approaching old problems. The peer-review process consists of established and largely successful practices that support good science and I am cognizant that caution is required when tampering with it. Nevertheless, I see the peer-review process gradually taking advantage of the advancements tested in public, peer-reviewed forums, such as Reddit, Slashdot.org, StackOverflow, Amazon, and many open source software projects. The successful processes in these domains have converged on solutions supported by psychological research, including the power of reputation, which I focused on in my entry.”

Runner-up

In contrast to the previous ideas, Dr. Hufkens’ entry suggests that the online platform facilitating peer review could be improved. He recommends creating an integrated review application that would, among other things, include an integrated PDF reader with annotation possibilities and generate a review template listing the changes to be made based upon annotations on the PDF. This would save reviewers’ time. He writes: “As a scientist, reviewing for journals is considered an integral part of your work. However, given the voluntary nature of this task, often little time is budgeted towards it. Talking to colleagues it became apparent to me that reviewing for journals is something done when one has a few minutes to spare, e.g., on a flight, train or bus to the next meeting or commuting home. I entered the competition with the idea that a lot of time could be saved by integrating some of today’s technology into a reviewing platform.

This platform would integrate reading and annotating on the original document, summarizing these ‘in line’ notes into a concise summary for the authors to read. Ideally this platform would be online and cloud-based so you could easily pick up and continue where you left of the day before. It’s easy to perceive how trends such as tablet applications could extend from this basic idea to provide even greater flexibility and ease of use for the reviewer on the go, or on the couch. As a good work-life balance is limited by available free time, I feel that providing the tools to free up extra time should be considered a major incentive for most scientists to continue reviewing for Elsevier.”

Possible application

Dr. Hufkens’ ideas chime perfectly with current plans for Evise, Elsevier’s next-generation editorial system. The roll-out of Evise will begin in the second half of 2013 and will include online viewing and annotation of submissions. Reviewers and Editors will be able to view the manuscript and add comments to it online. These comments will then be saved with the submission and can be made available to the author. Koen’s winning entry will be shared with the Evise team. — Linda Willems, Editor of Editors’ Update[/note]