Subscribe to receive posts by email

I agree to the data use terms outlined in Scholastica's Privacy Policy

×

4 ways to get higher quality peer review comments

Image Credit: J. Kelly Brito on Unsplash

Does the quality of the peer review comments your journal receives vary more than it should?

Well-thought-out reviewer comments aside, many editorial teams find that they too often receive comments that are:

Meandering and difficult to interpret

Sparse or lacking the level of detail needed to be constructive

Hyper-critical in incidental areas while missing the bigger picture

Such unfocused or insufficient reviewer comments can create snags in peer review and cause frustration for all parties involved. Many editorial teams struggle with wanting to ensure that they’re providing authors with robust feedback but feeling like reviewer comment quality is outside of their control. While you can’t guarantee that your journal will receive top-notch reviewer comments all of the time, there are some steps your editorial team can take to improve reviewer comment quality.

Use a peer review feedback form

One of the best steps editorial teams can take to improve the quality of reviewer comments their journal receives is to require all reviewers to complete a standardized feedback form. When reviewers are left to fashion comments from a set of instructions, no matter how thorough and well-formatted they may be, the likelihood of some reviewers misinterpreting or skimming over expectations is high. However, using a required, standard feedback form journals can ensure that all reviewers address the same key manuscript areas while deterring reviewers from giving tangential feedback. Reviewer feedback forms work best when they are automated using peer review software, so reviewers literally can’t submit comments without answering all of the necessary questions.

Find the right balance of feedback form questions and make them specific

The results of your journal’s reviewer feedback form will only be as good as its design. Once your journal has a feedback form set up, track the quality of responses submitted and make adjustments to the questions as needed to improve feedback outcomes.

The best feedback forms have a balanced number of questions that are easy to interpret. Aim to provide enough questions to adequately guide reviewers but not so many that you begin to overwhelm them. Remember, reviewer fatigue is real! Make questions specific so reviewers understand the goal of each question—this includes writing questions clearly and formatting questions to reflect the level of feedback needed. For example, journals can use open-ended questions for substantive feedback and Likert Scale questions for high-level assessments and recommendations.

Some common feedback form flaws to avoid are:

Requiring nonessential questions: As noted, it’s important to have a balanced number of feedback form questions. Avoid filling up your form with questions you don’t need.

Combining questions: Make sure that each question in your feedback form is only about one thing. When questions touch on multiple areas it’s more likely for reviewers to submit unclear or partial responses.

Not asking for a publication recommendation: While the decision to accept a manuscript, reject it, or ask for revisions is ultimately up to your editors, asking reviewers for a direct recommendation can help ensure you’re interpreting their comments correctly.

It’s a good idea to also include an open-ended response field for comments to the editor at the end of your feedback form. This will ensure that reviewers are able to comment on all aspects of the manuscript that they think are necessary, even if your form doesn’t address them all. Over time, this field can help you to keep improving your feedback form questions by revealing any important assessment areas you’ve missed.

Limit revise and resubmit rounds

In addition to taking steps to improve initial reviewer comments, journals should set clear parameters for revise and resubmit rounds to ensure subsequent comments remain helpful and on track. As explained by former managing editor of Aztlán Journal of Chicano Studies, Wendy Laura Belcher, reviewer feedback can become less focused over multiple rounds of revisions and review. Belcher said that a common problem that editors should look out for is reviewers picking apart new areas of a submission in each round of review. This can turn into a frustrating cycle for authors and reviewers who just want to fulfill their respective expectations. Journals can avoid these situations by setting clear parameters for which aspects of the manuscript reviewers should be commenting on as well as a limit to the number of rounds of revision a manuscript can go through.

Help reviewers improve

Finally, your journal can help reviewers grow and improve the reviewer comments you receive by giving reviewers insight into the quality of their feedback. Journals can either provide reviewers with direct feedback on their comments, noting what was most useful and whether any comments were unnecessary or confusing, or they can send reviewers copies of author decision letters to provide insight into how their comments were interpreted and relayed to the author. Additionally, journals working with more early-career reviewers may want to provide some training materials such as a guide to writing constructive reviewer comments.

Related Posts

In this blog series, we interview open access experts about how they're working to make research more accessible and promoting academic-led publishing. In this post we welcome Jesse P. Karlsberg, the Senior Digital Scholarship Strategist at Emory Center for Digital Scholarship.

If you want your journal to reach the broadest possible readership, it's imperative for you to prioritize search engine optimization or SEO. In this post, we overview 4 often overlooked areas of SEO that all journals should focus on and why.

Despite the benefits of embracing modern innovations, some industries have been slow to adopt them - academic publishing being a prime example. In the post below, we break out 3 vestiges of print publishing that journals are holding onto online and why it's holding them back.

In the past few weeks, we've introduced some exciting updates to Scholastica's typesetting service to better serve the needs of journals across disciplines. We're making it easier for editors to submit articles for typesetting, giving journals new options for styling their PDF articles, and more.

In this interview Jacklyn Rander, the Library Publishing Services Manager at Grand Valley State University, and Matt Ruen, the Scholarly Communications Outreach Coordinator at Grand Valley State University, discuss the academic-led publications that they work with and their thoughts on the future of academic-led open access publishing.