Menu

Talking Peer Review: Q&A with AGU

On the 19th of September we will be announcing the winner of our Sentinel Award - for outstanding advocacy, innovation or contribution to scholarly peer review.

In this series of Q&A posts leading up to the announcement, we meet our eight finalists and get to know a bit more about them.

Up today, we speak with the American Geophysical Union (AGU).

Can you tell us a bit about AGU?

AGU is the largest Earth and space science society and will celebrate its 100th anniversary in 2019. Our mission is to promote discovery in the Earth and space science for the benefit of humanity. AGU has 60,000 members in 139 countries representing many disciplines including scientists who explore the surface, interior, oceans, atmosphere, and space environment of Earth. AGU holds one of the largest scientific meetings (about 25,000 attendees presenting 22,000 talks and posters recently) among several smaller conferences. AGU publishes 20 leading peer-reviewed journals and more than 6000 research papers each year. Four of these are Gold open-access journals. AGU has a long history of working to improve the integrity of research: AGU was one of the first societies to have a position statement and policy on open data (in 1997 and 1993, respectively), and AGU is working to address harassment across science (http://stopharassment.agu.org/). AGU also is engaged in worldwide community programs including the Thriving Earth Exchange (TEX), which connects scientists with community leaders to address local problems.

Can you briefly describe AGU's recent study on gender bias in scholarly peer review, and how you're working towards an improved peer review process?

Studying bias in peer review, and peer review in general, has been difficult because most publishers do not collect or release relevant data. For example, to understand gender or other bias in peer review and publication, age data--and gender data of course--are needed because most fields were historically dominated by men. These data are not captured in most editorial systems. Previous studies have assigned gender algorithmically, which can lead to uncertainties about as large as some effects. We were able to combine our member data, where age and gender are self-reported (and we checked for false reporting) with our editorial data. This gave us a large sample size given AGU’s large member base and number of publications.

We found that submissions with a woman as a first author had higher acceptance rates, regardless of age or author group size, which was in contrast to other studies. In addition, women in most age groups were underutilized as reviewers, which can have negative career implications. We are addressing this concern in several ways—first by exposing and discussing this effect broadly, secondly by informing our editors, and finally we added information about this bias to our instructions for authors for one journal as an experiment (since the bias starts with their suggestions for possible reviewers; this also raises awareness broadly). We are analyzing these data now and not yet seeing a significant improvement from this, so are considering raising the visibility further.

The immediate study was started after Marcia McNutt suggested we analyze our large editorial system to see if we could see any effect for a conference AAAS was organizing on implicit bias and peer review. We were skeptical that we could find significance until we saw how complete and extensive the member data was, both for age and gender, and that it was possible to merge it reliably with the editorial data to have a large sample size. This meant that we could account for perhaps the largest confounding factor (authors and editors picking “more senior” reviewers who were more predominately male). We are continuing this and will present an update at the Peer Review Congress.

What does transparency in peer review (the theme of this year's Peer Review Week) mean to you?

First, we feel that ensuring integrity in peer review is critical not just for advancing science, but for the benefit of science. Increasingly, “peer reviewed” science is used by society, and its use is specifically codified now in our legal, regulatory, and advisory systems. Transparency is a critical part of enhancing and insuring integrity and trust in the system and science. Most important it is about the published research, and reliable links to underlying data, references, people, and methods that persist over time. AGU, in partnership with other organizations (e.g., ORCID and CrossRef) is focused on addressing and improving these essential links (e.g., see below on data).

Transparency in the review process is also important to this integrity, and can be practiced and promoted even where the reviews themselves and/or the identity of reviewers are confidential. This can be through regular information and data about operations, oversight by boards and committees, and release of blinded data that allows the community and researchers to study peer review. We released such data for the gender study, and will now continue to do so annually. Transparency around editorial operations is perhaps more important in addressing various editorial conflicts of interest (e.g., in manipulating impact factors and address conflicts in open access publishing, where there is an additional financial incentive to accept papers) and needs further development of best practices across scholarly publishing.

What are you plans for the future?

We are continuing to work around enhancing integrity in science. As noted above we are continuing the research on implicit bias. We have three other immediate initiatives. We have just announced a large effort, funded by the Laura and John Arnold Foundation, that will extend across the Earth and space sciences in implementing the FAIR data principles in scholarly publishing—this extends an earlier initiative we helped start on the Coalition on Publishing Data in the Earth and Space Sciences (http://COPDESS.org). This initiative involves a partnership with leading publishers and repositories. We are also actively exploring a preprint solution in the Earth and space sciences, and working within our community and also with other communities to include posters and rich meeting presentations in the preprint culture. Finally, for peer review specifically, we are about to implement annotations in peer review across all our journals—using hypothes.is--to allow a better dialog among reviewers, editors, and authors.