Impact of a Change in Editorial Policy at Nature Publication Group (NPG) on Their Reporting of Biomedical Research

Malcolm Macleod,1 for the NPQIP Collaborative Group

Objective To determine whether a change in editorial policy, including the implementation of a checklist, was associated with improved reporting of measures that might reduce the risk of bias.

Design In this before-after study, we included articles that described research in the life sciences published in Nature Publication Group (NPG) journals that were submitted after implementation of mandatory completion by authors of a checklist at the point of manuscript revision (May 1, 2013, to November 1, 2014). We compared these with articles describing research in the life sciences published in Nature journals that were submitted before May 2013. Similar articles in other journals were matched for date and topic. We investigated the change in proportion of articles published before and after May 2013 reporting 4 criteria: information on randomization, blinding, sample size calculation, and exclusions. We included 448 articles published in NPG journals (225 [50.2%] published before May 2013 and 223 [49.8%] published after) that were identified by an individual hired by the NPG for this specific task, working to a standard procedure; an independent investigator used PubMed’s Related Citations feature to identify 447 similar articles with a similar topic and date of publication in other journals. We then redacted all publications for time-sensitive information and journal name. Redacted articles were assessed by 2 trained reviewers against a 74-item checklist, with discrepancies resolved by a third reviewer.

Results In total, 392 NPG articles and 353 similar articles in other publications described in vivo research. The number of NPG articles meeting all 4 criteria increased from 0 of 203 prior to May 2013 to 31 of 181 (17.1%) after (2-sample test for equality of proportions without continuity correction; χ² = 36.156; df = 1; P < .001). There was no change in the proportion of similar articles in other publications meeting all 4 criteria (1 of 164 [0.6%] before; 1 of 189 [0.5%] after). Agreement between reviewers ranged from 72% (for “Does the manuscript describe which method of randomization was used to determine how samples/animals were allocated to experimental groups?”) to 90% (for “Does the manuscript describe how the sample size was chosen to ensure adequate power to detect a prespecified effect size?”).

Conclusions There was a substantial improvement in the reporting of measures that might reduce the risk of bias in in vivo research in NPG journals following implementation of a mandatory checklist policy, to a level that, to our knowledge, has not been previously observed in science journals. However, there remain opportunities for further improvement.

Funding/Support: The study was funded by the Laura and John Arnold Foundation.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract.