Pitfalls of Peer Review

June 11, 2002

For 200 years, medical journals have increasingly depended on scientific peers to review their colleagues' articles and weed out shoddy work and methodological errors -- so as to prevent authors from making claims that cannot be supported by the evidence they report.

The system is time-consuming and expensive. Although most of the research that survives the process is solid, problems -- errors, plagiarism, even outright fraud -- have persisted.

Last week, the Journal of the American Medical Association published 34 articles from its latest meeting to consider research on the peer review process. The conclusions were reportedly grim.

Drummond Rennie, a journal editor, says that despite improvements in peer review, "there still is a massive amount of rubbish."

An estimated two million new articles are published each year in journals of varying quality -- mostly paid for by the public through government grants.

Peer-review researchers found considerable evidence that many statistical and methodological errors were common in published papers and that authors often failed to discuss the limitations of their findings.

Even the press releases that journals issue to steer journalists to report peer-reviewed papers often exaggerate the perceived importance of findings and fail to highlight important caveats and conflicts of interest.

Because the peers chosen to review papers remain anonymous and are often the authors' competitors, jealousies and competitive advantage can become factors in the process. Reviewers have been caught publishing information they lifted from others' research manuscripts.