Peer Review Is Unscientific

A reviewer of peer review says our reverence for the practice borders on mysticism, not science.

Drummond Rennie has had a lot of experience reviewing peer review. He has seen the seedy side: fraud, plagiarism, destroyed careers. In Nature, he calls to “make peer review scientific” — a startling appeal about a practice assumed to represent a hallmark of science.

Peer review is touted as a demonstration of the self-critical nature of science. But it is a human system. Everybody involved brings prejudices, misunderstandings and gaps in knowledge, so no one should be surprised that peer review is often biased and inefficient. It is occasionally corrupt, sometimes a charade, an open temptation to plagiarists. Even with the best of intentions, how and whether peer review identifies high-quality science is unknown. It is, in short, unscientific.

A long time ago, scientists moved from alchemy to chemistry, from astrology to astronomy. But our reverence for peer review still often borders on mysticism. For the past three decades, I have advocated for research to improve peer review and thus the quality of the scientific literature. Here are some reflections on that winding, rocky path, and some thoughts about the road ahead.

Rennie opens a gory box of actual instances of abuse he has seen, and calls abuses widespread. Even clinical trials do not escape his prosecution:

Thanks to such research, we now know a great deal about the mechanics of peer review — the time taken to appraise papers, rates of disagreement between reviewers, the cost at certain journals, even the occurrence of misconduct during review.

Research has brought clear improvement to the biased reporting of clinical trials. Randomized clinical trials cost millions of dollars, are rarely repeated, and greatly influence what treatments patients receive. My colleagues and I showed that most trial results in submitted manuscripts favoured the treatment tested, and this was reflected in the results that were published. Other work revealed that more than 90% of the bias was due to authors failing to submit manuscripts that are unfavourable to the treatment, and that commercial sponsorship drove decisions not to submit. Although any single trial might have been conducted well, the system was skewed. Publication bias made drugs look better than they were.

Something is fishy here. Peer review is dead, and dead fish stink. Rennie calls for major overhauls of peer review, a process many in the public are fooled into believing helps guarantee scientific integrity. It’s ironic that something invented to ensure that research is scientific is itself unscientific.

Integrity takes morality. You can’t have scientific integrity without it. Morality cannot emerge from particles. Honesty cannot be described by particles in motion. Integrity cannot evolve. PhysOrg posted a piece titled, “How to get moral ‘free riders’ to cooperate.” But if social networks evolve, as Darwinians believe, there is no standard. Once the free-riders predominate, they take the new moral high ground, and their critics become the non-cooperators.

In short, you need a Biblical worldview to do science. Even then, it will be mediated by fallible sinners. Bible believers, though, will have the best motivation to work honestly, as long as they follow the Creator’s command, “You shall be holy, for I am holy.”