Introduction to Traditional Peer Review

Bonnie J. M. Swoger is a Science and Technology Librarian at a small public undergraduate institution in upstate New York, SUNY Geneseo. She teaches students about the science literature, helps faculty and students with library research questions and leads library assessment efforts. Bonnie started her professional life as a geologist, but realized that she was much more interested in how scientists communicate their research to one another. As a librarian, she gets to teach others about the topic. She has a BS in Geology from St. Lawrence University, an MS in Geology from Kent State University and an MLS from the University at Buffalo. She lives in upstate New York with her husband, two young daughters and two old cats. She would love to have some free time in which to indulge in hobbies. She also blogs at the Undergraduate Science Librarian and can be found on twitter @bonnieswoger.

Hadas Shema is an Information Science graduate student at Bar-Ilan University, Israel. She studies the characteristics of online scientific discourse and is a member of the European Union’s Academic Careers Understood through Measurement and Norms (ACUMEN) project. Hadas tweets at @Hadas_Shema. Follow on Twitter @infoculture2.

Hadas Shema is an information specialist at the Israeli Inter-University Center for E-Learning (Hebrew acronym: MEITAL). She has a B.Sc. in the Life Sciences and an MA and a PhD in Library & Information Science from Bar-Ilan University, Israel. Hadas tweets at @Hadas_Shema.

Peer review was introduced to scholarly publication in 1731 by the Royal Society of Edinburgh, which published a collection of peer-reviewed medical articles. Despite this early start, in many scientific journal publications the editors had the only say on whether an article will be published or not until after World War II. “Science and The Journal of the American Medical Association did not use outside reviewers until after 1940, “(Spier, 2002). The Lancet did not implement peer-review until 1976 (Benos et al., 2006). After the war and into the fifties and sixties, the specialization of articles increased and so did the competition for journal space. Technological advances (photocopying!) made it easier to distribute extra copies of articles to reviewers. Today, peer review is the “golden standard” in evaluation of everything from scholarly publication to grants to tenure decision (in the post I will focus on scholarly publication). It has been “elevated to a ‘principle’ — a unifying principle for a remarkably fragmented ﬁeld” (Biagioli, 2002).

Peer review takes time, effort, and can delay publishing considerably. It’s one of the bottle necks of scholarly publishing. We can distribute very quickly these days, but there are a limited number of experts in every field to review and their time is occupied by other tasks that are part of the academic life. Blaise Cronin, editor of JASIST, estimated that JASIST needs about 1,000 peer reviews a year (one person, of course, can review more than once) for 400 articles and that JASIST approach about 3,000 researchers to find the 1,000 peer reviewers.

Single or double blind?

Traditional peer review is usually single or double blind. Single blind review, where the reviewers are aware of the authors’ identities, but the authors are not aware of the reviewers’ identities, is the most common. Double-blind peer review is when neither the authors nor the reviewers are aware of the others’ identities. Authors tend to believe double-blind reviews are better and less biased, in principle, than single-blind, but they also doubt whether true blinding is possible (See review in Lee et al., 2013). A survey of editors-in-chief, editors and editorial board members of 590 chemistry journals found that 97% of the journals didn’t offer double-blind peer review. Most respondents considered double blinding needless because the content and references could not be truly masked. They thought double blinding would make the detection of fraud harder and considered the system satisfactory as it was (Brown, 2007).

An example: in my new work place we hold an annual e-learning in high education conference. When we receive proposals, one of my jobs is to anonymize them – remove their names and make sure that wherever the authors wrote their institute’s name it will be replaced with simply “the institute”. Unfortunately, the Israeli e-learning community is so small the process is pretty useless, because everyone knows everyone and everyone knows what goes on in which institute. A study by Justice et al. (1998) reached similar conclusions. They masked the authors’ identity in manuscripts sent to five prominent medical journals before sending them to be reviewed, but about 30% of the reviewers were able to identify the authors regardless (perhaps because self-references in the text weren’t removed). In small research fields this number can go even higher.

Peer review, fraud, and mistakes

Whenever cases of fraud in published article or significant flaws are revealed, it seems natural to wonder “where were the peer reviewers?” Researcher Jan Hendrick Schon, for example, published over a hundred articles in four years (1998-2002) and the reviewers failed to detect 16 cases of misconduct in his articles. Sometimes, it’s simply a matter of chance. The blog Retraction Watch reported lately that after a colleague pointed out an (honest) error in their article, a group of authors retracted a paper from Physical Review Letters. Had this colleague been their pre-publication peer reviewer, the article wouldn’t have been published to begin with.

In a study, eight weaknesses were entered into an article already accepted for publication and sent to JAMA reviewers (200 respondents) who, on average, found less than two per reviewer. Sixteen percent didn’t find any weakness and only 10% found more than four (Godlee et al., 1998). Callaham et. al. (1998) sent a fake manuscript, containing 23 deliberate flaws, to editors of Annals of Emergency Medicine and all the journal’s peer reviewers who reviewed at least three manuscripts before the study. On average, the reviewers detected 3.4 of the ten major flaws in the manuscript and 3.1 of the 13 minor ones. Peer reviewers are the ‘gate-keepers’ of science, but their gate-keeping is far from perfect.

Inter-reliability of peer review

Peer review tends to have low levels of inter-rater reliability between reviewers (0.2-0.4). This, at least from a statistical point of view, makes them pretty unreliable. However, this might not be a bad thing: “Too much agreement is in fact a sign that the review process is not working well, that reviewers are not properly selected for diversity, and that some are redundant” (Bailar, 1991). It could be the reviewers give different weight to different qualities of the reviewed article, or that the article’s subject has not reached a scientific consensus (e.g.: altmetrics, useful tools for evaluation or rubbish?).

I’m well aware this post covers only a small part of the discussions and arguments about peer review (note the post is called “Introduction to traditional peer review), and hope to discuss the topic again in the future.

Bornman, L. (2008). Scientiﬁc Peer Review: An Analysis of the Peer
Review Process from the Perspective of Sociology
of Science Theories Human Architecture: Journal of the Sociology of Self-Knowledge, 6 (2)

About the Author: Hadas Shema is an information specialist at the Israeli Inter-University Center for E-Learning (Hebrew acronym: MEITAL). She has a B.Sc. in the Life Sciences and an MA and a PhD in Library & Information Science from Bar-Ilan University, Israel. Hadas tweets at @Hadas_Shema.

Excellent article on Traditional Peer Review.
Drummond Rennie, deputy editor of Journal of the American Medical Association is an organizer of the International Congress on Peer Review and Biomedical Publication, which has been held every four years since 1986. He remarked:
There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.
Richard Horton, editor of the British medical journal The Lancet, said:
The mistake, of course, is to have thought that peer review was any more than just a crude means of discovering the acceptability—not the validity—of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.
The interposition of editors and reviewers between authors and readers may enable the intermediators to act as gatekeepers. Some sociologists of science argue that peer review makes the ability to publish susceptible to control by elites and to personal jealousy. The peer review process may suppress dissent against “mainstream” theories. Reviewers tend to be especially critical of conclusions that contradict their own views, and lenient towards those that match them. At the same time, established scientists are more likely than others to be sought out as referees, particularly by high-prestige journals/publishers. As a result, ideas that harmonize with the established experts’ are more likely to see print and to appear in premier journals than are iconoclastic or revolutionary ones. This accords with Thomas Kuhn’s well-known observations regarding scientific revolutions. A theoretical model has been established whose simulations imply that peer review and over-competitive research funding foster mainstream opinion to monopoly. A marketing professor argued that invited papers are more valuable because papers that undergo the conventional system of peer review may not necessarily feature findings that are actually important.
Peer review failures occur when a peer-reviewed article contains fundamental errors that undermine at least one of its main conclusions. Many journals have no procedure to deal with peer review failures beyond publishing letters to the editor.
Peer review in scientific journals assumes that the article reviewed has been honestly prepared and the process is not designed to detect fraud.
Dr.A.Jagadeesh Nellore(AP),India

Together with the growing tendency of researchers & research institutions to block access to the data they claim supports their conclusions, the scientific community needs to excise these ideological tumours & get back to basic integrity

One more problem: reviewers receive nothing or very little compensation for a review, so are likely to do the job very superficially. In fact, articles for review are often given to Ph.D. students to do in spare time and so on.

Talking as the guy whose paper once received a positive review, only the reviewer talked about the different species of study animal than the paper!

Most important: peer review is a product of the times when science journals were few, but the number of articles grew quickly. Journals had to strictly filter articles to fit inside a limit of paper pages. This limit of space ceased to exist as the electronic publishing became dominant (today even paper journals are usually accessed online).

So pre-publication peer review should be replaced by post-publication peer review, or some sort of community voting, or similar.

Most important novelty in science may be completely fresh ideas and paradigm shifts. So how peer review might evaluate that? Review is about technical details. Faced with totally new idea, a reviewer might ‘get it’ or not. Most likely, he/she will be conservative and old member of his/her research discipline and reject it. Perhaps this is the reason why there are few truly novel discoveries, despite exponential growth of published papers.

I agree with Jerzy — the issues we have with peer review being slow, inefficient, and/or ineffective largely stem from the lack of reward reviewers get for their work.

The reward doesn’t need to be financial; we just need a way to turn reviews into an acceptable research output that can added to one’s resume. If each review is recognized (and appropriately rewarded) as a valuable contribution to science, with some measure of the speed and quality of the reviewer, we would expect to see the speed and quality of peer review increase across the board.

Hi!: I’d say that the first peer reviewed scientific assessment, and also the first Clinical Trial ever, can be considered when Louis ‘XVI of France’ hired Benjamin Franklin, then US ambassador in Paris, along with Antoine Lavoisier, Joseph Ignace Guillotin and Jean Sylvian Bailly, to ascertain if the so called: ‘Animal magnetism’, proposed by Mesmer, was a true fact or a fake, the teamsters’ conclusion was in the line of non-existence of this ‘Instant hypnosis’, or ‘Will’.