Peer review highly sensitive to poor refereeing, claim researchers

Just a small number of bad referees can significantly undermine the ability of the peer-review system to select the best scientific papers. That is according to a pair of complex systems researchers in Austria who have modelled an academic publishing system and showed that human foibles can have a dramatic effect on the quality of published science.

Scholarly peer review is the commonly accepted procedure for assessing the quality of research before it is published in academic journals. It relies on a community of experts within a narrow field of expertise to have both the knowledge and the time to provide comprehensive reviews of academic manuscripts.

While the concept of peer review is widely considered the most appropriate system for regulating scientific publications, it is not without its critics. Some feel that the system's reliance on impartiality and the lack of remuneration for referees mean that in practice the process is not as open as it should be. This may be particularly apparent when referees are asked to review more controversial ideas that could damage their own standing within the community if they give their approval.

Questioning referee competence

Stefan Thurner and Rudolf Hanel at the Medical University of Vienna set out to make an assessment of how the peer-review system might respond to incompetent refereeing. "I wanted to know what would be the effects on peer review as a selection mechanism if referees were not all good, but behaved according to different interests," Thurner told physicsworld.com.

The researchers created a model of a generic specialist field where referees, selected at random, can fall into one of five categories. There are the "correct" who accept the good papers and reject the bad. There are the "altruists" and the "misanthropists", who accept or reject all papers respectively. Then there are the "rational", who reject papers that might draw attention away from their own work. And finally, there are the "random" who are not qualified to judge the quality of a paper because of incompetence or lack of time.

I wanted to know what would be the effects on peer review as a selection mechanism if referees were not all good, but behaved according to different interests
Stefan Thurner

Within this model community, the quality of scientists is assumed to follow a Gaussian distribution where each scientist produces one new paper every two time-units, the quality reflecting an author's ability. At every step in the model, each new paper is passed to two referees chosen at random from the community, with self-review excluded, with a reviewer being allowed to either accept or reject the paper. The paper is published if both reviewers approve the paper, and rejected if they both do not like it. If the reviewers are divided, the paper gets accepted with a probability of 0.5.

Big impact on quality

After running the model with 1000 scientists over 500 time-steps, Thurner and Hanel find that even a small presence of rational or random referees can significantly reduce the quality of published papers. When just 10% of referees do not behave "correctly" the quality of accepted papers drops by one standard deviation. If the fractions of rational, random and correct referees are about 1/3 each, the quality selection aspect of peer review practically vanished altogether.

"Our message is clear: if it can not be guaranteed that the fraction of rational and random referees is confined to a very small number, the peer-review system will not perform much better than by accepting papers by throwing (an unbiased!) coin," explain the researchers.

Daniel Kennefick, a cosmologist at the University of Arkansas with a special interest in sociology, believes that the study exposes the vulnerability of peer review when referees are not accountable for their decisions. "The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says.

Kennefick feels that the current system also encourages scientists to publish findings that may not offer much of an advance. "Many authors are nowadays determined to achieve publication for publication's sake, in an effort to secure an academic position and are not particularly swayed by the argument that it is in their own interests not to publish an incorrect article."

Don't forget the editors

But Tim Smith, senior publisher for New Journal of Physics at IOP Publishing, which also publishes physics world.com, feels that the study overlooks the role of journal editors. "Peer-review is certainly not flawless and alternatives to the current process will continue to be proposed. In relation to this study however, one shouldn't ignore the role played by journal editors and Boards in accounting for potential conflicts of interest, and preserving the integrity of the referee selection and decision-making processes," he says.

Michèle Lamont a sociologist at Harvard University who analyses peer review in her 2009 book, How Professors Think: Inside the Curious World of Academic Judgment, feels that we expect too much from peer review. Lamont believes that we should never hope for "uncorrupted" evaluation of new science as all researchers are embedded in social and psychological networks. She feels that one way to improve the system, however, is to make assessment criteria more relevant to specific disciplines.

When asked by physicsworld.com to offer an alternative to the current peer-review system, Thurner argues that science would benefit from the creation of a "market for scientific work". He envisages a situation where journal editors and their "scouts" search preprint servers for the most innovative papers before approaching authors with an offer of publication. The best papers, he believes, would naturally be picked up by a number of editors leaving it up to authors to choose their journal. "Papers that no-one wants to publish remain on the server and are open to everyone – but without the 'prestigious' quality stamp of a journal," Thurner explains.

This research is described in a paper submitted to the arXiv preprint server.

About the author

30 comments

Good modelling, good article, I am reminded of In Search of the Black Swans. As to the role of journal editors, I've spoken to people who have submitted "breakthrough" papers with a robust connection to scientific evidence to a particular journal, only to have them rejected because they are "of little interest". But then the accepted papers turn out to be of little interest. So when I read:

"the peer-review system will not perform much better than by accepting papers by throwing an unbiased(!) coin"

...I wonder if we'd be better off just throwing the coin. In fairness though, this doesn't apply to all journals, or just to journals. Some papers do make it through, but then we don't see them on the front page of The Times.

Good modelling, good article, I am reminded of In Search of the Black Swans. As to the role of journal editors, I've spoken to people who have submitted "breakthrough" papers with a robust connection to scientific evidence to a particular journal, only to have them rejected because they are "of little interest". But then the accepted papers turn out to be of little interest. So when I read:

"the peer-review system will not perform much better than by accepting papers by throwing an unbiased(!) coin"

...I wonder if we'd be better off just throwing the coin. In fairness though, this doesn't apply to all journals, or just to journals. Some papers do make it through, but then we don't see them on the front page of The Times.

Truthfully, without putting too fine a point on it, peer review is an antiquitated system, which is truly unsuitable to todays scientific movement. Any slightly new idea however good recieves a rough ride - unless of course you already happen to be a professor of physics, in which case you are not likely to produce new ideas.

Peer review requires a substantive overhaul, and that means more say for the authors and more accountability by the reviewer. Some reviwers actually make things up to support their biased or negative views- and the author should be allowed to rebut such a reviewer without fear of outright rejection. These are just a few pointers as to how to improve the system.

This study overlooks not only the role of the editor, but also the process in which the authors are able to answer the referees' objections. When the referees are competent, this leads to better papers through useful suggestions. On the other hand, when they aren't, overcoming the exasperation of the authors, their objections are easily brushed away, and the paper eventually gets through. Also, when the case is particularly contentious, there's still the option of calling for an adjudicator. In summary, the peer-review process is far more complex than this simulation might suggest. On the dark side, I’ve also noticed that referees are sometimes reluctant to object papers from certain renowned authors. The human factor is hard to remove. I guess many people will agree that there’s a need to look for better approval systems, specially today, when there’s an explosion of submissions. However we must also acknowledge that the present system has served its purpose of maintaining a certain quality.

As the peers are humans, the peer review will have the strengths and deficiencies of humans. As an Author, I have had a communication rejected by a Journal, because one reviewer felt the question we were addressing was already answered and should not be opened up again. Other referee was excited but for the editor, it had to be two unqualified YES. I did ask for an adjudicator and that did not help. In the end, that paper was published in another equally good journal, in which (quite unusually for me) the Editor wrote a note that he was quite pleased to accept the trend setting paper and thanked me for submitting to that Journal. Of course, this was after two referees had accepted the paper.One way we can reduce the flaws in this system is to stop counting the number of publications in making decisions about promotions, awards and fellowships etc... This would encourage the Authors to be more careful before sending a paper, reduce the number of submissions and give more time for referees to critically look at the paper. Of course, there has to be a balance and I would say referees should be given one month rather than the 1 or 2 weeks, many Journals give now. The bottom line is that peer review is not flawless but I do not see anything that would be better. Both as an author and reviewer, I have seen the quality of several publications improving due to the review process.

50/50

1. One enormous parameter is the 50/50 rate of acceptance/rejection by the editor if the reviewers are not in agreement. This parameter could modify the stability state of the system. Editors could report such a ratio, every issue or so, but they won't. 2. Another major problem is the time delay between submission and reviewing; is it due to the lack of care of reviewers or incompetence or something else; how many papers are rejected by slow reviewers; this could also be a number given by editors; they won't3. I also tried to get information about the evolution of a paper content through revised versions and timing; everybody knows that sometimes/often a reviewer will accept a paper only if some (!) reference is made to some particular (!) work; editors I contacted refused access to such an information4. Finally on the role of editors; it is know that in journals with high impact factor, some papers are rejected without going to a reviewer, for various reasons; although most of the time it is said "because the paper is inappropriate or too specialized" for the readership, while it appears clearly that it depends much on the "potential sensationalism" and "affiliation of authors"; too many examples are well known to each of us. This filter is also relevant for modeling scientific knowledge spreading

New scientific knowledge beyond standard science.

The 'opinion' in 1893 when peer review was initiated, reflected the flawed belief there was no new science yet to be discovered and therefore there would always be an equal or authority to review. Occasionally when a 'significant' breakthrough occurs and 'new scientific knowledge' is to be presented that is beyond 'standard science': there exists none who can be considered an 'equal' or 'authority' to review. When this occurs in the field of medicine, many patients die needlessly: science in general, is denied the knowledge that is needed. Robert Wood-Smith. Consultant.

To my opinion, the proposed 'market' does already exist. Journal editors do approach authors of posted preprints to submit their works to their journal. But mostly it is the authors who decide whether or not they will submit and, if yes, where. In this author/editor market then it's the task of the editors, possibly with help from reviewers, to decide if they want to take up that offer.After having worked for more than ten years as a physics journal editor, I am fully convinced that the editorial and peer-review system fulfills a crucial function in selecting the most relevant content and improving submitted work through editor and referee suggestions, often quite substantially. That does not mean that the system is perfect and error-free. Another trend has indeed been increasing responsibilites of editors, i.e. the peer-review process is much different from a mechanical procedure where editors simply make an averaged decision from referees' recommendations. This concerns both pre-selection and screening of submitted material (for reasons of topical relevance, inflation of submissions, salami-slicing tendencies to present only incremental progress etc.) and the decision-making based on referee reports (and their interpretation), Editor/Board assessments, author responses, rebuttals, adjudication and so on. Of course, all such decisions are done by human beings and require certain experience, so in that sense results may vary.

Twin-Tori cosmological Model.

Dear friends,

I am happy to read about the "peer-review" to be an antique system !!

The new cosmological model for a double torus of dark energy and dark matter is a good example for completely new science to be ignored. Why? Because at the moment, the model is provided with higher-mathematics "never been seen", unfolding a subquantumlevel to lead to new physics and cosmology.

However, the "Journal of Mathematics" rejected the first papers by the fear to judge a new hypothesis, that might be a threat for existing theories. Isn't that much money is invested in experiments to prove existing theories to be true, so that referees filter subjectively? I think it is !!

A submission to Nature, about the developement of the Twin-Tori Model, and about where to find information, was to be acknowledged as to be "out of their reach" and "too special". I didn't believe a word of it ! It was just a diplomatic way of saying: Do not interfere with our existing beliefs !

However, many of you responded to me enthousiastically. I am thankfull for that. But:

When will Physics World for example will have the courage to invite me to explain to a journalist what this Twin-Tori Model is about?

Peer review cannot work well under the situation, when the density of informations and the degree of specialization increases up to level, only experts which are very close to authors can judge their article in qualified way. It violates the anonymity of referees and their objectiveness undeniably.

If everything we know now, is based on everything that we've known before this moment... How are we ever supposed to learn something new?

Fortunately or unfortunately, the answer is: "By leaps and bounds!"

I'm not saying science should walk off the plank holding hands with subjective faith. But I am saying is that sometimes, the thought experiment just can't be put into words. It can be modeled, and objectified, but will never the "EUREKA" moment that concieved of the idea.

Germs didn't exist until the microscope was invented... or did they?

My point is that the mind is the most powerful tool. It is the only thing that drives us. Idea's cannot be thrown out because we don't know. They need NOT be thrown out, because we SHOULD know!

An open refereeing system

I agree that "The problem is ANONYMOUS peer review that holds nobody accountable!", and that it is an "antique system".Five years ago I suggested to use the arXiv to establish an open refereeing system. The referee will be responsible for the report.See section 3.2 here: arxiv.org…0508525v1.pdf

Re: "If everything we know now, is based on everything that we've known before this moment... How are we ever supposed to learn something new? Fortunately or unfortunately, the answer is: "By leaps and bounds!"

Exactly! And exactly these leaps and bounds are prevented by "scientific" lobbies, dominating the universities, media and review. See: An Open Letter to the Scientific Community www.cosmologystatement.org

What is denied by mainstream "science" is the MIND that produces the EUREKA.

Are we SERIOUS when reviewing?

Well the point here is: how to know if the referees (the peers that review) lies into one or another category that the authors of the book have discerned? I agree that we can found examples of these five all around. From my personal experience reviewing for several journals, and reading published papers into that areas I specialize, I'm convinced that at present there are an increasing number of papers that are really useless. However, the real problem are not these papers, but those PAPERS that HAVE BEEN ACCEPTED and PUBLISHED even by well recognized journals and containing not only unreal information and results, but also INVENTED and COOCKED results. Of course, the editors can't take in charge the task of read and qualify every review he/she receives. It's on the credibility of the peers that lies the problem. I'm also a member of an editorial board and I know very well that the editors ask every year on the ways to promote the journal. But, the editor doesn't need to accept anything else to fill every issue. There are a lot of submissions awaiting for publication at every journal out there. A lot more that they can publish at any time. The QUALITY of the review lies ONLY onto the SERIOUS WORK of the peers. After reading a number of PUBLISHED papers plenty of flaws, the question is: how can we now trust the contents in scientific papers? SERIOUS refereeing is really being undermined by some peers that are not taking its responsability as it must be. Of course, we would want to be paid for this work. And after the actual prices of subscriptions, the editorial houses could already start considering this way. At the present state, editorials have the work of reviewing made FOR FREE. But they sell subscriptions to their journals at very high prices. And not only this, but now they ask for this FREE WORK made every time quicker. Nevertheless, this won't be the solution to the REAL problem: poor refereeing. This is a question of ETHICS.

New "Peer" Review System Critical

The Peer Review process of yesteryear, which is still abused today by far too many, should be replaced as recommended above. The "open market" suggestion is quite the way to go; the system is in place and ready on the WWW.

Constipated Science

Re: "If everything we know now, is based on everything that we've known before this moment... How are we ever supposed to learn something new? Fortunately or unfortunately, the answer is: "By leaps and bounds!"

Exactly! And exactly these leaps and bounds are prevented by "scientific" lobbies, dominating the universities, media and review. See: An Open Letter to the Scientific Community www.cosmologystatement.org/

What is denied by mainstream "science" is the MIND that produces the EUREKA.

Standard Models

Guesses, suppositions and assumptions are at the core of developing science. The problem lies in over-complexed, mathematical assurances by the 'guessers', 'assumers' and 'supposers'. Black Holes and Event horizons were assumptions and have been held as fact since conception. Few have challenged these assumptions, and when they have, those challengers have suffered ridicule and derision by a majority - just like Galileo and the spherical Earth.The truth of a thing, when found, is simple and not over complicated. Nature is not complicated, it is our attempts at understanding it that are. History shows this. Heisenbergs Principle of uncertainty complicates, and has literally, held sway over theoretical physics for 80 years or more, spawning yet more guesses and assumptions as yet unratified. This is not science; Science is defined as "to know". Careers have been formed on suppositions still unratified. Editorial censorship is rife, boat-rocking is "prohibited" and physical science has largely, become a closed shop.An 17yr old student once told me that his Physics master had said, "When you get to advanced Physics level, you'll find that a proportion of the Physics I've taught you is wrong.". The boy then asked, "So why did you teach it to me then?", to which his master replied, "Because it's on the curriculum." Several students have verified this position to me.Nikola Tesla assumed the quantity of Earth's electrical capacitance. He asserted that it was only an assumption. It wasn't proven until after his death. Tesla was honest about his assumptions.I give no credentials here, merely observations and experiment.I wish Physics would adhere to those principles.

Funding agencies?

My experience has been that this is a far larger issue at funding agencies than at journals. This is especially true of the NSF, where program officers appear to believe that they have no responsibilities other than serving as a conduit for paperwork. In particular, no one there seems to be in charge of insuring that merit review guidelines are actually followed by review panels, with the result that they are in fact not followed at all, and cliques have developed that essentially hand money around in a big circle.

With a publication, you can always carry it to another journal. This is not true of funding agencies. So the stakes in favor of bad behavior are much higher.

Anonymous Peer Review at Funding Agencies

My experience has been that this is a far larger issue at funding agencies than at journals . . . .

With a publication, you can always carry it to another journal. This is not true of funding agencies. So the stakes in favor of bad behavior are much higher.

PJ Camp is exactly right.

Anonymous peer review at funding agencies is the process that makes young scientists comply with NAS/FRS dictates in exchange for grant funds, publications, tenure, promotion, etc.

That is also why modern science became stagnant and unattractive.

Thanks to Steve McIntyre and others that exposed the deep roots of the Climategate scandal - an international alliance that includes Al Gore, the UN’s Intergovernmental Panel on Climate Change (IPCC), world leaders, the US National Academy of Sciences (NAS), the Royal Society, the Norwegian Nobel Prize Committee, and editors of leading research journals - there will now be a revolution in science, sweeping away many of the obsolete concepts that have been protected for decades by anonymous peer review.

"The scientific-technological elite", the "education elite", together with the "funding, reviewing, publishing, reporting and what have you elite", are only prostituting themselves to globalizing commercial interests.

"The secret of commerce, that kids drive purchases, – wrote John Taylor Gatto in his prise-winning book, The Underground History of American Education – meant that schools had to become psychological laboratories where training in consumerism was the central pursuit. . . The truth is that America's unprecedented global power and spectacular material wealth are a direct product of a third-rate educational system, upon whose inefficiency in developing intellect and character they depend. If we educated better we could not sustain the corporate utopia we have made. Schools build national wealth by tearing down personal sovereignty, morality, and family life. It was a trade-off."

Thus, as Lewis Mumford has expressed it, "We have created an industrial order geared to automatism, where feeblemindedness, native or acquired, is necessary for docile productivity in the factory; and where a pervasive neurosis is the final gift of the meaningless life that issues forth at the other end." … "Today our best plans miscarry because they are in the hands of people who have undergone no inner growth. Most of these people have shrunk from facing the world crisis and they have no notion of the manner in which they themselves have helped to bring it about. Into every new situation they carry only a fossilized self."

What all those mainstream ""elites"" are promoting, are their ossified convictions, each producing its own what C.H. Waddington, the famous biologist, genetist and epigenesist well characterized as the Conventional Wisdom of the Dominant Group, and pregnantly acronymed for "COWDUNG", which – so Waddington – "is memorizable, appropriate and accurate enough. During most of this century [20th], the conventional wisdom of the dominant group about the nature of living organisms has been a rather exaggerated form of the 'thing' view; and when this applied to man and his social affairs, it seems, to me at least, to fall appropriately under the heading COWDUNG. It argues that the world and everything in it is constituted from arrangements of essentially unchanging material particles, whose nature has essentially been largely, if not entirely, discovered by the researches of physics and chemistry. These physico-chemical entities are supposed to constitute the whole of objective reality."

It appears to be generally ignored that never in the human history did something new originate from the mainstream, but from spearheading individuals.

Fairly well thought out alternatives do exist...

Embedded in "EarthWeb", by Marc Stiegler, is a lengthy analysis of a "reputation" based process. "Reputation" literally evolves in an ongoing process of rating and establishing metric for individuals' credibility, trustworthiness, fair-play, capability, "what is your Word worth", etc.

Granted, in the context of the book, "peer review" would tend to be more of a Delphic exercise, where judgments of multiple reviewers would be "weighted" by their "reputations", before being aggregated as a consensus response. And, to some extent, those reviewers who were seriously contradicted by the consensus would lose some "credibility" points. Ongoing evolution, remember.

Nicely, the economic incentives of "how to GET reviewers to take the time" become driven by the mechanism whereby the "QUALITY" of a "reputation" is improved by demonstrated expertise -- such that people would go LOOKING for opportunities to do reviews, specifically focusing in their areas of expertise, as one relatively low-commitment method of "bettering" their recognized "standing" in a reputation-driven ecology.

Facebook, Linked-in, Twitter: looking for "the next BIG thing in Social Networking"?? Significant work has already been done; now just have to work out how to "get it out there."