London 31 October: A new report published by the Global Warming Policy Foundation reveals the extent to which current policy-making is reliant on untrustworthy peer-reviewed research, much of which cannot be replicated and “may be simply untrue”.

click on the image above to watch a short video about the GWPF report

“Fraudulent research makes it past gatekeepers at even the most prestigious journals,” says Donna Laframboise, the study’s author and the investigative journalist behind the 2011 exposé of the Intergovernmental Panel on Climate Change (IPCC) entitled: The Delinquent Teenager Who Was Mistaken for the World’s Top Climate Expert.

The report, entitled Peer Review: Why Skepticism is Essential, describes the peer-review process as “haphazard and byzantine”, raising serious questions about the state of modern science and casting doubt on policies that claim to be ‘evidence-based’.

Laframboise explains: “A policy cannot be considered evidence-based if the evidence on which it depends was never independently verified… News from the worlds of astrobiology, ecology, economics, chemistry, computer science, management studies, medicine, neuroscience, psychology, and physics all tell the same tale: ’peer-reviewed’ does not equal ’policy-ready’.”

This has striking implications for climate change policy, and particularly for the IPCC, which relies on the credibility of the peer-review process to provide support for its conclusions, and is quick to dismiss research that has not been peer-reviewed.

Laframboise describes how the now-disgraced former IPCC chairman Rajendra Pachauri was once asked if an Indian environment ministry report might alter the IPCC’s pessimistic view of Himalayan glaciers. The ‘IPCC studies only peer-review science’, Rajendra Pachauri replied dismissively. Until the report’s data appears in ‘a decent credible publication’, he said, ‘we can just throw it into the dust- bin’.”

We’ve all heard the buzzword. Whether it’s an anti-bullying program in Finland, an alcohol awareness initiative in Texas, or climate change responses around the globe, we’re continually assured that government policies are ‘evidence-based.’ Science itself guides our footsteps.

There’s just one problem: science is in deep trouble. Last year, Richard Horton, editor of The Lancet, admitted that “much of the scientific literature, perhaps half, may simply be untrue.” In his words, “science has taken a turn toward darkness.”

Medical research, psychology, and economics are all in the grip of a ‘reproducibility crisis.’ A pharmaceutical company attempting to confirm the findings of 53 landmark cancer studies was successful in only six instances, a failure rate of 89%. In 2012, a psychology journal devoted an entire issue to reliability problems in that discipline, with one essay titled “Why science is not necessarily self-correcting.” Likewise, a 2015 report prepared for the Board of Governors of the US Federal Reserve concluded that “economics research is usually not replicable.” Its authors were able to verify the findings of only one third of 67 papers published in reputable economics journals. After enlisting the help of the original researchers, the success rate rose to a still dismal 49%.

Government policies can’t be considered evidence-based if the evidence on which they depend hasn’t been independently verified, yet the vast majority of academic research is never put to this test. Instead, something called peer review takes place. When a research paper is submitted, journals invite a couple of people to evaluate it. Known as referees, these individuals recommend that the paper be published, modified, or rejected.

If one gets what one pays for, it’s worth observing that referees typically work for free. They lack both the time and the resources to perform anything other than a cursory overview. Nothing like an audit occurs. No one examines the raw data for accuracy or the computer code for errors. Peer review doesn’t guarantee that proper statistical analyses were employed, or that lab equipment was used properly.

Referees at the most prestigious of journals have given the green light to research that was later found to be wholly fraudulent. Conversely, they’ve scoffed at work that went on to win Nobel Prizes. Richard Smith, a former editor of the British Medical Journal, describes peer review as a roulette wheel, a lottery, and a black box. He points out that an extensive body of research finds scant evidence that this vetting process accomplishes much at all. On the other hand, a mountain of scholarship has identified profound deficiencies.

Peer review’s random and arbitrary nature was demonstrated as early as 1982. Twelve already published papers were assigned fictitious author and institution names before being resubmitted to the same journal 18-32 months later. The duplication was noticed in three instances, but the remaining nine papers underwent review by two referees each. Only one paper was deemed worthy of seeing the light of day the second time it was examined by the same journal that had already published it. Lack of originality wasn’t among the concerns raised by the second wave of referees.

Anyone can start a scholarly journal and define peer review however they wish. No minimum standards apply and no enforcement mechanisms ensure that a journal’s publicly described policies are actually followed. Some editors admit to writing up fake reviews under the cover of anonymity rather than going to the trouble of recruiting bona fide referees. In 2014, a news story reported that 120 papers containing computer-generated gibberish had nevertheless survived the peer review process of reputable publishers.

Politicians and journalists have long found it convenient to regard peer-reviewed research as de facto sound science. If that were the case, Nature would hardly have subtitled a February 2016 article: “Mistakes in peer-reviewed papers are easy to find but hard to fix.” Over a period of 18 months, a team of researchers attempted to correct dozens of substantial errors in nutrition and obesity research. Among these was the claim that the height change in a group of adults averaged nearly three inches (7 cm) over eight weeks. The team reported that editors “seemed unprepared or ill-equipped to investigate, take action or even respond.”

In Kafkaesque fashion, after months of effort culminated in acknowledgement of a gaffe, journals then demanded that the team pay $1,700 in one instance and $2,100 in another before a letter calling attention to other people’s mistakes could be published.

Which brings us back to the matter of public policy. We’ve long been assured that reports produced by the UN’s Intergovernmental Panel on Climate Change (IPCC) are authoritative because they rely entirely on peer-reviewed, scientific literature. A 2010 InterAcademy Council investigation found this claim to be false, but that’s another story. Even if all IPCC source material did meet this threshold, the fact that one out of an estimated 25,000 academic journals conducted an unspecified and unregulated peer review ritual is no warranty that a paper isn’t total nonsense.

If half of the scientific literature “may simply be untrue,” then half of the climate research cited by the IPCC may also be untrue. This appalling unreliability extends to work on dietary cholesterol, domestic violence, air pollution – in short, to all research currently being generated by the academy.

The US National Science Foundation recently reminded us that a scientific finding “cannot be regarded as an empirical fact” unless it has been “independently verified.” Peer review does not perform that function. Until governments begin authenticating research prior to using it as the foundation for new laws and huge expenditures, don’t fall for the claim that policy X is evidence-based.

The Intergovernmental Panel on Climate Change has repeatedly and falsely claimed that it depends entirely on peer-reviewed papers. Donna Laframboise used volunteers to check this claim and found that a significant part of the references in the fourth assessment report of the IPCC were to ‘grey literature’ – that is, press releases, ‘reports’ from pressure groups and the like, which are not remotely the normal peer reviewed scientific literature.

Yet even if all the citations used by the IPCC were peer-reviewed, this would not mean they were infallible. Peer review is not, never was, and never can be a general protection against prejudice, error, or misconception about scientific matters. That it seems otherwise to some people is a misapprehension on their part, reflecting widespread myths about the reality of human investigations into the natural world.

It is startling for non-scientists who actually visit the sausage factory of science for the first time. There, peer review proves to be an often biased, prejudicial, and perfunctory process contrary in every respect to popular expectations about science. But scientists know that no increased regulation or standards can ever improve things, because there are no higher authorities to appeal to in a domain of human endeavour where no one knows or ever knew the answers – hence the name ‘peer review’ and not ‘expert correction’.

Donna Laframboise observes that, ‘There is a reason publishing insiders are among peer review’s most derisive critics. They know it’s mostly just a game. Everyone is pretending that all is well despite a mountain of evidence to the contrary.’ Most scientists grudgingly tolerate peer review because they cannot think of anything better. Experienced ones do not expect much from it, even if they must play along to succeed given modern customs (until about the mid-20th century it hardly existed).

Most scientists cringe when they hear other scientists claim that because their work is peer-reviewed, they do not have to respond to criticisms, even those from qualified colleagues, whether peer-reviewed or not. Some surely do make such claims: ‘…many academics insist that the research they present to the world has been fully vetted. Indeed, they often behave as though it meets a standard unrivalled elsewhere’, observes Laframboise.

Furthermore, those same scientists retreat to the truth about the state of human knowledge of Nature when facts go against their claims. She points out that: ‘On the other hand, they take no responsibility when information they’ve produced turns out to be mistaken. In such cases everyone is then reminded that scholarly publishing is vii really just an exchange of ideas.’ Few competent scientists regard current scientific thinking as anything more than provisional. It is always fully open to challenge.

Peer-review is also abused as a form of gatekeeping to defend orthodox ideas from challenge, as Laframboise says: ‘Alternative schools of thought are more likely to encounter scorn than a fair hearing, and the secretive nature of peer review provides ample cover for intolerance and tribalism. . .It places unconventional thinkers at the mercy of their more conventional colleagues. Indeed, this approach seems designed to extinguish – rather than nurture – the bold, original thinking that leads to scientific breakthroughs.’ Many unorthodox ideas prove to be wrong, but they are the lifeblood of scientific advance. They challenge our orthodoxies, either sharpening them or overthrowing them. Thus the notion of challenging the orthodox is accepted in science by necessity, even if grudgingly.

Gatekeeping against the unorthodox is not remotely a new problem. Oracular mediocrities down the centuries have doggedly resisted human advances in knowledge from Galileo to Semmelweis to Einstein, and thousands of other cases that only the most learned science historians will ever know. Spectacular scandals come and go, but science is in the end a long game of the generations, not something played out in news cycles. So why then has the public debate about the perfunctory, crony, gate-keeping aspects of peer review grown in volume in the media now? Partly it is because science has become a ‘bigger’ and more centralized endeavor, with massive budgets invested in conventional wisdom, and more politicians involved in pushing certain conclusions. How else can one comprehend the term ‘orchestrated’ used by one founder of the IPCC to describe how scientific opinion was designed to be treated for the use of policymakers?

It is clear that people who have never studied the history of science, or have never been on the unfashionable side of a scientific debate are in for a shock upon encountering this messy and sordid reality for the first time. Not least in for a shock is the media, which has been busy identifying heartbreaking science scandals in medicine, social science, neuroscience, and economics. But curiously they offer none from the subject of climate, despite it being one of the most policy driven and lavishly funded branches of science today.

Is this because there are few examples of bad practice, irreproducibility, retraction, pal review and gatekeeping in climatology? Far from it. The Climategate emails of 2009 revealed gatekeeping at its most blatant. Who can forget Phil Jones writing to Michael Mann on 8 July 2004 ‘can’t see either of these papers being in the next IPCC report. Kevin and I will keep them out somehow – even if we have to redefine what the peer-review literature is!’ Or Steve McIntyre and Ross McKitrick struggling to publish (leading to a US Congressional hearing, no less) their comprehensive demolition of the statistical errors and data-selection issues in the infamous ‘hockey stick’ paper? Or Richard Tol’s exposure of the practices employed in the Cook et al ‘97% viii consensus’ paper? Again and again, peer-reviewed climate papers have fallen apart under post-publication scrutiny from the likes of McIntyre, Willis Eschenbach, Donna Laframboise, Judith Curry and Nic Lewis. And these do not even touch on the challenge of independently reproducing climate model output without the machinery and resources necessary to do so, as Laframboise rightly observes in the following paper.

Indeed, the field of climate science could supply a rich harvest of examples of this crisis of scientific credibility all on its own. Yet it is the scandal that dare not speak its name. The discussions of the crisis in peer review in Nature, Science, the Economist and elsewhere studiously ignore any examples from climate science. Why is this? It is an article of faith among certain scientists and science journalists that because climate scepticism is also a position supported by those on the right of politics, so nobody in science must give fodder to the sceptics.

This is nothing less than the modern manifestation of gatekeeping continuing its ancient legacy, driven by sheer ignorance and self-delusion, to keep the forces that actually advance science away from the door. Scientific research stretches human faculties to their limits, and it is at such limits where human frailties become most prominent.

Humans are fallible. That is one of the greatest lessons from the history of science. The message to be taken from these heartbreaking scientific scandals and absurdities is not one of chagrin and a temptation to adopt cynicism. The true authors of such scandals are the laymen, academics, journalists, and policymakers who do not give a fair hearing to the many highly trained scientists motivated by alternative views who would put such dubious claims to the test. A pervasive uneducated appeal to science as a monolithic incomprehensible authority, assessed only in terms of moral purity rather than factual accuracy, has made such a fair hearing nearly impossible, and done great harm to science and us all.

The London-based Global Warming Policy Forum is a world leading think tank on global warming policy issues. The GWPF newsletter is prepared by Director Dr Benny Peiser - for more information, please visit the website atwww.thegwpf.com.

No comments:

Post a Comment

Thanks for engaging in the debate!

Because this is a public forum, we will only publish comments that are respectful and do NOT contain links to other sites. We appreciate your cooperation.

Please note - if you use the new REPLY button for comments, please start your comments AFTER the code. Also, the Blogger comment limit is 4,096 characters, so to post something longer, you may wish to use Part 1, Part 2 etc.

Subscribe To

Welcome to Breaking Views

Breaking Views brings you expert commentary on topical political and policy issues. The views expressed are those of the author alone. The blog is administered by the New Zealand Centre for Political Research, an independent public policy think tank at NZCPR.com - register for the free weekly NZCPR newsletterHERE.