Two Cheers for Peer Review

I have been asked to write about ‘peer review’, the process through which decisions are reached about which research projects to fund, which people to choose, which papers to publish. It is a most human and familiar process: when in doubt, and the matter is serious enough, you ask someone who knows — indeed, you ask a few people who know, and consider all the advice you are getting. We do it all the time.

In the university world, and in the wider world of government policy, it is the basic decision-making tool about the use of knowledge. In the research world, it is a system. Every journal, every research-funding body, maintains a list of experts who are relied upon to give that advice. In selecting people for jobs and for scholarships, we rely on referees, some chosen by the candidate, others chosen by the appointing body. It is the same system. The ones we rely on are thought to know about the field and/or the candidate, and to give disinterested advice. The decision will still be made by the organisation, but it will have received a lot of comment about the proposal or the person.

It is an area I know very well. I was appointed to the Australian Research Grants Committee in 1981, and served on it, and its successor, the Australian Research Council (ARC), for ten years. As Chairman in turn of each body, I was able to visit other countries and learn about the ways they used peer review, and about the ways they came to decisions about whom to fund. I served for two terms on the Australian Science and Technology Council, and learned even more about how to evaluate scientific projects. When I retired as a vice-chancellor I did a lot of work, mostly in Canada, in the same areas, and helped to review the peer-review system of that country’s Social Sciences and Humanities Research Council. And of course I have been asked to serve as a peer reviewer myself, scores of times. You do it willingly, because it is part of the collaborative process that is at the heart of the university world.

The chief complaint about peer review comes from those who have not been funded. They will argue, if they are unhappy enough, that the assessors were biassed, unfair, protective of their own patch, ignorant or lazy. Those who run the systems are all too aware of the possibility that these objections might have some force to them, and do their best to winnow the judgments of the peers they have selected. In my day I was alert to what I called ‘positive and negative mafias’ — networks of peers who supported one another, or one another’s proteges, or one another’s research fields (the positive ones), or who gave faint support (if not outright scorn) in the case of other people, their work and their students.

I have been on the editorial boards of several journals, and acted as a reviewer myself of papers seeking publication. Much the same process applies. Someone has written a paper and sends it to what he or she thinks is the right journal. The editor sends it out for review to people who have published in the same area. The reviewers express an opinion, usually with comments on particular aspects of the paper. The editor finally comes to a decision. Positive and negative mafias are alive and well here too. Churchill once said that democracy was the worst form of government, apart from all the others that had been tried. I would say something comparable about peer review.

That’s a long introduction, and its point is how all this works in the area of climate science. My answer is that you can see all of its virtues and vices at work there. The ARC funds a large number of projects in this area, and from the titles it seems likely to me that all the researchers take for granted that human activity is responsible for global warming and that the outcomes will be serious if not catastrophic. That is the orthodoxy, and the proposals will have been sent to the orthodox. Indeed, there is hardly anyone else to send them too. Department of climate science are very new, and all seem to be colonised by those who support the orthodoxy. What you see here is the positive mafia at work. You will see the same in the journals. Paper after paper (I’ve seen a few) report on a findings of one kind or another, but manage to slip in a phrase or two about global warming. Why? The researchers hope that this will get them publication, and this is especially the case where their findings don’t actually support the AGW orthodoxy.

There is nothing unusual about any of this. It is characteristic of the world of knowledge. Where there is funding to gain, researchers quickly adapt to the rules of the game, and every research-funding body and every journal comes to realise that it is being used, and does its best to change the rules. But more, every discipline has its orthodoxy. In economics, markets rule. In the natural sciences, the current orthodoxy rules in each case. It is hard to get a paper published that confutes orthodoxy, or to start a line of enquiry that has that as it purpose, or to win a scholarship to do such work. That’s not the way the world operates, because peer review is a conservative system. When West and East Germany reunited after the fall of the Berlin Wall, several thousand East German academics lost their jobs. Why? They had been teaching Marxism/Leninism, which was the orthodoxy in East Germany, indeed compulsory for most students. But it had no place in the new united Germany.

The Climategate emails from the Climate Research Unit of the University of East Anglia showed in some detail how a few researchers tried to game the system, and indeed that they were by and large successful. To repeat, that was not unusual in itself. It was important in this case because the stakes are said to be so high, and governments are constructing expensive policies on the basis of what they understand to be the findings of climate science research.

Critics of the orthodoxy are often told, by supporters of the orthodoxy, to write their own papers and submit them to the journals: publication there will ensure that they will be taken notice of. And indeed some have done so, and found it extraordinarily difficult to get past the orthodoxy. Criticism, if it is honest and serious, requires no paper. It is enough to be able to point out the errors in what is already written. In general, journals don’t carry correspondence about past papers; they publish the new.

So I give only two cheers for peer review. It is familiar territory for me, and we cannot do without it. But it is human, and open to abuse. In the long run, science corrects itself, and I have no doubt that this will happen in climate science, which is a young discipline. Indeed, it is more of a quarry than a discipline, a quarry to which researchers in other, older disciplines, bring their tools, skills and interests. Nature will tell us, in time, whether we got AGW right. My current guess is that there is much more to know about climate than we know now, and the future is not so dark as the catastrophists claim. We shall see.

Share this:

Join the discussion 6 Comments

It is interesting to note that some of the scientists with an interest in climate science through their work in allied disciplines have a view of climate change which is not about whether or not in their opinion anthropogenic global warming will cause catastrophic climate change if it is not addressed by public policy.
Rather they have a view on the public policy position, for example the economic costs and benefits of action or where the social costs of action are most likely to be borne.These are, however, second order issues.
Perhaps journals could distinguish between these two important aspects of the debate when considering publication of papers?

Fortunately, the unorthodox have a much broader range of avenues for stating their case than were previously available – this blog being one example. Moreover, the interweb also allows for the ‘crowd-sourcing’ of peer review while reducing the need for restricting the number of articles being published (since there are no space limitations in an online-journal). I predict the ‘withering away’ of old-style peer review processes and look forward to a more democratic (and effective) open system of idea correction and improvement.

A NZ court has found against a band of climate “skeptics/agnostics” who bid to have the NZ temperature record declared invalid. Can they pay the $100,000 legal fee after losing the case? Probably not, leaving the taxpayer to cough up for this bit of nonsense.

Don, peer review isn’t the only place where the “it’s not warming” views fail to meet the test of science or common sense.

As you say, peer review isn’t perfect, and the truth will come out over time. Indeed, it seems to be coming out right now – melting Arctic sea ice is a strand of evidence that climate models have been too conservative.

[…] those who work in the same field. I don’t want to get into the world of peer review, here. I’ve done that recently, and there’s no point in repeating myself. How big is anyone’s field? When I was […]

[…] that aren’t always correct. I’ve made this point before about climate change, and about peer review itself. It is a necessary instrument in deciding whether or not a given research proposal should be […]