Friday, 22 February 2008

Peer Review Evil says Torygraph

The Torygraph newspaper, last bastion of conservative common sense - or whatever they're calling it these days - had an article the other day entitled "Peer review: the myth of the noble scientist" in which the Vice Chancellor of the University of Buckingham*, Terence Kealey, puts forward the myth that scientists are noble creatures - we're not - and suggests that the peer-review process is fatally flawed.

He begins with this little snippet of wisdom:

"One day coffee is bad for us, then it's good, then it's bad again. The generous explanation for these see-saws is that science is always developing our understanding. But there is a more sinister concern: fraud."

What? The reason that the public gets the impression that coffee or red wine or sweeteners are bad for us one week and geed the next is because the media insists on combing recent science papers for any slight suggestion of the above and publishing stories with headlines like "Shock News: Coffee Cures Alzheimers!" or similar. This idea that the scientific community is see-sawing back and forth is false. Published papers may conflict, but the truly scientific approach is to weigh all of the evidence and come to a conclusion, not wave your hands in the air and scream "fraud!"

It is true that fraud exists in the scientific community, as it does pretty much anywhere you find humans. One of the jobs of the peer-review process is to minimise the impact of fraudulent claims, even if it cannot directly test for them. Also, outside of peer-review, the scientific community is very good at picking up fraudulent claims. However, these processes take time - just as with everything else in the gradualist methodology of science - as Jan Hendrik Schön proved by getting more than 25 papers out and winning several awards before being picked up for manufacturing data on a monumental scale.

"The myth is that science is the noble search for truth. The reality is that scientists are selfish. In the old days, scientists often published secretly to safeguard - and profit from - their discoveries."

Science is the search for truth. It's nobility is surely dependent on the participants who are, as we are well aware, only human. Of course there are selfish scientists, fraudulent scientists, incompetent scientists and so on. They are ordinary people. However, their job requires them to be able to defend their work against critics and the community in which they work is strongly meritocratic, requiring a certain selfishness in order to survive. This said, there are many scientists who are perfectly capable of working in large groups and sharing information and I have benefitted from several willing to give up their time and energy to provide assistance that does not benefit their own research.

Kealey goes on to explain that peer-review creates a "closed club" that can block extraordinary or unexpected findings and allows "unscrupulous" reviewers to steal the ideas they read in review papers. I know this is only anecdotal but nobody in my office can remember an incident of a peer-reviewer stealing ideas from review papers. It would probably require a conspiracy of reviewers in order for that to occur.

Science may be somewhat of a closed club, but I see that as a requirement for it to maintain its credibility. Kealey says we should open up science publishing and allow everyone to publish their own papers but I think this would be horribly damaging to the edifice of science. Suddenly any old creationist nutjob could publish their papers about how God did it and the public (and the media) would have no way of telling how reliable the research was. We'd just be giving undue credibility to pseudoscientific nonsense.

The peer-review process may have flaws (it is slow and it may occasionally disallow "ground-breaking" papers) but it is about the best system we could have. It allows for some measure of fraud detection. It ensures that the wildest claims must be backed up with evidence. It may often swing towards the status quo but then extraordinary claims do require extraordinary evidence. Leakey should consider what science would be like if we removed peer-review: unrealiable and often meaningless.

*As I understand it, the University of Buckingham is a private university founded in the 1970s as an arts college. I cannot find an entry for it on the Times league tables available to me. It has the dubious honour of being "the only private University in the United Kingdom".