So does the public finally ‘get’ peer review?

How a 10-year collaboration is raising awareness of the role of evidence in policy, news and society.

We’ve come a long way in 10 years. When Sense about Science was set up in 2002 as a charity to help people make sense of science and evidence, the practices of scientific publishing seemed a strangely kept secret. It was a time when the front pages of many newspapers raged with headlines and horror stories about genetic modification, the MMR vaccine and nuclear waste disposal. Often, little attention was being paid to the quality or status of research findings, and in particular to whether they had been peer reviewed.

Meanwhile, when asked about peer review, rather than considering the public interest, scientists’ minds went either to the paper they had in the system or to the papers sitting threateningly on their desks awaiting review.

The early work of Sense about Science was to open up that process. We convened a working group of scientists in 2004 to look at public questions about science. They concluded in a report that greater effort was needed by research institutions, journals, publishers and others to share the workings of peer review in scholarly publishing. In response to their recommendations, Sense about Science produced a public guide to peer review – I don’t know what to believe – published in 2005 with help from Elsevier, Wiley, the Medical Research Council, the Royal Society of Biology, and the Royal Pharmaceutical Society. It explains the peer review process to the public as a tool for helping people make sense of science claims.

Many researchers and publishers were skeptical. For a start, they didn’t think the public would find the dry academic process of peer review interesting. And they also worried that promoting the value of peer review would promote false certainties. What about times when the system doesn’t work and cases of fraud, bias and misconduct get through? What if the public thinks that peer reviewed means true?

We argued that people can accept that mistakes exist while finding the general rule of thumb – that peer reviewed means scrutinized – useful. They can understand that one paper is not the end of the story and that conflicting views coexist and compete. They see this in the criminal justice system all the time, after all. Like peer review, the criminal justice system is not flawless; bias, delays and mistakes happen, and sometimes the wrong decisions are made. Most people understand those flaws and still see the value in the system.