A personal view on statistics in earth sciences

Menu

Why media reporting on discussion papers can lead to manipulated science

During this week discussions on twitter and on the blogs focused on a discussion paper by James Hansen et al in Earth System Dynamics Discussions. The paper forms part of a legal case in the US and basically states that the current and expected warming over the next decades is unprecedented since the last interglacial. In this context the Guardian has run an article on the paper. While they state that the science is not yet peer-reviewed, the authors have run a series of interviews and comments, which is usually happening only when a paper is actually published. As usual I refrain in this blog from commenting on climate politics, but as I have written my PhD on scientific publication processes I would like to focus here on the implications of the media scrutiny within the discussion paper phase of a scientific publication.

To understand the problematic of this action it is important to explain what a discussion paper actually is. Discussion journals came around a few years ago. They form part of an open-peer-review process, in which a manuscript is published before a peer-review took place. On this anyone is allowed to comment and two or more invited reviewers perform the actual reviews by publishing their reports openly (but anonymous) alongside the other comments. It is important to understand, that the discussion papers, even when they seem to be published in a (discussion) journal do not form part of the accepted published science. They are comparable with a manuscript under review and are actually in a process I have described in Quadt et al. 2012 as quality assurance. This step lie between the actual analysis and the peer review and it is suppose to enhance the quality of the paper. It has to be seen independent of the peer review, as this process incorporates a judgement on the scientific validity of the presented science on whether it is worth to be published in the specified journal.

The publication process (Quadt et al 2012)

So far this process is useful, it might deliver better paper, it helps in the transparency of the process and still the peer review process happens classically. Consequently, the resulting paper can be accepted formally as a scientific publication as it is established for centuries (even when the actual establishment of peer review (so the requirement to perform the process prior to an accepted scientific product) is just a hundred years old). But what happens now when this process is under massive public scrutiny, potentially under the influence of political interest groups? I will argue in the following that a non-standardised influence of the quality assurance on the peer review process can lead to manipulated science and so an endangering of the valid peer-review status of a resulting paper.

Why is it so important that the two processes, quality assurance and peer review are separated? As I had developed the terminology in context of data publication, it might be the best to illustrate it with data. So assume we have a dataset under review and additionally a quality assurance like quality evaluation is performed on it. Myself have developed schemes, which argued for the application of standardised tests in the QA process, also to help the reviewers in their tasks. But what happens if when the tests are manipulated? What if instead of the standardised testing, outsiders with personal interest change the parameters of the tests without the reviewer noticing it? Can a reviewer in this case perform a good quality assessment required for the review report? Probably not. Same is true within the discussion paper phase. A reviewer expect that the other comments are written by fellow scientists in the best interest. This is the assumption, which makes the process a standardised one. As soon as the comments do not reflect anymore this assumption the process becomes tricky. Reviewers might be guided by flawed comments, and the additional pressure due to the press reports can endanger the independence of the review reports (what happens if a reviewer does not reject a paper, because he/she feels moraly obliged by the press reports to accept it (or the other way round)). This is comparable to have a jury in a court case under public spotlight and before the verdict the jurors get the current tabloid press into their hand, including a rich comment section where everyone can communicate their believes. Would we trust the judgement?

Therefore, the danger is too hure that the final decision is flawed, even when I trust that reviewers act in the best interest of science. Media scrutiny has no place within the discussion paper process, it endangers science and poses the big question, whether the peer review process at this point should be concluded at all. In the past years we had to make new experiences with new forms of publication processes, but most, at least all accepted, are basing on classic principles. A flawed discussion paper phase, as it is possible in this case, breaks with this view. Manipulation is always possible, be it within peer review or any other step of the creation process of a scientific publication, but when it is encountered, usually the paper get retracted immediately. Here the manipulation of the process is not yet proven, but made possible by the actions of the media and those commenting publicly on the discussion paper. I am sure the journal and editor, as they have already announced in the discussion phase, will try their best to keep the discussion phase within limits. Anyway, there are guideleines for the discussion paper phase to minimise the risks and they should be followed:

do not cite unpublished work (and discussion papers are unpublished and uncitable work) as an important evidence

refrain from reporting on the process as long as the peer-review process is not yet finished

as a scientist do not comment on the content of the manuscript of the review apart from the comment section of the discussion paper

as author refrain from doing PR for the paper apart from generally inviting an audience to participate in discussion phase

Without these simple rules, the open peer-review is at the risk to be flawed and this might lead to manipulated science.

3 thoughts on “Why media reporting on discussion papers can lead to manipulated science”

I am not so sure that the publicity influences the peer review. Like you already wrote it could go both ways and I think that most reviewers being good scientists can control their biases reasonably. Public pressure on them would be limited as they can elect to be anonymous. They will also not see any comment by outsiders as valid and let that influence their judgement unduly.

My main worry would be that it misinforms the public. It is already bad enough that the public typically gets its science from reporting on single new studies, which are by definition more unreliable than established science. When the press now starts reporting on manuscripts that have not been vetted the quality of the science in mass media goes down even more.

A discussion paper is no more than a blog post or a email newsletter. Journalists should ignore it, like they would ignore a blog post.

You can also not forbid scientists from talking about science that is not peer reviewed (yet). Every month we see tweets with the latest global mean temperature estimate, often that makes the press. That is not reviewed, but not controversial and thus fine. What is controversial is something the experts can judge best and something you cannot make juridical rules on.

I do not know how controversial the present manuscript is. The previous manuscript clearly was and that Hansen send out a press release about it has hurt his reputation with me and I am less likely to work with him. Most small violations of community rules are “enforced” by such reputation impacts. This kind of problem would also be something to regulate by reputation in theory. The problem is that Hansen probably no longer cares that much about his reputation within science.

I do not agree here. From the view of scientific publishing blog posts are inexistent, they do not even qualify for grey-literature. They can be edited any time and can be quietly deleted. Therefore, they are more comparable with personal communication, which sometimes is cited by some articles. In contrast to this discussions paper do qualify for grey-literature, like published technical reports, master- and PhD-thesis. Still, like them, they shouldn’t be cited in reliable literature.

You can also not forbid scientists from talking about science that is not peer reviewed (yet). Every month we see tweets with the latest global mean temperature estimate, often that makes the press. That is not reviewed, but not controversial and thus fine. What is controversial is something the experts can judge best and something you cannot make juridical rules on.

As I set discussion papers equal with papers under review, it has to be treated similar in communications. It is not about not talking about it, but more like using a careful tone.

You can go to the Winnower to give your blog post a doi. The person citing it can make a copy at archive.is. Some journals do not allow citing grey literature. I spend more time on a manuscript than I do on a blog post. On the other hand some of the manuscripts I have had to review were of less quality than any blog post I have written. So, I do not see a fundamental difference.

At least from the perspective of a journalist and for the question whether you should write a press release, I feel blog posts and manuscripts are similar. The only reason a manuscript in a discussion journal is a bit published is to make sure that no one steals the ideas while the manuscript is being openly reviewed.

Thus I agree with you that we should talk about such manuscripts using a more careful tone. And we should educate journalists about their status. The weird thing of the Guardian article is that the author is an editor of the journal and was aware he was writing in a major newspaper about a mere manuscript.