Resolving technical issues in science

August 14th, 2009 by group

One of the strengths of science is its capacity to resolve controversies by generally accepted procedures and standards. Many scientific questions (especially more technical ones) are not matters of opinion but have a correct answer.

Scientists document their procedures and findings in the peer-reviewed literature in such a way that they can be double-checked and challenged by others. The proper way to challenge results is, of course, also through the peer-reviewed literature, so that the challenge follows the same standards of documentation as did the original finding.

Such a challenge can either be in form of a new, independent paper, or in the form of a comment to a published paper. The latter is the appropriate avenue if the challenge is not based on new data (and is thus a piece of research in its own right), but is a criticism of the methods used in a paper.

Such technical comments are routinely published in journals, and RealClimate authors have of course also been involved in writing or receiving such comments. One prominent example was a comment in Science showing that a challenge by Von Storch et al. (2004) to the “hockey stick” climate reconstruction of Mann et al. (1998) “was based on incorrect implementation of the reconstruction procedure”. We discussed the implications on Realclimate after the comment appeared. Another recent example was a comment by Schmith et al. on a Sciencepaper on sea level rise by Stefan, noting that he failed to account for the effect of smoothing on the autocorrelation in the data he used. In his response, Stefan acknowledged this mistake but showed that it does not affect his main conclusions.

That the original authors are allowed to respond to a comment in the same journal issue, and the comment’s authors get to consider this response before deciding to go ahead with their comment, are key hallmarks of a fair procedure, in addition to a neutral journal editor and independent reviewers overseeing the process. Even if the authors of comment and reply continue to disagree to some extent, this comment process in most cases resolves the issue to the satisfaction of the scientific community. It lays out the facts in a fair and transparent way and gives outsiders a good basis for judging whom is right. In this way it advances science.

There is however a different way of criticizing scientific papers that is prevalent in blogs like ClimateAudit. This involves challenging, ‘by all means necessary’, any paper whose conclusions are not liked. This can be based on simple typos, basic misunderstandings of the issues and ‘guilt by association’ though there is sometimes the occasional interesting point. Since these claims are rarely assessed to see if there is any actual impact on the main result, the outcome is a series of misleading critiques, regardless of whether any of these criticisms are in fact even valid or salient, that give the impression that every one of these papers is worthless and that all their authors incompetent at best and dishonest at worst. It is the equivalent of claiming to have found spelling errors in a newspaper article. Fun for a while, but basically irrelevant for understanding any issue or judging the worth of the journalist.

While commentary — even quite negative commentary — of papers on blogs is entirely reasonable (after all, we do it here occasionally), claims that a particular paper has been ‘discredited’ or ‘falsified’ that have not withstood (at minimum) the process of peer-review should be viewed with extreme skepticism. So should accusations of dishonesty or misconduct that have not already been conclusively and unequivocally substantiated.

This brings us to the recent claim by Hu McCulloch that a post on ClimateAudit.org, detailing an error in Steig et al’s paper in Nature on Antarctic temperature change, was not given due credit by Steig et al. when they published a Corrigendum earlier this month. In this case, McCulloch’s comment on the paper were perfectly valid, but he chose to avoid the context of normal scientific exchange — instead posting his comments on ClimateAudit.org — and then playing a game of ‘gotcha’ by claiming plagiarism when he wasn’t cited.