“Without oversight or challenge”

One of my long-standing concerns of Climate Audit and its readers has been a concern over the role of Eugene Wahl in changing the IPCC assessment of the McMc-Mann dispute in the Final Report – a role that Fred Pearce described in The Climate Files as a “subversion” of IPCC policies of openness and transparency. Jones’ delete-all-emails request was a direct request to erase all evidence of Wahl’s surreptitious contact with Briffa. Even this year, the University of East Anglia has continued to refuse FOI requests for Wahl’s actual proposed changes to IPCC AR4 text. The indolent Muir Russell inquiry doesn’t appear to have bothered obtaining and examining the critical attachments to Wahl’s emails, seeing no need to bother looking behind Briffa’s soothing reassurances on the matter.

For review, the assessment change directly affected Julia SLingo’s evidence to the SciTech Committee that the McMc criticisms of MBH did not have much impact – a point with which we disagree. This observation was based on the IPCC assessment which stated in the Final Report (quoted on page 80 of Muir Russell):

Wahl and Ammann (2006) also show that the impact on the amplitude of the final reconstruction is very small (~0.05°C; for further discussion of these issues see also Huybers, 2005; McIntyre and McKitrick, 2005c,d; von Storch and Zorita, 2005).

This was a substantial change in Wahl’s (and Mann’s) favor from the assessment in the Second Draft – the assessment sent to reviewers.

it is unclear whether it [the MM criticisms] has a marked impact upon the final reconstruction (Von Storch et al., 2004; Huybers, 2005; McIntyre and McKitrick, 2005).

No review comments on the record supported the changed assessment. Together with some puzzling content in the author responses to Review Comments, both David Holland and I surmised that there had been off-balance-sheet contact between Ammann and Wahl and Briffa that had not been archived with other review comments. (This becomes a long story.)

When the Climategate emails became available, it became clear that there had in fact been surreptitious contact between Wahl and Briffa – the emails evidenced that the correspondents were aware of the furtiveness of their exchange, the emails being marked “confidential”, were not copied to their usual cc list. Muir Russell didn’t bother asking Briffa why they were being so furtive.

One of the emails showed that Wahl had sent proposed changes to the assessment. Since reading the emails, it has been my opinion that Wahl drafted the language of the Final Draft that favored his assessment and that Briffa used Wahl’s amended language in the final assessment. While theoretically, there is joint responsibility for the final product, I very much doubted whether the Wahl-Briffa changes to the final assessment on this point were specifically reviewed by the overall committee at the time.

This raised the interesting question: could Wahl and Briffa between the two of them have inserted the changed assessment without anyone else in the world paying any attention prior to the publication of the final report?

In one of the documents at the Muir Russell website, this question is more or less asked of Chapter 6 Review Editor John Mitchell (whose untrue prevarications to refuse FOI requests have been discussed here in the past.) The question:

It would be helpful to understand the particular practices of the chapter 6 writing team as follows: …

3. to what extent would it be possible for individual team members to ensure inclusion of material in a report without oversight or challenge by other team members?

Good question in respect of the Wahl-Briffa exchange. Could it have been inserted in the Final Draft without oversight? (This is my opinion.)

Mitchell’s answer is as follows:

8. The palaeoclimate chapter depends on inputs from a wide variety of scientific fields. The early stage of assembly of evidence was done by individuals and groups of authors according to their discipline. It was then assembled into a first draft on which expert reviewers comments were received. The review editors were involved in plenary meetings prior to the second order and final drafts, where they ensured that reviewers comments had been properly dealt with.

9. All the authors would meet at least two or three times in each of the meetings I attended to discuss main issues in the chapter, the wording of the summary and cross chapter issues (I missed the beginning of the Bergen meeting). There were over a thousand comments in each review, and over sixty pages in the final chapter, so it was not practical or the best use of time for everyone to go over every part of the chapter line by line. (Nevertheless, I think after the first review, I think most of the authors went through most of the reviewers’ comments together at the beginning of the New Zealand meeting). The so-called “hockey stick” curve of Mann et al (1998) was widely discussed, as was the relevant wording. The executive summary was reviewed by the whole group to ensure that it accurately represented the evidence presented in the chapter, and it is the whole group that should be regarded as responsible, rather than any one person. [my bold]

10. In particular, it was accepted, including by Briffa, that the original Mann et al (98) paper had flaws relating to the particular use of principle components. Other papers either used Mann’s approach without using the flawed method of applying principal components, or used a different approach altogether, and it was on these papers that assessment of the last 1000 years was based. Many of the critical comments were directed at the Mann et al (98) paper which was acknowledged to be problematic. Nevertheless, before the second meeting, I noted the reviewers’ concerns to both the convening lead authors and the other review editor before the Bergen meeting to make sure they were aware of the issues being raised, and to ensure they were discussed in the wider group.

As you can see, Mitchell’s answer was mostly unresponsive to the question. The key phrase is the one that I bolded:

There were over a thousand comments in each review, and over sixty pages in the final chapter, so it was not practical or the best use of time for everyone to go over every part of the chapter line by line.

The answer, in short, is that the Review Editors did not in fact bother going over the chapter line by line because it wasn’t “practical”.

The implication for the Wahl insertion is quite clear. It was entirely possible for “individual team members to ensure inclusion of material in a report without oversight or challenge by other team members”.

Muir Russell
Now let’s look at Muir Russell’s coverage of this long-standing and central dispute – the dispute at the heart of the delete-all-emails request.

Muir Russell stated:

Wahl did not write any of the main text, though he did make some suggestions for very minor edits. p 82

From the perspective of the Hockey Stick dispute that was the most contentious issue of this chapter, Wahl’s suggestions were not “very minor”. They altered the assessment of the dispute from being “unclear” to being “very small”. This is not a “very minor” change.

The evidence of the Review Editor suggests that no one person in the writing team could have overridden the team responsibility for the text. (p. 83)

and on page 76:

24. Irrespective of the above comments on issues i) and ii), the evidence of the Review Editor underlines the team responsibility for the text, and the unlikelihood that a single voice could dominate on an important issue. p 76

In making this assessment, Muir Russell did not quote directly from Mitchell and his direct evidence (see here) not only does not support Muir Russell’s finding, but concedes that the final draft wasn’t gone over “line by line” and that it was entirely possible that Briffa could have inserted Wahl’s changed assessment into the IPCC report without “oversight or challenge by other team members”. (Whether or not the Lead Authors were or were not jointly “responsible” for the text is a different issue.)

In real world engineering projects, there is one (and only one) valid copy of a working document. All (and that means ALL) changes are versioned and logged with the identity of the person making the change and the reason for the cjange logged with it. It would be possible to reconstruct any version of the document with a simple command.

This process has been automated since the 1970s (at least). That is over 30 years ago. That this would be considered to be impractical is astonishing. How do they think that large projects operate without descending into chaos?

That is one of the key differences between real-world engineering projects and some of these other kinds of endeavors. In an engineering context, the entire focus is getting the engineering correct and knowing exactly what was done. In academia and in policy projects (like the IPCC) the focus is often on protecting pet opinions, soothing fragile egos, behind-the-scenes negotiations about how to frame a particular point, how strongly to word it, etc.

The process — and the resulting substance — are an entirely different kind of “science” than the hands-on, testable, repeatable, real-world applied science engineers are used to. I think this is part of the reason so many engineers are (perhaps rightly) astounded at the way climate science is carried out.

It doesn’t necessarily mean that climate science could be carried out any other way (after all, we’re talking about attempts to peer into the distant past and the distant future, rather than actual lab science), but it does mean that we should regard its results and conclusions with an appropriate amount of skepticism. In terms of certainty, accuracy, concreteness and so forth, so very much of climate science is not at all on the same footing as applied science and technology.

Good source control software for programming exists, and yet there are still plenty of people not using it, or managing to defeat the purpose of the system. It’s sadly true that many large projects do descend into chaos, even in the real world.

Good show, Steve. Your web site is akin to a Sherlock Holmes story, though the clues, issues, and protagonists are not fiction, but all too real. The fantastic story of the Hockey Team unfolds, and unravels, clue by clue, to amuse and enlighten the thinking world.

Collaboration software (even Microsoft WORD) will highlight any changes between drafts. Document management systems will version all drafts and indicate line by line changes. The issue of line by line checking being “not practical” would be something that would be very surprising to vendors of these commonplace systems. Changes would be linked to the reviewer comments that prompted them so that reveiers (even senior review editors) could see the import and effect of each and every comment in a direct and easy way, The massive spreadsheeet of reviewer comments that I have seen is primitive and bespeaks of an organization that is that does not understand technology.

This is a solved problem and the fact that something as important as the IPPC draft does not seem to have used a functional document management system is very surprising. Perhaps the climate scientists could have had one less meeting in Tahiti or Portugal and simple picked up the telephone and called the IT department at their universities. They would easily have supplied highly capable team collaboration system.

Absolutely agree. A document management system is vital for a process as complex as that of reviewing and amending the IPCC documents. There are plenty that can be bought off the shelf. No doubt none of those in academia have any understanding of real-world engineering processes.

Gee, when I write a paper with other authors, they send their suggested changes with “track changes” and I either accept them or not. IPCC had no central person doing this for each chapter/section? Really? Wasn’t this Mitchell’s job? Or could Briffa simply make changes to text with no oversight/approval? Guess so!

What’s truly incredible is that any scientist with a reputation to protect would say:

Wahl and Ammann (2006) also show that the impact on the amplitude of the final reconstruction is very small (~0.05°C …)

instead of the much more honest and open (though still inadequate)

it is unclear whether it [the MM criticisms] has a marked impact upon the final reconstruction

Then when this wording and process is challended the whole apparatus of IPCC (despite its code of conduct), UEA (despite its FOI policies), Royal Society (throwing Nullius in Verba out the window) and all the rest is deployed to maintain the deception.

After all, as Steve and everyone has said, the Hockey Stick has never been central to anthropogenic global warming theory. Why on earth did this strange group of people choose to close ranks to this degree? In software design we are taught to avoid a single point of failure at all costs. Why this absurd attempt to stand firm on something obviously flawed but otherwise so unimportant? (I know it had PR value in 2001 but really.) Who knows why. The select committee should give them a rocket. We’ll see what they manage.

There is a different possible future dynamic in the USA. If one of the houses of Congress were to be taken over by Republicans following an election next week, there could become political incentive for Republicans to hold public hearings to carefully review AGW views advanced by the other party.