Sir Humphrey and the Boehlert Questions

Yes, Minister was a wonderfully funny British comedy of a generation ago, (some info here and here or google yes minister humphrey) which featured Sir Humphrey Appleby as a senior civil servant (deputy minister) "managing" the Minister, his nominal boss, but who was always outwitted by Sir Humphrey. Each episode would start off with the Minister having some bright idea of instituting change, often involving a cut to civil service budgets. Sir Humphrey would resist and no budget would ever be cut. In a way, Sir Humphrey is descended from PG Wodehouse’s Jeeves. Here are some typical bon mots:

“It is axiomatic in government that hornets’ nests should be left unstirred, cans of worms should remain unopened, and cats should be left firmly in bags and not set among the pigeons. Ministers should also leave boats unrocked, nettles ungrasped, refrain from taking bulls by the horns, and resolutely turn their backs to the music.”

“Almost anything can be attacked as a failure, but almost anything can be defended as not a significant failure.”

Keep that thought in mind as we see how Sir Humphrey (Cicerone) dealt with the Boehlert questions in classic Yes, Minister style.

You’ll remember that, on June 23, 2005, the Barton Committee sent a letter to Mann (with similar letters to Bradley and Hughes) asking:

7c. Did you calculate the R2 statistic for the temperature reconstruction, particularly for the 15th Century proxy record calculations and what were the results?7d. What validation statistics did you calculate for the reconstruction prior to 1820, and what were the results?

On July 15, 2005, Cicerone, the President of the National Academy of Sciences, in best Sir Humphrey Appleby fashion, politely invited Barton to butt out, suggesting that NAS expert panels would be much better at resolving this sort of question, citing the r2 question in particular. On July 23, Mann himself told the Barton Committee:

My colleagues and I did not rely on this statistic in our assessments of “skill” (i.e., the reliability of a statistical model, based on the ability of a statistical model to match data not used in constructing the model) because, in our view, and in the view of other reputable scientists in the field, it is not an adequate measure of “skill.”

(Anticipating my report on Mann’s presentation a little, I’ll mention that one of the panelists asked him what the verification r2 statistic was for the 15th century. Mann said that they did not calculate the r2 statistic -" that would be silly.") Given that MBH98 mentioned the term “r2” on no fewer than 7 occasions and even has a map showing the verification r2 statistic for the AD1820 step, you’d expect his nose to grow when he said this, but that’s a story for another day.

Last July, the House Science Committee objected to the Energy and Commerce Committee intruding onto their turf and, later, with matters still unsettled, commissioned NAS to investigate the MBH controversy through a NAS expert panel. Their requisition included the following:

I am writing to ask you to empanel a balanced group of scientist to provide Congress with expert guidance on the current scientific consensus on the paleoclimate recod and particularly on the work of Drs. Michael Mann, Raymond Bradley and Malcolm Hughges (the so-called "hockey stick" thesis. The group should, in a clar and concise report issued in a relatively short period of time answer the following questions …

2) (b) What are the principal scientific criticisms of their [Mann, Bradley and Hughes] work and how significant are they? (c) Has the information needed to replicate their work been available? (d) Have other scientists been able to replicate their work?

Now let’s look at the actual terms of reference for the Panel posted in full here (this is a climateaudit link, I’ve seen the summary at other URLs, but I don’t have a third party link to the full Statement of Task handy):

SUMMARY
This study will describe and assess the state of scientific efforts to reconstruct surface temperature records for the Earth over approximately the past 1,000-2,000 years. The committee will summarize current scientific information on the temperature record for the past 1,000-2,000 years, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used and any problems with these approaches, and explain how central the debate over the paleoclimate temperature record is to the state of scientific knowledge on global climate change. The committee will address tasks such as identifying the variables for which proxy records have been employed, describing the proxy records that have been used to reconstruct surface temperature records for the pre-instrumental period, assessing the methods employed to combine multiple proxy data to develop surface temperature reconstructions, discussing the geographical regions over which proxy data can be reliably extrapolated, and evaluating the overall accuracy and precision of such reconstructions.

….
There has been significant controversy over the techniques used to make such projections. Thus, in late 2005 Chairman Sherwood Boehlert of the US House of Representatives Committee on Science requested that the National Academy of Sciences empanel a committee to provide Congress with expert guidance on the current scientific consensus on surface temperature reconstructions. His request highlighted the need to understand the current scientific consensus on the temperature record for the last 1,000-2,000 years, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used (including description of the scientific criticisms and how significant these criticisms are), and explain how central the debate over the paleoclimate temperature record is to the overall scientific consensus on global climate change.

Technical Context:
The most useful data for climate reconstructions are annually or seasonally resolved proxies, which are typically combined into “multiple proxy” assemblages that are then compared against long temporally overlapping instrumental records to calibrate and verify the climate signal in the data. Most of the controversy surrounding surface temperature reconstructions centers on the method by which these multiple proxy assemblages are created and calibrated. The reliability of certain proxy data sets has also been the subject of debate, as has the geographical extent over which a limited number of proxy data can be reliably extrapolated. An authoritative evaluation of the reliability of proxy data, the methodologies employed to create multiple proxy assemblages, and the overall uncertainties associated with surface temperature reconstructions would help resolve these controversies and delineate the appropriate role for paleoclimate indicators in evaluating global change.
PLAN OF ACTION

Statement of Task
The committee will describe and assess the state of scientific efforts to reconstruct surface temperature records for the Earth over approximately the past 1,000-2,000 years. The committee will summarize current scientific information on the temperature record for the past 1,000-2,000 years, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used and any problems with these approaches, and explain how central the debate over the paleoclimate temperature record is to the state of scientific knowledge on global climate change. As part of this effort, the committee will address tasks such as:

* Identify the variables for which proxy records have been employed (e.g., temperatures averaged over specific geographical areas and time periods, extreme temperatures).
* Describe the proxy records that have been used to reconstruct surface temperature records for the pre-instrumental period (e.g., tree rings, sediment cores, isotopes in water and ice, biological indicators, indicators from coral formations, geological boreholes, historical accounts) and evaluate their limitations.
* Assess the various methods employed to combine multiple proxy data to develop surface temperature reconstructions and the major assumptions associated with each approach.
* Discuss the geographical regions over which proxy data can be reliably extrapolated.
* Evaluate the overall accuracy and precision of such reconstructions.
….

the committee will summarize current scientific information on the temperature record for the past 1,000-2,000 years, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used and any problems with these approaches, and explain how central the debate over the paleoclimate temperature record is to the state of scientific knowledge on global climate change. The committee will address tasks such as identifying the variables for which proxy records have been employed, describing the proxy records that have been used to reconstruct surface temperature records for the pre-instrumental period, assessing the methods employed to combine multiple proxy data to develop surface temperature reconstructions, discussing the geographical regions over which proxy data can be reliably extrapolated, and evaluating the overall accuracy and precision of such reconstructions.

Towards the end of von Storch’s presentation, he posted up the Boehlert questions and started giving answers (as we were about to do as well.) It turns out that most, if not all, of the panelists had not seen the Boehlert questions. Von Storch’s introduction of the Boehlert questions prompted a discussion about whether these questions were within the scope of the panel’s mandate. Von Storch criticized MBH replicability giving a very categorical answer to one of the Boehlert questions that was identical to ours. So the panel has testimony on the matter. We also pointed out that presenters D’Arrigo and Hegerl had not archived their data and had refused to make it available as part of the IPCC review process. This sparked responses from both D’Arrigo and Hegerl purporting to justify this and concerns by the chairman about whether replication and data archiving was going off topic and distracting from the questions that they were charged with answering, even though these were obviously questions specifically asked by Boehlert.

Goldston, representing the House Science Committee, closed off the first day’s proceedings by observing as a public comment that the Science Committee realized that there were many large questions associated with climate change and recognized that there were many big and contentious issues still to come in the future. However, the Science Committee had intentionally asked some finite questions associated with current controversies, since they wanted to take at least a few small issues off the table.

Document Comparisons
1. Let’s first compare the characterization of the Boehlert questions in the terms of reference document to see if NAS has accurately represented the scope of the Boehlert questions in its Statement of Task. NAS said that Boehlert had asked NAS to "describe the principal methodologies used (including description of the scientific criticisms and how significant these criticisms are)" Do you think that this is an accurate representation of the Boehlert question: " 2) (b) What are the principal scientific criticisms of their [Mann, Bradley and Hughes] work and how significant are they? (c) Has the information needed to replicate their work been available? (d) Have other scientists been able to replicate their work?"? Note that NAS completely left out any specific mention of MBH, which was front and centre of the Boehlert questions, and left out the specific concern over replication.

2. Next let’s compare the NAS characterization of the Boehlert questions to the Statement of Task in the running text shown above. As noted above, NAS characterized the Boehlert questions as asking them to "describe the principal methodologies used (including description of the scientific criticisms and how significant these criticisms are)". The Statement of Task in the corresponding section uses the term "describe the principal methodologies used and any problems with these approaches". Again no specific mention of MBH or replication. Is there a difference between the terms used in each case? Is the change in wording accidental or intentional? If it was intentional, what is intended by the change in wording? In the specific list of tasks, there is again no mention of specific tasks related to MBH or replication.

3. The Boehlert letter was not attached to the panel’s Statement of Task. Was this intentional or accidental? Both von Storch and ourselves, independently assumed that the Boehlert questions were being addressed by the panel, while the panel itself didn’t seem to be aware of the specific Boehlert questions.

The manoeuvring over terms of reference is really quite delicious in Yes, Minister terms. The Minister (Committee) has dared to ask to ask a question, has attempted to grasp a nettle, perhaps even to open up a can of worms. There are lots of possibilities.

Now a panel that was so minded could obviously take the position that replication is fundamental to the scientific method and that criticisms about replicability are "scientific criticisms". Indeed, I intend to urge the Committee to use the Boehlert questions as a guide to interpreting the scope of what is involved in a "description of the scientific criticisms" as the lack of replicability is obviously pretty fundamental. However, they might use the language of Cicerone’s Statement of Task to simply avoid doing any of the inevitably unpleasant work of sorting through MBH and MBH replicability.

If the latter, one can see a variety of outcomes. If the NAS panel doesn’t answer the Boehlert questions, then it seems to me (and I’m just a fascinated spectator to the process) that Cicerone will have totally sandbagged the Science Committee in front of the Energy and Commerce Committee. The Science Committee will have asked straightforward questions and got Sir Humphreyed by Cicerone. They will look like the gang that can’t shoot straight. No wonder Goldston asked the panel to pay attention to the Boehlert questions.

Similarly, if the NAS panel doesn’t answer the Boehlert questions (and the Barton questions before them), the Science Committee will hardly be in any position to object if the Energy and Commerce Committee starts hearings – after all, the Science Committee wouldn’t have been able to get a few simple questions answered. Nor would Cicerone or NAS be able to object to the Energy and Commerce Committee opening hearings having asked for the opportunity to answer the Barton questions (r2 etc.) and then failing to answer either them or the Boehlert questions.

The situation for the panel is pretty complicated. Should they go along with Sir Humphrey and just duck the issue or should they ask Cicerone for a clarification of their task (referring to the fact that both von Storch and ourselves brought the Boehlert questions to their attention and even presented them with answers to the Boehlert questions)? Or should they proceed to answer the Boehlert questions on their own? For what it’s worth, when Mann presented on Friday, no one even asked him for the record the Boehlert question: Has the information needed to replicate [your] work been available?

For what it’s worth, when Mann presented on Friday, no one even asked him for the record the Boehlert question: Has the information needed to replicate [your] work been available?

That being the case, one would presume that they MUST answer, “The panel received evidence that such information is not available, which evidence was not rebutted.” As they have evidence from both Von Storch and M&M to say that this is the case and don’t have any response from Mann on the matter.

It’s the risk a lawyer always runs in deciding whether or not to cross-examine evidence on a particular point, or put forward a rebuttal witness – if you don’t, that evidence stands unchallenged and the only remaining question is to allocate weight to its existence in the overall case.

[Of course, that’s if they decide to address the point at all in their report. They could also Sir Humphrey this one, by substituting “…which evidence was not rebutted”, for “…however the panel has not verified this.” Both statements are true, with one being more informative than the other.]

“Each episode would start off with the Minister having some bright idea of instituting change, often involving a cut to civil service budgets. Sir Humphrey would resist and no budget would ever be cut.”

Minor correction. Occasionally the Minister got one over Humphrey, rare to be sure, but on occasion.

One of the greatest shows ever, good theme song too. I rented all of them and watched them in a row last summer, was great.

The main time I remember when Jim Hacker (the Minister) beat Sir Humphrey was when he made a public announcement of his plans before Humphrey had a chance to kill them. Sunlight is the best disinfectant …

Firstly, I should say that if you change the terms of reference of a committee in mid-task, you are asking for trouble. Certainly, those giving evidence could complain that they were asked to give evidence on stuff that was not relevant, and were unable to reply to issues that other people could address.

Secondly, I have some sympathy with the NAS’s ‘genericisation’ of the issues. There are all sorts of problems with conducting some sort of kangaroo court on named people; if you are goint to make findings about X, then there will need to be due process, charges, representation, etc., and this all starts to look suspiciously like the witch-hunt alleged last year. X will of course consult their lawyer, and withdraw from the process if they have half-a-braincell, and it all starts to look very one-sided and sticky. Part of this is that I am not sure how you could come to a conclusion about someone’s work, and not make a conclusion about that someone (e.g. X’s work is non-replicable and flawed, but there is no reflection on X!)

If there is an issue with the ‘genericisation’ of the task, it may be that there will be a political bust-up between NAS and the committees; not for me to say or even understand. However, if the committee asked NAS to do something which wasn’t really possible (see above), then there is no issue.

#9. Very sensible comments. I agree that you can’t change terms of reference, although you can always interpret them if they are not clear. I also appreciate your points that there could be valid reasons in terms of due process as to why replicability issues seemed to get shuffled off to the side.

Net of the Sir Humphrey metaphor, which may be more amusing than illuminating in this particular case, there were still two pretty interesting developments at the panel which I was trying to highlight:

1. the surprise of the panel members when they saw the actual Boehlert questions;
2. the intervention of Goldston, of the House Science Committee staff, pointing out that they actually wanted some finite questions answered.

If the questions are not ones that a NAS panel of this format can reasonably answer (and you’ve given some plausibile reasons), then it’s still a pretty unstable situation in lots of ways.

It’s a NAS panel; they don’t do due process, or issue charges. They do report findings that reflect on various Xs.

Also, ‘genericization’ (nice word, but I spell it with a z) could have been done on such topics as replicability, public archiving of data and methods, and validity of proxies; invite the various authors of several multiproxy concoctions, and publish lists of who were invited, and who showed up.

Looking for problems in the work of named Xs seems a popular sport if X is, for example, John Christy, or Roy Spencer.

This is interesting for a number of reasons. I can believe that there were non-sinister reasons to “soften” the wording of the Statement of Task, such as avoiding the appearance that the NAS was being used as a tool for an unfair, political, and personal attack on members of the Hockey Team. Some may argue that the panel’s final report will now be viewed as more “legitimate’. However, Steve makes a very good (and humorous) point in that removing the explicit questions also gives the appearance on an intentional “Sir Humphrey” dilution and deflection of the original purpose of the panel. The fact that Chairman North appeared to be hesitant to include Von Storch’s answers is evidence that the panel’s course had been deflected (at least temporarily). If this was the case, then it may have backfired for a number of reasons. 1) The panel now has the mandate to comment on the whole field of paleoclimate reconstructions. 2) The existence of the original questions is now obvious to the whole panel now, so they may very well answer them directly. 3) The report may look less political if it is severely critical of the paleoclimate reconstructions. The eyewitness reports so far suggest that there may have been a net gain from Sir Humphrey. Well done, Steve!

#12 – the issue that’s really going to grow legs from this panel is the failure of the tree proxies in the 2nd half of the 20th century to behave “right”. This is very significant and Sir Humphrey must have been very unhappy with how this went. I haven’t posted up on this yet, but I’ll get to it.

I am not overly familiar with the proxy being used herein to model this particular aspect of life. How would Sir Humphrey deal with upstarts who tried to circumvent their manipulations? I am more familiar with the “House of Cards” model which was/is somewhat darker.

#15 jae, having looked at Cicerone’s role and the NAS in general, it appears to me that despite the aim of the NAS, there is potential for conflict between the politics of Capital Hill and the internal politics of academia/academic research. I would have thought that the discrepency between the instructions from Boehlert and the ToR that the committee met under comes from having to be accountable to two masters. The precise questions that Boehlert requested would create the tensions as described by Per(#9). It makes the fallout from what Steve highlighted above even more watchable for if the NAS can’t fulfill their advisory function to Congress due to internal conflicts of interest then how does NAS justify their role?

“… if the NAS can’t fulfill their advisory function to Congress due to internal conflicts of interest then how does NAS justify their role?”

NAS’s role is well defined. You meant to ask “Of what use is the NAS if they can’t fulfill their advisory function due to internal conflicts of interest?” It kind of answers itself. Blissfully hypothetical at this point. They have a chance to yank it back on the road. When does the panel issue it’s report?

Whatever the politics, its a really bad move to not answer the specific questions of the guy who’s paying you to get them answered. I don’t think Boehlert will be pleased to receive a report without them.

I can imagine what I’d do if I was Boehlert and my specific questions were ignored, and Joe Barton laughing behind me: I’d hold the hearings myself and get my own answers, and I’d make Mann, Bradley, Hughes all testify together and I’d make sure that they stayed for the responses.

I’d do it before Joe Barton did it and made me look foolish and weak.

Just imagine: you have Michael Mann sitting there in front of you and you don’t ask whether he had released all details in order to enable complete replication or what his calculated verification statistics were, never mind what he thought of them.

#19. Mann was asked what the verification r2 statistic was. He said that he didn’t calculate this statistic – “that would be silly”. In my opinion, Nychka, a statistician, should have stepped in here, but didn’t. How could a statistician let this pass? BTW Mann did tell the committee “I am not a statistician”. Neither is Ammann or Wahl.

Uh, Mann doesn’t collect the data he uses in his analysis. I would think he would rely solely on being a statistician since that’s pretty much all he does: offer statistical analysis of data others have collected.

I love your use of Sir Humphrey, from the show “Yes, Minister,” to make your point. I loved him in that excellent, intellectually stimulating show. By bringing it up, a lot of good memories of Sir Humphrey and his Minister have been coming back to me. Thank you.

BTW, I am not a scientist, just an interested observer. I read about you, and Ross McKitrick, (and your troubles in getting the climate change scientists to behave like real scientists, accepting challenges to their hypotheses) in the article “Two Inconvenient Canadians: The Unlikely Men Who Shook Up Global-Warming Science” in National Review (February 8, 2010).

Keep up the good work! We need more real science in this debate — not the wishy-washy pseudo-religious mantra we frequently get from the “man caused this” Global Warming, Err, Uh, Climate Change crowd.

3 Trackbacks

[…] Similar questions may have been asked in 2006 but the National Academy of Sciences panel did not carry out an “in depth” investigation into whether Mann had “deviated from accepted practices within the academic community for proposing, conducting, or reporting research or other scholarly activities.” Ralph Cicerone of NAS, together with Gerry North, drew up terms of reference that specifically excluded such an investigation. This is discussed in CA post Sir Humphrey and the Boehlert Questions. […]