The people -vs- the CRU: Freedom of information, my okole…

Foreword: Willis asked me to carry this post here. What follows is a long and detailed series of email exchanges that outline the difficult task of getting data so that scientific replication/reproduction can be done by people external to the tight knit group of scientists that make up climate science today. This is a must read for anyone trying to understand the issue and the dodges of the UK FOIA that CRU has been doing.

One of the foundational components of the scientific method is the idea of reproducibility (Popper 1959). In order for an experiment to be considered valid it must be replicated. This process begins with the scientists who originally performed the experiment publishing the details of the experiment. This description of the experiment is then read by another group of scientists who carry out the experiment, and ascertain whether the results of the new experiment are similar to the original experiment. If the results are similar enough then the experiment has been replicated. This process validates the fact that the experiment was not dependent on local conditions, and that the written description of the experiment satisfactorily records the knowledge gained through the experiment. From Rand and Wilensky 2006

CRU’s decision to withhold data and code from public inspection is not only against the scientific method, given the impact their work has on governmental policies and taxpayer funded programs, it is, in my opinion, unethical. – Anthony Watts

Guest post by Willis Eschenbach – originally posted on Omniclimate with an updated version here per Willis’ request.

UPDATED 11/24/09 8:30PM PST

People seem to be missing the real issue in the CRU emails. Gavin over at realclimate keeps distracting people by saying the issue is the scientists being nasty to each other, and what Trenberth said, and the Nature “trick”, and the like. Those are side trails. To me, the main issue is the frontal attack on the heart of science, which is transparency.

Science works by one person making a claim, and backing it up with the data and methods that they used to make the claim. Other scientists then attack the claim by (among other things) trying to replicate the first scientist’s work. If they can’t replicate it, it doesn’t stand. So blocking the FOIA allowed Phil Jones to claim that his temperature record (HadCRUT3) was valid science.

This is not just trivial gamesmanship, this is central to the very idea of scientific inquiry. This is an attack on the heart of science, by keeping people who disagree with you from ever checking your work and seeing if your math is correct.

As far as I know, I am the person who made the original Freedom Of Information Act to CRU that started getting all this stirred up. I was trying to get access to the taxpayer funded raw data out of which they built the global temperature record. I was not representing anybody, or trying to prove a point. I am not funded by Mobil, I’m an amateur scientist with a lifelong interest in the weather and climate. I’m not “directed” by anyone, I’m not a member of a right-wing conspiracy, or even right-wing. I’m just a guy trying to move science forwards.

The recent release of the hacked emails from CRU has provided an amazing insight into the attempt by myself, Steve McIntyre, and others from CA and elsewhere to obtain the raw station data from Phil Jones at the CRU. We wanted the data that was used to make the global temperature record that is relied on to claim “unprecedented” global warming. I want to give a chronological account of the interactions. While we don’t know if all of these emails are valid, the researchers involved such as Gavin Schmidt and Michael Mann have clearly indicated that they think they are authentic They certainly fit with my experience. I have only included the relevant parts of emails, and indicated where I have snipped by an ellipsis (…).

The story actually starts with Warwick Hughes, a climate researcher who had previously been in cordial contact with Phil Jones, the lead researcher of the CRU. I find only one email in the archive (0969308954) where Phil emails Warwick, from 2000. This is in response to some inconsistencies that Warwick had found in Phil’s work:

Warwick Hughes to Phil Jones, September ‘04:

Dear Phillip and Chris Folland (with your IPCC hat on),
Some days ago Chris I emailed to Tom Karl and you replied re the grid cells in north Siberia with no stations, yet carrying red circle grid point anomalies in the TAR Fig 2.9 global maps. I even sent a gif file map showing the grid cells barren of stations greyed out. You said this was due to interpolation and referred me to Phillip and procedures described in a submitted paper. In the last couple of days I have put up a page detailing shortcomings in your TAR Fig 2.9 maps in the north Siberian region, everything is specified there with diagrams and numbered grid points.

[1] One issue is that two of the interpolated grid cells have larger anomalies than the parent cells !!!!?????
This must be explained.

[2] Another serious issue is that obvious non-homogenous warming in Olenek and Verhojansk is being interpolated through to adjoining grid cells with no stations, like cancer.

[3] The third serious issue is that the urbanization affected trend from the Irkutsk grid cell neare Lake Baikal, looks to be interpolated into its western neighbour.

I am sure there are many other cases of this, 2 and 3 happening.
Best regards,
Warwick Hughes (I have sent this to CKF)

Phil to Warwick, same email:

Warwick,
I did not think I would get a chance today to look at the web page. I see what boxes you are referring to. The interpolation procedure cannot produce larger anomalies than neighbours (larger values in a single month). If you have found any of these I will investigate. If you are talking about larger trends then that is a different matter. Trends say in Fig 2.9 for the 1976-99 period require 16 years to have data and at least 10 months in each year. It is conceivable that at there are 24 years in this period that missing values in some boxes influence trend calculation. I would expect this to be random across the globe.

Warwick,
Been away. Just checked my program and the interpolation shouldn’t produce larger anomalies than the neighbouring cells. So can you send me the cells, months and year of the two cells you’ve found ? If I have this I can check to see what has happened and answer (1). As for (2) and (3) we compared all stations with neighbours and these two stations did not have problems when the work was done (around 1985/6). I am not around much for the next 3 weeks but will be here most of this week and will try to answer (1) if I get more details. If you have the names of stations that you’ve compared Olenek and Verhojansk with I would appreciate that.

Cheers
Phil

OK, so far we have a couple of scientists discussing issues in a scientific work, no problem. But as he found more inconsistencies, in order to understand what was going on, in 2005 Warwick asked Phil for the dataset that was used to create the CRU temperature record. Phil Jones famously replied:

Subject: Re: WMO non respondo
… Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it. …
Cheers Phil

Hmmm … not good. Or as they say in “1984”, double-plus ungood. Science can only progress if there is a free exchange of scientific data The scientific model works like this:

A scientist makes claims, and reveals the data and methods he used to come to his conclusions.

Other scientists who don’t agree attack the claim by (inter alia) seeing if they can replicate the result, using the first scientist’s data and methods.

If the claims cannot be replicated, the claim is adjudged to be false.

Obviously, if the data or the methods are kept secret, the claims cannot be verified. Attacking other scientist’s claims is what what scientists do. This adversarial system is the heart of science. Refusing scientific data because someone will attack it is an oxymoron, of course they will attack it. That’s what scientists do.

When I found out about this, I couldn’t believe it. I thought, a scientist can’t do that, can they? This is science, not hide and seek. So I wrote to the University of East Anglia (of which the CRU is a Department) on September 8, 2006, saying:

I would like to obtain a list of the meteorological stations used in the preparation of the HadCRUT3 global temperature average, and the raw data for those stations. I cannot find it anywhere on the web. The lead author for the temperature average is Dr. Phil Jones of the Climate Research Unit.

Many thanks, Willis Eschenbach

I got no response from Phil Jones or anyone at CRU or UEA. So I filed a Freedom of Information act request for the data.

Now at this point, let me diverge to what was happening at CRU during this time. The first reference to Freedom of Information in their emails is from 2005, before they had received a single request. Immediately, they start to plan how to evade requests should some come in:

Tom Wigley, Former Director CRU, to Phil Jones, 21/01/2005

Phil,
…

I got a brochure on the FOI Act from UEA. Does this mean that, if someone asks for a computer program we have to give it out?? Can you check this for me (and Sarah). …
Thanks,
Tom.

Phil replies to Tom:

Tom,
…
On the FOI Act there is a little leaflet we have all been sent. It doesn’t really clarify what we might have to do re programs or data. Like all things in Britain we will only find out when the first person or organization asks. I wouldn’t tell anybody about the FOI Act in Britain. I don’t think UEA really knows what’s involved.

As you’re no longer an employee I would use this argument if anything comes along. I think it is supposed to mainly apply to issues of personal information – references for jobs etc.

..
Cheers
Phil

So the coverup starts immediately, even before the first request. “I wouldn’t tell anyone about the FOI act in Britain”.

Tom to Phil

Phil,

Thanks for the quick reply. The leaflet appeared so general, but it was prepared by UEA so they may have simplified things. From their wording, computer code would be covered by the FOIA. My concern was if Sarah is/was still employed by UEA. I guess she could claim that she had only written one tenth of the code and release every tenth line.
…
Tom

You can see how they plan to observe the spirit of the FOI Act. Claim a temporary employee isn’t really an employee so they are not covered.

Phil to Tom

Tom,
…
As for FOIA Sarah isn’t technically employed by UEA and she will likely be paid by Manchester Metropolitan University. I wouldn’t worry about the code. If FOIA does ever get used by anyone, there is also IPR to consider as well. Data is covered by all the agreements we sign with people, so I will be hiding behind them. I’ll be passing any requests onto the person at UEA who has been given a post to deal with them.
Cheers
Phil

Phil Jones has just gotten the news that FOI will apply, and immediately he starts to plan how he is going to hide from an FOI request. Cite technicalities, claim IPR rights, those are good hiding places.

The next email (1109021312) is later in 2005:

At 09:41 AM 2/2/2005, Phil Jones wrote to Michael Mann :

Mike,
…
Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites – you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? – our does ! The UK works on precedents, so the first request will test it.

We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it – thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who’ll say we must adhere to it !

….

Phil

So now we have two more ways for Phil to hide from the FOI Act … along with a threat to delete the data rather than release it. Astounding. And this is before they’ve even received a single FOI request.

Mann replies to Jones:

Thanks Phil,

Yes, we’ve learned out lesson about FTP. We’re going to be very careful in the future what gets put there. Scott really screwed up big time when he established that directory so that Tim could access the data.

Yeah, there is a freedom of information act in the U.S., and the contrarians are going to try to use it for all its worth. But there are also intellectual property rights issues, so it isn’t clear how these sorts of things will play out ultimately in the U.S….
mike

Next, from February 05. Jones to Mann, cc to Hughes and Bradley (co-authors of the “hockeystick” study)

Leave it to you to delete as appropriate !
Cheers
Phil
PS I’m getting hassled by a couple of people to release the CRU station temperature data. Don’t any of you three tell anybody that the UK has a Freedom of Information Act !

The first rule of the Freedom of Information act … nobody talks about the Freedom of Information Act.

With that as a prologue, let me return to my FOI request. On February 10, 2007, I received my reply from Mr. Dave Palmer of CRU:

Dear Mr. Eschenbach

FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04)

Your request for information received on 28 September now been considered and I can report that the information requested is available on non-UEA websites as detailed below.

The Global Historical Climatology Network (GHCN-Monthly) page within US National Climate Data Centre website provides one of the two US versions of the global dataset and includes raw station data. This site is at: http://www.ncdc. noaa.gov/ oa/climate/ ghcn-monthly/ index.php

This page is where you can get one of the two US versions of the global dataset, and it appears that the raw station data can be obtained from this site.

Datasets named ds564.0 and ds570.0 can be found at The Climate & Global Dynamics Division (CGD) page of the Earth and Sun Systems Laboratory (ESSL) at the National Center for Atmospheric Research (NCAR) site at: http://www.cgd. ucar.edu/ cas/tn404/

Between them, these two datasets have the data which the UEA Climate Research Unit (CRU) uses to derive the HadCRUT3 analysis. The latter, NCAR site holds the raw station data (including temperature, but other variables as well). The GHCN would give their set of station data (with adjustments for all the numerous problems).

They both have a lot more data than the CRU have (in simple station number counts), but the extra are almost entirely within the USA. We have sent all our data to GHCN, so they do, in fact, possess all our data.

In accordance with S. 17 of the Freedom of Information Act 2000 this letter acts as a Refusal Notice, and the reasons for exemption are as stated below

Exemption Reason
s. 21, Information accessible to applicant via other means Some information is publicly available on external websites

I was outraged. So the next day, I made a second request:

Dear Mr. Palmer:

Thank you for your reply (attached below). However, I fear that it is totally unresponsive. I had asked for a list of the sites actually used. While it may (or may not) be true that “it appears that the raw station data can be obtained from [GHCN]“, this is meaningless without an actual list of the sites that Dr. Jones and his team used.

The debate about changes in the climate is quite important. Dr. Jones’ work is one of the most frequently cited statistics in the field. Dr. Jones has refused to provide a list of the sites used for his work, and as such, it cannot be replicated. Replication is central to science. I find Dr. Jones attitude quite difficult to understand, and I find your refusal to provide the data requested quite baffling.

You are making the rather curious claim that because the data “appears” to be out on the web somewhere, there is no need for Dr. Jones to reveal which stations were actually used. The claim is even more baffling since you say that the original data used by CRU is available at the GHCN web site, and then follow that with the statement that some of the GHCN data originally came from CRU. Which is the case? Did CRU get the data from GHCN, or did GHCN get the data from CRU?

Rather than immediately appealing this ruling (with the consequent negative publicity that would inevitably accrue to CRU from such an action), I am again requesting that you provide:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and

2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.

I find it somewhat disquieting that an FOI request is necessary to force a scientist to reveal the data used in his publicly funded research … is this truly the standard that the CRU is promulgating?

Thank you for your cooperation in this matter.

Willis Eschenbach

Hey, I was trying to be a nice guy, not make a public scene. On April 12, 2007, I got my second reply:

In regards the “gridded network” stations, I have been informed that the Climate Research Unit’s (CRU) monthly mean surface temperature dataset has been constructed principally from data available on the two websites identified in my letter of 12 March 2007. Our estimate is that more than 98% of the CRU data are on these sites.

The remaining 2% of data that is not in the websites consists of data CRU has collected from National Met Services (NMSs) in many countries of the world. In gaining access to these NMS data, we have signed agreements with many NMSs not to pass on the raw station data, but the NMSs concerned are happy for us to use the data in our gridding, and these station data are included in our gridded products, which are available from the CRU web site. These NMS-supplied data may only form a very small percentage of the database, but we have to respect their wishes and therefore this information would be exempt from disclosure under FOIA pursuant to s.41. The World Meteorological Organization has a list of all NMSs.

That didn’t help one bit. Without knowing which data was used, it was meaningless. They’ve tried s.21, they’ve tried s.41, neither exemption applies. So the next day, I replied:

While it is good to know that the data is available at those two web sites, that information is useless without a list of stations used by Jones et al. to prepare the HadCRUT3 dataset. As I said in my request, I am asking for:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and

2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.

Without knowing the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service), it is not possible to access the information. Thus, Exemption 21 does not apply – I still cannot access the data.

I don’t understand why this is so hard. All I am asking for is a simple list of the sites and where each site’s data is located. Pointing at two huge piles of data and saying, in effect, “The data is in there somewhere” does not help at all.

To clarify what I am requesting, I am only asking for a list of the stations used in HadCRUT3, a list that would look like this:

etc. for all of the stations used to prepare the HadCRUT3 temperature data.

That is the information requested, and it is not available “on non-UEA websites”, or anywhere else that I have been able to find.

I appreciate all of your assistance in this matter, and I trust we can get it resolved satisfactorily.

Best regards,

I received another letter, saying that they could not identify the locations of the requested information. I wrote back again, saying:

Dear Mr. Palmer:

It appears we have gone full circle here, and ended up back where we started.

I had originally asked for the raw station data used to produce the HadCRUT3 dataset to be posted up on the UEA website, or made available in some other form.

You refused, saying that the information was available elsewhere on non-UEA websites, which is a valid reason for FOI refusals.

I can report that the information requested is not available on non-UEA websites as detailed below.

Your most recent letter (Further _information_ letter_final_ 070418_rev01. doc), however, says that you are unable to identify the locations of the requested information. Thus, the original reason for refusing to provide station data for HadCRUT3 was invalid.

Therefore, since the information requested is not available on non-UEA websites, I wish to re-instate my original request, that the information itself be made available on your website or in some other form. I understand that a small amount of this data (about 2%, according to your letter) is not available due to privacy requests from the countries involved. In that case, a listing of which stations this applies to will suffice.

The HadCRUT3 dataset is one of the fundamental datasets in the current climate discussion. As such, it is vitally important that it can be peer reviewed and examined to verify its accuracy. The only way this can be done is for the data to be made available to other researchers in the field.

Once again, thank you for your assistance in all of this. It is truly not a difficult request, and is fully in line with both standard scientific practice and your “CODE OF PRACTICE FOR RESPONDING TO REQUESTS FOR INFORMATION UNDER THE FREEDOM OF INFORMATION ACT 2000″. I am sure that we can bring this to a satisfactory resolution without involving appeals or unfavorable publicity.

Further to your email of 14 April 2007 in which you re-stated your request to see

“a list of stations used by Jones et al. to prepare the HadCRUT3 dataset” I am asking for: 1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and 2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.”

In your note you also requested “the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service)”,

I have contacted Dr. Jones and can update you on our efforts to resolve this matter.

We cannot produce a simple list with this format and with the information you described in your note of 14 April. Firstly, we do not have a list consisting solely of the sites we currently use. Our list is larger, as it includes data not used due to incomplete reference periods, for example. Additionally, even if we were able to create such a list we would not be able to link the sites with sources of data. The station database has evolved over time and the Climate Research Unit was not able to keep multiple versions of it as stations were added, amended and deleted. This was a consequence of a lack of data storage in the 1980s and early 1990s compared to what we have at our disposal currently. It is also likely that quite a few stations consist of a mixture of sources.

I have also been informed that, as the GHCN and NCAR are merely databases, the ultimate source of all data is the respective NMS in the country where the station is located. Even GHCN and NCAR can’t say with precision where they got their data from as the data comes not only from each NMS, but also comes from scientists in each reporting country.

In short, we simply don’t have what you are requesting. The only true source would be the NMS for each reporting country. We can, however, send a list of all stations used, but without sources. This would include locations, names and lengths of record, although the latter are no guide as to the completeness of the series.

This is, in effect, our final attempt to resolve this matter informally. If this response is not to your satisfaction, I will initiate the second stage of our internal complaint process and will advise you of progress and outcome as appropriate. For your information, the complaint process is within our Code of Practice and can be found at: http://www1. uea.ac.uk/ polopoly_ fs/1.2750! uea_manual_ draft_04b. pdf

Yours sincerely David Palmer Information Policy Officer University of East Anglia

I loved the story line in this one “we do not have a list consisting solely of the sites we currently use”. Say what? How do they produce updates that change the temperature all the way back to 1870 if they don’t have the data or a list of the sites? But I digress …

So I advised him that I was appealing. His letter was passed to a Ms. Kitty Inglis, who replied

Following David Palmer’s letter of 27th April 2007 to you regarding your dissatisfaction with our response to your FOI request of 25th January 2007, I have undertaken a thorough review of the contents of our file and have spoken with both Mr. Palmer and Professor Jones.

As a result of this investigation, I am satisfied that we have done all we can to fulfil [sic] your request and to provide you with the information you require where it is possible for us to do so.

I confirm that we are able to make available on the Climatic Research Unit website a list of stations, including name, latitude, longitude, elevation and WMO number (where available).

We are unable to provide a simple list of sources for these stations as we do not hold this information. Nor do we hold the raw (i.e. unadjusted) station data, as you describe it, at UEA. As stated in prior letters to you, raw station data are available on the NCAR and GHCN websites and gridded data are available on the Climatic Research Unit website. If these data are insufficient for your requirements, you will need to contact the NMS for the country in which the station is located to obtain the information you require.

I hope you are able to accept this response. We have contacted the Information Commissioner’s Office in relation to this matter and their advice is that if you are still dissatisfied with this response, you can, at this time, exercise your right of appeal to the
Information Commissioner by contacting them at:
Information Commissioner’ s Office
Wycliffe House

At that point, I let it go. I had a small victory, we got a list of the stations. Of course, it took me a couple more letters to actually get them to post the list. But I got nothing else of what I had requested, and the list was full of all kinds of errors.

Meanwhile, behind the scenes at CRU, I now find out that they were circling the wagons … what follows are their internal discussions about a series of FOI requests from myself, Steve McIntyre, Doug Keenan and others to CRU for various data. Phil Jones to Tom Keenan and Wei-Chyung Wang, 6/19/2007:

Wei-Chyung and Tom,
…
1. Think I’ve managed to persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.
2. Had an email from David Jones of BMRC, Melbourne. [EMAIL NOT FOUND IN CRU EMAILS – Willis] He said they are ignoring anybody who has dealings with CA, as there are threads on it about Australian sites.
3. CA is in dispute with IPCC (Susan Solomon and Martin Manning) about the availability of the responses to reviewer’s at the various stages of the AR4 drafts. They are most interested here re Ch 6 on paleo.
Cheers
Phil

Well, that explains a few things … they’ve managed to “persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.” I hadn’t noticed that exemption in the FOI documentation I’d seen. Call me crazy, but I don’t think that’s in FOI Exemptions, I doubt if it’s legal, and it definitely isn’t ethical. I note that they are circling the wagons in Australia as well … this is followed by:

Phil Jones to Thomas Peterson of NOAA, 6/20/2007 AM (1182342470) :

Tom P.

Just for interest. Don’t pass on.

Might be a precedent for your paper to J. Climate when it comes out. There are a few interesting comments on the CA web site. One says it is up to me to prove the paper from 1990 was correct, not for Keenan to prove we’re wrong. Interesting logic.
Cheers
Phil

Wei-Chyung, Tom,
I won’t be replying to either of the emails below [FROM STEVE MCINTYRE AND DOUG KEENAN], nor to any
of the accusations on the Climate Audit website. I’ve sent them on to someone here at UEA to see if we
should be discussing anything with our legal staff. The second letter seems an attempt to be nice to me,
and somehow split up the original author team. I do now wish I’d never sent them the data after their FOIA
request!

Cheers
Phil

He obviously views sending data in response to an FOIA request as optional.

Thomas Peterson to Jones, same email:

Fascinating. Thanks for keeping me in the loop, Phil. I won’t pass it on but I will keep it in the back of my mind when/if Russ asks about appropriate responses to CA requests. Russ’ view is that you can never satisfy them so why bother to try?

Again, responding to an FOIA request is viewed as optional.

Phil Jones :

…

PS to Gavin – been following (sporadically) the CA stuff about the GISS data and release of the code etc by Jim. May take some of the pressure off you soon, by releasing a list of the stations we use – just a list, no code and no data. Have agreed to under the FOIA here in the UK.

Oh Happy days!

So I see … that’s why I only got the station list and not the data, just to ” take some of the pressure off “. Thanks, Phil.

Jones to Bradley and Amman, 5/9/08 (1210341221):

Mike, Ray, Caspar,

A couple of things – don’t pass on either.
…
2. You can delete this attachment if you want. Keep this quiet also, but this is the person [DAVID HOLLAND – Willis] who is putting in FOI requests for all emails Keith and Tim have written and received re Ch 6 of AR4. We think we’ve found a way around this.

Finding ways around FOI requests seems to be a popular sport at CRU. This is in reference to Steve trying to get the review comments to Chapter 6 of the UN IPCC Fourth Assessment Report.

Next, here’s the brilliant way that they had found around the FOIA, a bombshell of an idea, Jones to Michael Mann, 29 May 2008 (1212063122):

Mike,

Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment – minor family crisis.

Can you also email Gene and get him to do the same? I don’t have his new email address.

We will be getting Caspar to do likewise.

…

Cheers
Phil

Again, call me crazy, but deleting evidence in the face of an FOI request must be illegal. Gene is Eugene Wahl. Of course, what these guys don’t realize is that there are multiple copies of most emails floating around. In some ways, I hope they deleted them, so that it can be proven. The story continues …

Have a look at Climate Audit. Holland has put all the responses and letters up. There are three threads – two beginning with Fortress and a third later one. Worth saving the comments on a Jim Edwards – can you do this Tim?

I’ve saved all three threads as they now stand. No time to read all the comments, but I did note in “Fortress Met Office” that someone has provided a link to a website that helps you to submit FOI requests to UK public institutions, and subsequently someone has made a further FOI request to Met Office and someone else made one to DEFRA. If it turns into an organised campaign designed more to inconvenience us than to obtain useful information, then we may be able to decline all related requests without spending ages on considering
them. Worth looking out for evidence of such an organised campaign.

Tim

Another thing to hide behind, a false claim of an “organised campaign”. I loved the “Confidential Agent Amman” …

Gavin,
…
Keith/Tim still getting FOI requests as well as MOHC and Reading. All our FOI officers have been in discussions and are now using the same exceptions not to respond – advice they got from the Information Commissioner. As an aside and just between us, it seems that Brian Hoskins has withdrawn himself from the WG1 Lead nominations. It seems he doesn’t want to have to deal with this hassle.

The FOI line we’re all using is this. IPCC is exempt from any countries FOI – the skeptics have been told this. Even though we (MOHC, CRU/UEA) possibly hold relevant info the IPCC is not part our remit (mission statement, aims etc) therefore we don’t have an obligation to pass it on.

Cheers
Phil

So now the Information Commissioner is in on the deal, s/he’s advising them to use the same exceptions not to respond. No need to think about it, all of the wheels have been greased.

My personal opinion is that both FOI requests (1) and (2) are intrusive and unreasonable. Steven McIntyre provides absolutely no scientific justification or explanation for such requests. I believe that McIntyre is pursuing a calculated strategy to divert my attention and focus away from research. As the recent experiences of Mike Mann and Phil Jones have shown, this request is the thin edge of wedge. It will be followed by further requests for computer programs, additional material and explanations, etc., etc.

Quite frankly, Tom, having spent nearly 10 months of my life addressing the serious scientific flaws in the Douglass et al. IJoC paper, I am unwilling to waste more of my time fulfilling the intrusive and frivolous requests of Steven McIntyre. The supreme irony is that Mr. McIntyre has focused his attention on our IJoC paper rather than the Douglass et al. IJoC paper which we criticized. As you know, Douglass et al. relied on a seriously flawed statistical test, and reached incorrect conclusions on the basis of that flawed test.

I believe that our community should no longer tolerate the behavior of Mr. McIntyre and his cronies. McIntyre has no interest in improving our scientific understanding of the nature and causes of climate change. He has no interest in rational scientific discourse. He deals in the currency of threats and intimidation. We should be able to conduct our scientific research without constant fear of an “audit” by Steven McIntyre; without having to weigh every word we write in every email we send to our scientific colleagues.

In my opinion, Steven McIntyre is the self-appointed Joe McCarthy of climate science. I am unwilling to submit to this McCarthy-style investigation of my scientific research. As you know, I have refused to send McIntyre the “derived” model data he requests, since all of the primary model data necessary to replicate our results are freely available to him. I will continue to refuse such data requests in the future. Nor will I provide McIntyre with computer programs, email correspondence, etc. I feel very strongly about these issues. We should not be coerced by the scientific equivalent of a playground bully.

I will be consulting LLNL’s Legal Affairs Office in order to determine how the DOE and LLNL should respond to any FOI requests that we receive from McIntyre. I assume that such requests will be forthcoming.

I am copying this email to all co-authors of our 2008 IJoC paper, to my immediate superior at PCMDI (Dave Bader), to Anjuli Bamzai at DOE headquarters, and to Professor Glenn McGregor (the editor who was in
charge of our paper at IJoC).

I’d be very happy to discuss these issues with you tomorrow. I’m sorry that the tone of this letter is so formal, Tom. Unfortunately, after today’s events, I must assume that any email I write to you may be subject to FOI requests, and could ultimately appear on McIntyre’s “ClimateAudit” website.

With best personal wishes,

Ben

Well, he got the last paragraph right, at least. He also thinks that an FOIA request must serve some “scientific justification”, with the justification determined by … well … by the person receiving the request, of course. Another previously unknown part of the FOI Exemptions comes to light.

Ben Santer to Tom Wigly, 12 Dec 07 (1228330629):

At 01:17 03/12/2008, Ben Santer wrote:

Dear Tom,
…
One of the problems is that I’m caught in a real Catch-22 situation. At present, I’m damned and publicly vilified because I refused to provide McIntyre with the data he requested. But had I acceded to McIntyre’s initial request for climate model data, I’m convinced (based on the past experiences of Mike Mann, Phil, and Gavin) that I would have spent years of my scientific career dealing with demands for further explanations, additional data, Fortran code, etc. (Phil has been complying with FOIA requests from McIntyre and his cronies for over two years). And if I ever denied a single request for further information, McIntyre would have rubbed his hands gleefully and written: “You see – he’s guilty as charged!” on his website.

You and I have spent over a decade of our scientific careers on the MSU issue, Tom. During much of that time, we’ve had to do science in “reactive mode”, responding to the latest outrageous claims and inept science by John Christy, David Douglass, or S. Fred Singer. For the remainder of my scientific career, I’d like to dictate my own research agenda. I don’t want that agenda driven by the constant need to respond to Christy, Douglass, and Singer. And I certainly don’t want to spend years of my life interacting
with the likes of Steven McIntyre.

I hope LLNL management will provide me with their full support. If they do not, I’m fully prepared to seek employment elsewhere.

With best regards,
Ben

Dr. Santer, here’s a novel idea. Put enough information out when you publish the work so that your work can be replicated. Put on the web whatever is necessary in the way of code, data, and methods to allow your work to be checked by someone else. If you do that, not only will you not be bothered, but you will be following the scientific method. None of us at ClimateAudit are doing this to harass anyone, as you claim. We’re doing this because we cannot replicate your work, and thus your work is purely anecdotal rather than scientific.

Phil responds (same email):

Cc: mann , Gavin Schmidt, Karl Taylor, peter gleckler

Ben,
When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions – one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school – the head of school and a few others) became very supportive. I’ve got to know the FOI person quite well and the Chief Librarian – who deals with appeals. The VC is also aware of what is going on – at least for one of the requests, but probably doesn’t know the number we’re dealing with. We are in double figures.

One issue is that these requests aren’t that widely known within the School. So I don’t know who else at UEA may be getting them. CRU is moving up the ladder of requests at UEA though – we’re way behind computing though. We’re away [aware?]of requests going to others in the UK – MOHC, Reading, DEFRA and Imperial College.

So spelling out all the detail to the LLNL management should be the first thing you do. I hope that Dave is being supportive at PCMDI. The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian, saying that the email maligned his scientific credibility with his peers!

If he pays 10 pounds (which he hasn’t yet) I am supposed to go through my emails and he can get anything I’ve written about him. About 2 months ago I deleted loads of emails, so have very little – if anything at all. This legislation is different from the FOI – it is supposed to be used to find put why you might have a poor credit rating !

In response to FOI and EIR requests, we’ve put up some data – mainly paleo data. Each request generally leads to more – to explain what we’ve put up. Every time, so far, that hasn’t led to anything being added – instead just statements saying read what is in the papers and what is on the web site! Tim Osborn sent one such response (via the FOI person) earlier this week. We’ve never sent programs, any codes and manuals.

In the UK, the Research Assessment Exercise results will be out in 2 weeks time. These are expensive to produce and take too much time, so from next year we’ll be moving onto a metric based system. The metrics will be # and amounts of grants, papers and citations etc. I did flippantly suggest that the # of FOI requests you get should be another.

When you look at CA, they only look papers from a handful of people. They will start on another coming out in The Holocene early next year. Gavin and Mike are on this with loads of others. I’ve told both exactly what will appear on CA once they get access to it!

Cheers
Phil

Well, that explains why David Palmer and Ms. Kitty Inglis, the Chief Librarian, were so unsupportive. Took a couple of half hour sessions, but at the end of that, rather than being a representative of the FOI process, they were functioning as the personal representatives of Phil Jones. We have a new reason I hadn’t noticed in the FOI law for refusing a request, because the requester posts at CA.

And since they have the FOI person, and the FOI Appeals person, and the Information Commissioner in their pockets, and they have the standard terms of refusal figured out … just how difficult can it be to deny an FOI Request?

Jones to Ben Santer again, 10 Dec 2008:

Ben,

Haven’t got a reply from the FOI person here at UEA. So I’m not entirely confident the numbers are correct. One way of checking would be to look on CA, but I’m not doing that. I did get an email from the FOI person here early yesterday to tell me I shouldn’t be deleting emails – unless this was ‘normal’ deleting to keep emails manageable! McIntyre hasn’t paid his £10, so nothing looks likely to happen re his Data Protection Act email.

Anyway requests have been of three types – observational data, paleo data and who made IPCC changes and why. Keith has got all the latter – and there have been at least 4. We made Susan aware of these – all came from David Holland. According to the FOI Commissioner’ s Office, IPCC is an international organization, so is above any national FOI. Even if UEA holds anything about IPCC, we are not obliged to pass it on, unless it has anything to do with our core business – and it doesn’t! I’m sounding like Sir Humphrey here! McIntyre often gets others to do the requesting, but requests and responses all get posted up on CA regardless of who sends them.

On observational data, there have been at least 5 including a couple from McIntyre. Others here came from Eschenbach and also Douglas Keenan. The latter relate to Wei-Chyung Wang, and despite his being exonerated by SUNY, Keenan has not changed his web site since being told the result by SUNY!

The paleo data requests have all been to Keith, and here Tim and Keith reply. The recent couple have come from McIntyre but there have been at least two others from Holland. So since Feb 2007, CRU is in double figures. We never get any thanks for putting things up – only abuse and threats. The latest lot is up in the last 3-4 threads on CA.

I got this email over the weekend – see end of this email. This relates to what Tim sent back late last week. There was another one as well – a chatty one saying why didn’t I respond to keep these people on CA quiet. I’ve ignored both. Finally, I know that DEFRA receive Parliamentary Questions from MPs to answer. One of these 2 months ago was from a Tory MP asking how much money DEFRA has given to CRU over the last 5 years. DEFRA replied that they don’t give money – they award grants based on open competition. DEFRA’s system also told them there were no awards to CRU, as when we do get something it is down as UEA!

Since he and Mann and the others have already deleted their emails, looks like David Palmer (the “FOI person”) was a bit too late with his excellent advice … however, I did get a “Mentioned In Dispatches” from Phil, at least …

I also like the sly way he tells Ben how to illegally delete emails, just do it as part of ‘normal’ deleting to keep emails manageable! Yeah, right, that’s the ticket.

Phil Jones to Raymond Pierrehumbert, 16 Jan 09 (1200493432):

Cc: Michael Mann , Gavin Schmidt

Ray,
…

I have had a couple of exchanges with Courtillot. This is the last of them from March 26, 2007. I sent him a number of papers to read. He seems incapable of grasping the concept of spatial degrees of freedom, and how this number can change according to timescale. I also told him where he can get station data at NCDC and GISS (as I took a decision ages ago not to release our station data, mainly because of McIntyre). I told him all this as well when we met at a meeting of the French Academy in early March.
…
Cheers, Phil

This is a very clear statement of what he has done. He has refused to release the data, not because there is any logical reason to do so, but “because of McIntyre”. This is shameful, and the fact that the FOI people, Dave Peters and Kitty Inglis and the Information Commissioner, went along with this is dereliction of duty.

“Thanks” Ben for this, hi all and happy new year. I had a similar experience– but not FOIA since we at Climatic Change are a private institution- -with Stephen McIntyre demanding that I have the Mann et al cohort publish all their computer codes for papers published in Climatic Change I put the question to the editorial board who debated it for weeks. The vast majority opinion was that scientists should give enough information on their data sources and methods so others who are scientifically capable can do their own brand of replication work, but that this does not extend to personal computer codes with all their undocumented sub routines etc. It would be odious requirement to have scientists document every line of code so outsiders could then just apply them instantly. Not only is this an intellectual property issue, but it would dramatically reduce our productivity since we are not in the business of producing software products for general consumption and have no resources to do so. The NSF, which funded the studies I published, concurred–so that ended that issue with Climatic Change at the time a few years ago.

This continuing pattern of harassment, as Ben rightly puts it in my opinion, in the name of due diligence is in my view an attempt to create a fishing expedition to find minor glitches or unexplained bits of code–which exist in nearly all our kinds of complex work–and then assert that the entire result is thus suspect. Our best way to deal with this issue of replication is to have multiple independent author teams, with their own codes and data sets, publishing independent work on the same topics–like has been done on the “hockey stick”. That is how credible scientific replication should proceed.

Let the lawyers figure this out, but be sure that, like Ben is doing now, you disclose the maximum reasonable amount of information so competent scientists can do replication work, but short of publishing undocumented personalized codes etc. The end of the email Ben attached shows their intent–to discredit papers so they have no “evidentiary value in public policy”–what you resort to when you can’t win the intellectual battle scientifically at IPCC or NAS.

Good luck with this, and expect more of it as we get closer to international climate policy actions, We are witnessing the “contrarian battle of the bulge” now, and expect that all weapons will be used.

Cheers, Steve

PS Please do not copy or forward this email.

Now, why would Dr. Schneider not want his email copied or forwarded … perhaps because he is saying don’t follow the spirit of the Freedom of Information Act, don’t release the code that shows the math and reveals how you got your results? He foolishly thinks that studies can be “replicated” by using different data and different codes … but that says absolutely nothing about the original study and whether it contains any mistakes. The only way to determine whether a study like a historical temperature reconstruction contains errors is to examine the scientists actual work You can’t just pick another different bunch of proxies, analyze them, and say “I’ve found mathematical errors in your reconstruction”. You can only find those errors if you examine the actual math the researcher used, and to do that you need access to “their” codes.

I put “their” codes, in quotes because, under the policies of the University of East Anglia (and many other Universities), the codes do not belong to Phil Jones. They were developed as a part of his employment, and as such they belong to the University, and not to Phil.

The researchers complain in various places that they do not want to reveal their “primary data” because it is available on the web. While this is often true, as I saw in my FOIA requests to CRU, it is not sufficient Just saying “I got the information from Website X” as CRU did is often totally inadequate to locate the data in question. Santer makes this charge, that anyone could go the CMIP website and get the data themselves … but unless he says exactly which data from which run of which model, the website address is meaningless.

The main impression that I get from the emails is that the various scientists think that I and other requesters are simply doing this to harass them. Nothing could be further from the truth. I respect actual scientists, I’m short of time myself so I understand time pressures, so I have no desire to put any scientist to any extra effort beyond providing what science requires – a full accounting of the data, the methods, and in some cases the computer code used to do the research. Anything more is harassment … but anything less is scientific obstruction. And if they would provide those things when they publish their results, they’d never hear from me. And if Nature Magazine and Science Magazine and the National Science Foundation and all of the journals and funders would just enforce their own existing rules on archiving and transparency, the problem would be solved. But noooo, for the select friends of Phil these bothersome transparency regulations are ignored and overlooked.

As I said, the issue is not Trenberth or the Nature “trick” or scientists talking smack about each other. It is the illegal evasion of legitimate scientific requests for data needed to replicate a scientific study. Without replication, science cannot move forwards. And when you only give data to friends of yours, and not to people who actually might take a critical look at it, care to guess what you might end up with?

People seem to be missing the real issue in the CRU emails. Gavin over at realclimate keeps distracting people by saying the issue is the scientists being nasty to each other, and what Trenberth said, and the Nature “trick”, and the like. Those are side trails. To me, the main issue is the frontal attack on the heart of science, which is transparency.Science works by one person making a claim, and backing it up with the data and methods that they used to make the claim. Other scientists then attack the claim by (among other things) trying to replicate the first scientist’s work. If they can’t replicate it, it doesn’t stand. So blocking the FOIA allowed Phil Jones to claim that his temperature record (HadCRUT3) was valid science.This is not just trivial gamesmanship, this is central to the very idea of scientific inquiry. This is an attack on the heart of science, by keeping people who disagree with you from ever checking your work and seeing if your math is correct.As far as I know, I am the person who made the original Freedom Of Information Act to CRU that started getting all this stirred up. I was trying to get access to the taxpayer funded raw data out of which they built the global temperature record. I was not representing anybody, or trying to prove a point. I am not funded by Mobil, I’m an amateur scientist with a lifelong interest in the weather and climate. I’m not “directed” by anyone, I’m not a member of a right-wing conspiracy. I’m just a guy trying to move science forwards.The recent release of the hacked emails from CRU has provided me with an amazing insight into the attempt by myself, Steve McIntyre, and others from CA and elsewhere to obtain the raw station data from Phil Jones at the CRU. We wanted the data that was used to make the global temperature record that is relied on to claim “unprecedented” global warming. I want to give a chronological account of the interactions. While we don’t know if all of these emails are valid, the researchers involved such as Gavin Schmidt and Michael Mann that clearly indicate that they think they are authentic They certainly fit with my experience. I have only included the relevant parts of emails, and indicated where I have snipped by an ellipsis (…).The story actually starts with Warwick Hughes, a climate researcher who had previously been in cordial contact with Phil Jones, the lead researcher of the CRU. I find only one email in the archive (0969308954) where Phil emails Warwick, from 2000. This is in response to some inconsistencies that Warwick had found in Phil’s work:

Warwick Hughes to Phil Jones, September ‘04:

Dear Phillip and Chris Folland (with your IPCC hat on),
Some days ago Chris I emailed to Tom Karl and you replied re the grid cells in north Siberia with no stations, yet carrying red circle grid point anomalies in the TAR Fig 2.9 global maps. I even sent a gif file map showing the grid cells barren of stations greyed out. You said this was due to interpolation and referred me to Phillip and procedures described in a submitted paper. In the last couple of days I have put up a page detailing shortcomings in your TAR Fig 2.9 maps in the north Siberian region, everything is specified there with diagrams and numbered grid points.

[1] One issue is that two of the interpolated grid cells have larger anomalies than the parent cells !!!!?????
This must be explained.

[2] Another serious issue is that obvious non-homogenous warming in Olenek and Verhojansk is being interpolated through to adjoining grid cells with no stations, like cancer.

[3] The third serious issue is that the urbanization affected trend from the Irkutsk grid cell neare Lake Baikal, looks to be interpolated into its western neighbour.

I am sure there are many other cases of this, 2 and 3 happening.
Best regards,
Warwick Hughes (I have sent this to CKF)

Phil to Warwick, same email:

Warwick,
I did not think I would get a chance today to look at the web page. I see what boxes you are referring to. The interpolation procedure cannot produce larger anomalies than neighbours (larger values in a single month). If you have found any of these I will investigate. If you are talking about larger trends then that is a different matter. Trends say in Fig 2.9 for the 1976-99 period require 16 years to have data and at least 10 months in each year. It is conceivable that at there are 24 years in this period that missing values in some boxes influence trend calculation. I would expect this to be random across the globe.

Warwick,
Been away. Just checked my program and the interpolation shouldn’t produce larger anomalies than the neighbouring cells. So can you send me the cells, months and year of the two cells you’ve found ? If I have this I can check to see what has happened and answer (1). As for (2) and (3) we compared all stations with neighbours and these two stations did not have problems when the work was done (around 1985/6). I am not around much for the next 3 weeks but will be here most of this week and will try to answer (1) if I get more details. If you have the names of stations that you’ve compared Olenek and Verhojansk with I would appreciate that.

Cheers
Phil

OK, so far we have a couple of scientists discussing issues in a scientific work, no problem. But as he found more inconsistencies, in order to understand what was going on, in 2005 Warwick asked Phil for the dataset that was used to create the CRU temperature record. Phil Jones famously replied:

Subject: Re: WMO non respondo
… Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it. …
Cheers Phil

Hmmm … not good. Or as they say in “1984”, double-plus ungood. Science can only progress if there is a free exchange of scientific data The scientific model works like this:

A scientist makes claims, and reveals the data and methods he used to come to his conclusions.

Other scientists who don’t agree attack the claim by (inter alia) seeing if they can replicate the result, using the first scientist’s data and methods.

If the claims cannot be replicated, the claim is adjudged to be false.

Obviously, if the data or the methods are kept secret, the claims cannot be verified. Attacking other scientist’s claims is what what scientists do. This adversarial system is the heart of science. Refusing scientific data because someone will attack it is an oxymoron, of course they will attack it. That’s what scientists do.

When I found out about this, I couldn’t believe it. I thought, a scientist can’t do that, can they? This is science, not hide and seek. So I wrote to the University of East Anglia (of which the CRU is a Department) on September 8, 2006, saying:

I would like to obtain a list of the meteorological stations used in the preparation of the HadCRUT3 global temperature average, and the raw data for those stations. I cannot find it anywhere on the web. The lead author for the temperature average is Dr. Phil Jones of the Climate Research Unit.

Many thanks, Willis Eschenbach

I got no response from Phil Jones or anyone at CRU or UEA. So I filed a Freedom of Information act request for the data.

Now at this point, let me diverge to what was happening at CRU during this time. The first reference to Freedom of Information in their emails is from 2005, before they had received a single request. Immediately, they start to plan how to evade requests should some come in:

Tom Wigley, Former Director CRU, to Phil Jones, 21/01/2005

Phil,
…

I got a brochure on the FOI Act from UEA. Does this mean that, if someone asks for a computer program we have to give it out?? Can you check this for me (and Sarah). …
Thanks,
Tom.

Phil replies to Tom:

Tom,
…
On the FOI Act there is a little leaflet we have all been sent. It doesn’t really clarify what we might have to do re programs or data. Like all things in Britain we will only find out when the first person or organization asks. I wouldn’t tell anybody about the FOI Act in Britain. I don’t think UEA really knows what’s involved.

As you’re no longer an employee I would use this argument if anything comes along. I think it is supposed to mainly apply to issues of personal information – references for jobs etc.

..
Cheers
Phil

So the coverup starts immediately, even before the first request. “I wouldn’t tell anyone about the FOI act in Britain”.

Tom to Phil

Phil,

Thanks for the quick reply. The leaflet appeared so general, but it was prepared by UEA so they may have simplified things. From their wording, computer code would be covered by the FOIA. My concern was if Sarah is/was still employed by UEA. I guess she could claim that she had only written one tenth of the code and release every tenth line.
…
Tom

You can see how they plan to observe the spirit of the FOI Act. Claim a temporary employee isn’t really and employee so they are not covered.

Phil to Tom

Tom,
…
As for FOIA Sarah isn’t technically employed by UEA and she will likely be paid by Manchester Metropolitan University. I wouldn’t worry about the code. If FOIA does ever get used by anyone, there is also IPR to consider as well. Data is covered by all the agreements we sign with people, so I will be hiding behind them. I’ll be passing any requests onto the person at UEA who has been given a post to deal with them.
Cheers
Phil

Phil Jones has just gotten the news that FOI will apply, and immediately he starts to plan how he is going to hide from an FOI request. Cite technicalities, claim IPR rights, those are good hiding places.

The next email (1109021312) is later in 2005:

At 09:41 AM 2/2/2005, Phil Jones wrote to Michael Mann :

Mike,
…
Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites – you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? – our does ! The UK works on precedents, so the first request will test it.

We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it – thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who’ll say we must adhere to it !

….

Phil

So now we have two more ways for Phil to hide from the FOI Act … along with a threat to delete the data rather than release it. Astounding. And this is before they’ve even received a single FOI request.

Mann replies to Jones:

Thanks Phil,

Yes, we’ve learned out lesson about FTP. We’re going to be very careful in the future what gets put there. Scott really screwed up big time when he established that directory so that Tim could access the data.

Yeah, there is a freedom of information act in the U.S., and the contrarians are going to try to use it for all its worth. But there are also intellectual property rights issues, so it isn’t clear how these sorts of things will play out ultimately in the U.S….
mike

Next, from February 05. Jones to Mann, cc to Hughes and Bradley (co-authors of the “hockeystick” study)

Leave it to you to delete as appropriate !
Cheers
Phil
PS I’m getting hassled by a couple of people to release the CRU station temperature data. Don’t any of you three tell anybody that the UK has a Freedom of Information Act !

The first rule of the Freedom of Information act … nobody talks about the Freedom of Information Act.

With that as a prologue, let me return to my FOI request. On February 10, 2007, I received my reply from Mr. Dave Palmer of CRU:

Dear Mr. Eschenbach

FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04)

Your request for information received on 28 September now been considered and I can report that the information requested is available on non-UEA websites as detailed below.

The Global Historical Climatology Network (GHCN-Monthly) page within US National Climate Data Centre website provides one of the two US versions of the global dataset and includes raw station data. This site is at: http://www.ncdc. noaa.gov/ oa/climate/ ghcn-monthly/ index.php

This page is where you can get one of the two US versions of the global dataset, and it appears that the raw station data can be obtained from this site.

Datasets named ds564.0 and ds570.0 can be found at The Climate & Global Dynamics Division (CGD) page of the Earth and Sun Systems Laboratory (ESSL) at the National Center for Atmospheric Research (NCAR) site at: http://www.cgd. ucar.edu/ cas/tn404/

Between them, these two datasets have the data which the UEA Climate Research Unit (CRU) uses to derive the HadCRUT3 analysis. The latter, NCAR site holds the raw station data (including temperature, but other variables as well). The GHCN would give their set of station data (with adjustments for all the numerous problems).

They both have a lot more data than the CRU have (in simple station number counts), but the extra are almost entirely within the USA. We have sent all our data to GHCN, so they do, in fact, possess all our data.

In accordance with S. 17 of the Freedom of Information Act 2000 this letter acts as a Refusal Notice, and the reasons for exemption are as stated below

Exemption Reason
s. 21, Information accessible to applicant via other means Some information is publicly available on external websites

I was outraged. So the next day, I made a second request:

Dear Mr. Palmer:

Thank you for your reply (attached below). However, I fear that it is totally unresponsive. I had asked for a list of the sites actually used. While it may (or may not) be true that “it appears that the raw station data can be obtained from [GHCN]“, this is meaningless without an actual list of the sites that Dr. Jones and his team used.

The debate about changes in the climate is quite important. Dr. Jones’ work is one of the most frequently cited statistics in the field. Dr. Jones has refused to provide a list of the sites used for his work, and as such, it cannot be replicated. Replication is central to science. I find Dr. Jones attitude quite difficult to understand, and I find your refusal to provide the data requested quite baffling.

You are making the rather curious claim that because the data “appears” to be out on the web somewhere, there is no need for Dr. Jones to reveal which stations were actually used. The claim is even more baffling since you say that the original data used by CRU is available at the GHCN web site, and then follow that with the statement that some of the GHCN data originally came from CRU. Which is the case? Did CRU get the data from GHCN, or did GHCN get the data from CRU?

Rather than immediately appealing this ruling (with the consequent negative publicity that would inevitably accrue to CRU from such an action), I am again requesting that you provide:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and

2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.

I find it somewhat disquieting that an FOI request is necessary to force a scientist to reveal the data used in his publicly funded research … is this truly the standard that the CRU is promulgating?

Thank you for your cooperation in this matter.

Willis Eschenbach

Hey, I was trying to be a nice guy, not make a public scene. On April 12, 2007, I got my second reply:

In regards the “gridded network” stations, I have been informed that the Climate Research Unit’s (CRU) monthly mean surface temperature dataset has been constructed principally from data available on the two websites identified in my letter of 12 March 2007. Our estimate is that more than 98% of the CRU data are on these sites.

The remaining 2% of data that is not in the websites consists of data CRU has collected from National Met Services (NMSs) in many countries of the world. In gaining access to these NMS data, we have signed agreements with many NMSs not to pass on the raw station data, but the NMSs concerned are happy for us to use the data in our gridding, and these station data are included in our gridded products, which are available from the CRU web site. These NMS-supplied data may only form a very small percentage of the database, but we have to respect their wishes and therefore this information would be exempt from disclosure under FOIA pursuant to s.41. The World Meteorological Organization has a list of all NMSs.

That didn’t help one bit. Without knowing which data was used, it was meaningless. They’ve tried s.21, they’ve tried s.41, neither exemption applies. So the next day, I replied:

While it is good to know that the data is available at those two web sites, that information is useless without a list of stations used by Jones et al. to prepare the HadCRUT3 dataset. As I said in my request, I am asking for:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and

2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.

Without knowing the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service), it is not possible to access the information. Thus, Exemption 21 does not apply – I still cannot access the data.

I don’t understand why this is so hard. All I am asking for is a simple list of the sites and where each site’s data is located. Pointing at two huge piles of data and saying, in effect, “The data is in there somewhere” does not help at all.

To clarify what I am requesting, I am only asking for a list of the stations used in HadCRUT3, a list that would look like this:

etc. for all of the stations used to prepare the HadCRUT3 temperature data.

That is the information requested, and it is not available “on non-UEA websites”, or anywhere else that I have been able to find.

I appreciate all of your assistance in this matter, and I trust we can get it resolved satisfactorily.

Best regards,

I received another letter, saying that they could not identify the locations of the requested information. I wrote back again, saying:

Dear Mr. Palmer:

It appears we have gone full circle here, and ended up back where we started.

I had originally asked for the raw station data used to produce the HadCRUT3 dataset to be posted up on the UEA website, or made available in some other form.

You refused, saying that the information was available elsewhere on non-UEA websites, which is a valid reason for FOI refusals.

I can report that the information requested is not available on non-UEA websites as detailed below.

Your most recent letter (Further _information_ letter_final_ 070418_rev01. doc), however, says that you are unable to identify the locations of the requested information. Thus, the original reason for refusing to provide station data for HadCRUT3 was invalid.

Therefore, since the information requested is not available on non-UEA websites, I wish to re-instate my original request, that the information itself be made available on your website or in some other form. I understand that a small amount of this data (about 2%, according to your letter) is not available due to privacy requests from the countries involved. In that case, a listing of which stations this applies to will suffice.

The HadCRUT3 dataset is one of the fundamental datasets in the current climate discussion. As such, it is vitally important that it can be peer reviewed and examined to verify its accuracy. The only way this can be done is for the data to be made available to other researchers in the field.

Once again, thank you for your assistance in all of this. It is truly not a difficult request, and is fully in line with both standard scientific practice and your “CODE OF PRACTICE FOR RESPONDING TO REQUESTS FOR INFORMATION UNDER THE FREEDOM OF INFORMATION ACT 2000″. I am sure that we can bring this to a satisfactory resolution without involving appeals or unfavorable publicity.

Further to your email of 14 April 2007 in which you re-stated your request to see

“a list of stations used by Jones et al. to prepare the HadCRUT3 dataset” I am asking for: 1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and 2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.”

In your note you also requested “the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service)”,

I have contacted Dr. Jones and can update you on our efforts to resolve this matter.

We cannot produce a simple list with this format and with the information you described in your note of 14 April. Firstly, we do not have a list consisting solely of the sites we currently use. Our list is larger, as it includes data not used due to incomplete reference periods, for example. Additionally, even if we were able to create such a list we would not be able to link the sites with sources of data. The station database has evolved over time and the Climate Research Unit was not able to keep multiple versions of it as stations were added, amended and deleted. This was a consequence of a lack of data storage in the 1980s and early 1990s compared to what we have at our disposal currently. It is also likely that quite a few stations consist of a mixture of sources.

I have also been informed that, as the GHCN and NCAR are merely databases, the ultimate source of all data is the respective NMS in the country where the station is located. Even GHCN and NCAR can’t say with precision where they got their data from as the data comes not only from each NMS, but also comes from scientists in each reporting country.

In short, we simply don’t have what you are requesting. The only true source would be the NMS for each reporting country. We can, however, send a list of all stations used, but without sources. This would include locations, names and lengths of record, although the latter are no guide as to the completeness of the series.

This is, in effect, our final attempt to resolve this matter informally. If this response is not to your satisfaction, I will initiate the second stage of our internal complaint process and will advise you of progress and outcome as appropriate. For your information, the complaint process is within our Code of Practice and can be found at: http://www1. uea.ac.uk/ polopoly_ fs/1.2750! uea_manual_ draft_04b. pdf

Yours sincerely David Palmer Information Policy Officer University of East Anglia

I loved the story line in this one “we do not have a list consisting solely of the sites we currently use”. Say what? How do they produce updates that change the temperature all the way back to 1870 if they don’t have the data or a list of the sites? But I digress …

So I advised him that I was appealing. His letter was passed to a Ms. Kitty Inglis, who replied

Following David Palmer’s letter of 27th April 2007 to you regarding your dissatisfaction with our response to your FOI request of 25th January 2007, I have undertaken a thorough review of the contents of our file and have spoken with both Mr. Palmer and Professor Jones.

As a result of this investigation, I am satisfied that we have done all we can to fulfil [sic] your request and to provide you with the information you require where it is possible for us to do so.

I confirm that we are able to make available on the Climatic Research Unit website a list of stations, including name, latitude, longitude, elevation and WMO number (where available).

We are unable to provide a simple list of sources for these stations as we do not hold this information. Nor do we hold the raw (i.e. unadjusted) station data, as you describe it, at UEA. As stated in prior letters to you, raw station data are available on the NCAR and GHCN websites and gridded data are available on the Climatic Research Unit website. If these data are insufficient for your requirements, you will need to contact the NMS for the country in which the station is located to obtain the information you require.

I hope you are able to accept this response. We have contacted the Information Commissioner’s Office in relation to this matter and their advice is that if you are still dissatisfied with this response, you can, at this time, exercise your right of appeal to the
Information Commissioner by contacting them at:
Information Commissioner’ s Office
Wycliffe House

At that point, I let it go. I had a small victory, we got a list of the stations. Of course, it took me a couple more letters to actually get them to post the list. But I got nothing else of what I had requested, and the list was full of all kinds of errors.

Meanwhile, behind the scenes at CRU, I now find out that they were circling the wagons … what follows are their internal discussions about a series of FOI requests from myself, Steve McIntyre, Doug Keenan and others to CRU for various data. Phil Jones to Tom Keenan and Wei-Chyung Wang, 6/19/2007:

Wei-Chyung and Tom,
…
1. Think I’ve managed to persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.
2. Had an email from David Jones of BMRC, Melbourne. [EMAIL NOT FOUND IN CRU EMAILS – Willis] He said they are ignoring anybody who has dealings with CA, as there are threads on it about Australian sites.
3. CA is in dispute with IPCC (Susan Solomon and Martin Manning) about the availability of the responses to reviewer’s at the various stages of the AR4 drafts. They are most interested here re Ch 6 on paleo.
Cheers
Phil

Well, that explains a few things … they’ve managed to “persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.” I hadn’t noticed that exemption in the FOI documentation I’d seen. Call me crazy, but I don’t think that’s in FOI Exemptions, I doubt if it’s legal, and it definitely isn’t ethical. I note that they are circling the wagons in Australia as well … this is followed by:

Phil Jones to Thomas Peterson of NOAA, 6/20/2007 AM (1182342470) :

Tom P.

Just for interest. Don’t pass on.

Might be a precedent for your paper to J. Climate when it comes out. There are a few interesting comments on the CA web site. One says it is up to me to prove the paper from 1990 was correct, not for Keenan to prove we’re wrong. Interesting logic.
Cheers
Phil

Wei-Chyung, Tom,
I won’t be replying to either of the emails below [FROM STEVE MCINTYRE AND DOUG KEENAN], nor to any
of the accusations on the Climate Audit website. I’ve sent them on to someone here at UEA to see if we
should be discussing anything with our legal staff. The second letter seems an attempt to be nice to me,
and somehow split up the original author team. I do now wish I’d never sent them the data after their FOIA
request!

Cheers
Phil

He obviously views sending data in response to an FOIA request as optional.

Thomas Peterson to Jones, same email:

Fascinating. Thanks for keeping me in the loop, Phil. I won’t pass it on but I will keep it in the back of my mind when/if Russ asks about appropriate responses to CA requests. Russ’ view is that you can never satisfy them so why bother to try?

Again, responding to an FOIA request is viewed as optional.

Phil Jones :

…

PS to Gavin – been following (sporadically) the CA stuff about the GISS data and release of the code etc by Jim. May take some of the pressure off you soon, by releasing a list of the stations we use – just a list, no code and no data. Have agreed to under the FOIA here in the UK.

Oh Happy days!

So I see … that’s why I only got the station list and not the data, just to ” take some of the pressure off “. Thanks, Phil.

Jones to Bradley and Amman, 5/9/08 (1210341221):

Mike, Ray, Caspar,

A couple of things – don’t pass on either.
…
2. You can delete this attachment if you want. Keep this quiet also, but this is the person [DAVID HOLLAND – Willis] who is putting in FOI requests for all emails Keith and Tim have written and received re Ch 6 of AR4. We think we’ve found a way around this.

Finding ways around FOI requests seems to be a popular sport at CRU. This is in reference to Steve trying to get the review comments to Chapter 6 of the UN IPCC Fourth Assessment Report.

Next, here’s the brilliant way that they had found around the FOIA, a bombshell of an idea, Jones to Michael Mann, 29 May 2008 (1212063122):

Mike,

Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment – minor family crisis.

Can you also email Gene and get him to do the same? I don’t have his new email address.

We will be getting Caspar to do likewise.

…

Cheers
Phil

Again, call me crazy, but deleting evidence in the face of an FOI request must be illegal. Gene is Eugene Wahl. Of course, what these guys don’t realize is that there are multiple copies of most emails floating around. In some ways, I hope they deleted them, so that it can be proven. The story continues …

Have a look at Climate Audit. Holland has put all the responses and letters up. There are three threads – two beginning with Fortress and a third later one. Worth saving the comments on a Jim Edwards – can you do this Tim?

I’ve saved all three threads as they now stand. No time to read all the comments, but I did note in “Fortress Met Office” that someone has provided a link to a website that helps you to submit FOI requests to UK public institutions, and subsequently someone has made a further FOI request to Met Office and someone else made one to DEFRA. If it turns into an organised campaign designed more to inconvenience us than to obtain useful information, then we may be able to decline all related requests without spending ages on considering
them. Worth looking out for evidence of such an organised campaign.

Tim

Another thing to hide behind, a false claim of an “organised campaign”. I loved the “Confidential Agent Amman” …

Gavin,
…
Keith/Tim still getting FOI requests as well as MOHC and Reading. All our FOI officers have been in discussions and are now using the same exceptions not to respond – advice they got from the Information Commissioner. As an aside and just between us, it seems that Brian Hoskins has withdrawn himself from the WG1 Lead nominations. It seems he doesn’t want to have to deal with this hassle.

The FOI line we’re all using is this. IPCC is exempt from any countries FOI – the skeptics have been told this. Even though we (MOHC, CRU/UEA) possibly hold relevant info the IPCC is not part our remit (mission statement, aims etc) therefore we don’t have an obligation to pass it on.

Cheers
Phil

So now the Information Commissioner is in on the deal, s/he’s advising them to use the same exceptions not to respond. No need to think about it, all of the wheels have been greased.

My personal opinion is that both FOI requests (1) and (2) are intrusive and unreasonable. Steven McIntyre provides absolutely no scientific justification or explanation for such requests. I believe that McIntyre is pursuing a calculated strategy to divert my attention and focus away from research. As the recent experiences of Mike Mann and Phil Jones have shown, this request is the thin edge of wedge. It will be followed by further requests for computer programs, additional material and explanations, etc., etc.

Quite frankly, Tom, having spent nearly 10 months of my life addressing the serious scientific flaws in the Douglass et al. IJoC paper, I am unwilling to waste more of my time fulfilling the intrusive and frivolous requests of Steven McIntyre. The supreme irony is that Mr. McIntyre has focused his attention on our IJoC paper rather than the Douglass et al. IJoC paper which we criticized. As you know, Douglass et al. relied on a seriously flawed statistical test, and reached incorrect conclusions on the basis of that flawed test.

I believe that our community should no longer tolerate the behavior of Mr. McIntyre and his cronies. McIntyre has no interest in improving our scientific understanding of the nature and causes of climate change. He has no interest in rational scientific discourse. He deals in the currency of threats and intimidation. We should be able to conduct our scientific research without constant fear of an “audit” by Steven McIntyre; without having to weigh every word we write in every email we send to our scientific colleagues.

In my opinion, Steven McIntyre is the self-appointed Joe McCarthy of climate science. I am unwilling to submit to this McCarthy-style investigation of my scientific research. As you know, I have refused to send McIntyre the “derived” model data he requests, since all of the primary model data necessary to replicate our results are freely available to him. I will continue to refuse such data requests in the future. Nor will I provide McIntyre with computer programs, email correspondence, etc. I feel very strongly about these issues. We should not be coerced by the scientific equivalent of a playground bully.

I will be consulting LLNL’s Legal Affairs Office in order to determine how the DOE and LLNL should respond to any FOI requests that we receive from McIntyre. I assume that such requests will be forthcoming.

I am copying this email to all co-authors of our 2008 IJoC paper, to my immediate superior at PCMDI (Dave Bader), to Anjuli Bamzai at DOE headquarters, and to Professor Glenn McGregor (the editor who was in
charge of our paper at IJoC).

I’d be very happy to discuss these issues with you tomorrow. I’m sorry that the tone of this letter is so formal, Tom. Unfortunately, after today’s events, I must assume that any email I write to you may be subject to FOI requests, and could ultimately appear on McIntyre’s “ClimateAudit” website.

With best personal wishes,

Ben

Well, he got the last paragraph right, at least. He also thinks that an FOIA request must serve some “scientific justification”, with the justification determined by … well … by the person receiving the request, of course. Another previously unknown part of the FOI Exemptions comes to light.

Ben Santer to Tom Wigly, 12 Dec 07 (1228330629):

At 01:17 03/12/2008, Ben Santer wrote:

Dear Tom,
…
One of the problems is that I’m caught in a real Catch-22 situation. At present, I’m damned and publicly vilified because I refused to provide McIntyre with the data he requested. But had I acceded to McIntyre’s initial request for climate model data, I’m convinced (based on the past experiences of Mike Mann, Phil, and Gavin) that I would have spent years of my scientific career dealing with demands for further explanations, additional data, Fortran code, etc. (Phil has been complying with FOIA requests from McIntyre and his cronies for over two years). And if I ever denied a single request for further information, McIntyre would have rubbed his hands gleefully and written: “You see – he’s guilty as charged!” on his website.

You and I have spent over a decade of our scientific careers on the MSU issue, Tom. During much of that time, we’ve had to do science in “reactive mode”, responding to the latest outrageous claims and inept science by John Christy, David Douglass, or S. Fred Singer. For the remainder of my scientific career, I’d like to dictate my own research agenda. I don’t want that agenda driven by the constant need to respond to Christy, Douglass, and Singer. And I certainly don’t want to spend years of my life interacting
with the likes of Steven McIntyre.

I hope LLNL management will provide me with their full support. If they do not, I’m fully prepared to seek employment elsewhere.

With best regards,
Ben

Dr. Santer, here’s a novel idea. Put enough information out when you publish the work so that your work can be replicated. Put on the web whatever is necessary in the way of code, data, and methods to allow your work to be checked by someone else. If you do that, not only will you not be bothered, but you will be following the scientific method. None of us at ClimateAudit are doing this to harass anyone, as you claim. We’re doing this because we cannot replicate your work, and thus your work is purely anecdotal rather than scientific.

Phil responds (same email):

Cc: mann , Gavin Schmidt, Karl Taylor, peter gleckler

Ben,
When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions – one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school – the head of school and a few others) became very supportive. I’ve got to know the FOI person quite well and the Chief Librarian – who deals with appeals. The VC is also aware of what is going on – at least for one of the requests, but probably doesn’t know the number we’re dealing with. We are in double figures.

One issue is that these requests aren’t that widely known within the School. So I don’t know who else at UEA may be getting them. CRU is moving up the ladder of requests at UEA though – we’re way behind computing though. We’re away [aware?]of requests going to others in the UK – MOHC, Reading, DEFRA and Imperial College.

So spelling out all the detail to the LLNL management should be the first thing you do. I hope that Dave is being supportive at PCMDI. The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian, saying that the email maligned his scientific credibility with his peers!

If he pays 10 pounds (which he hasn’t yet) I am supposed to go through my emails and he can get anything I’ve written about him. About 2 months ago I deleted loads of emails, so have very little – if anything at all. This legislation is different from the FOI – it is supposed to be used to find put why you might have a poor credit rating !

In response to FOI and EIR requests, we’ve put up some data – mainly paleo data. Each request generally leads to more – to explain what we’ve put up. Every time, so far, that hasn’t led to anything being added – instead just statements saying read what is in the papers and what is on the web site! Tim Osborn sent one such response (via the FOI person) earlier this week. We’ve never sent programs, any codes and manuals.

In the UK, the Research Assessment Exercise results will be out in 2 weeks time. These are expensive to produce and take too much time, so from next year we’ll be moving onto a metric based system. The metrics will be # and amounts of grants, papers and citations etc. I did flippantly suggest that the # of FOI requests you get should be another.

When you look at CA, they only look papers from a handful of people. They will start on another coming out in The Holocene early next year. Gavin and Mike are on this with loads of others. I’ve told both exactly what will appear on CA once they get access to it!

Cheers
Phil

Well, that explains why David Palmer and Ms. Kitty Inglis, the Chief Librarian, were so unsupportive. Took a couple of half hour sessions, but at the end of that, rather than being a representative of the FOI process, they were functioning as the personal representatives of Phil Jones. We have a new reason I hadn’t noticed in the FOI law for refusing a request, because the requester posts at CA.

And since they have the FOI person, and the FOI Appeals person, and the Information Commissioner in there pockets, and they have the standard terms of refusal figured out … just how difficult can it be to deny an FOI Request?

Jones to Ben Santer again, 10 Dec 2008:

Ben,

Haven’t got a reply from the FOI person here at UEA. So I’m not entirely confident the numbers are correct. One way of checking would be to look on CA, but I’m not doing that. I did get an email from the FOI person here early yesterday to tell me I shouldn’t be deleting emails – unless this was ‘normal’ deleting to keep emails manageable! McIntyre hasn’t paid his £10, so nothing looks likely to happen re his Data Protection Act email.

Anyway requests have been of three types – observational data, paleo data and who made IPCC changes and why. Keith has got all the latter – and there have been at least 4. We made Susan aware of these – all came from David Holland. According to the FOI Commissioner’ s Office, IPCC is an international organization, so is above any national FOI. Even if UEA holds anything about IPCC, we are not obliged to pass it on, unless it has anything to do with our core business – and it doesn’t! I’m sounding like Sir Humphrey here! McIntyre often gets others to do the requesting, but requests and responses all get posted up on CA regardless of who sends them.

On observational data, there have been at least 5 including a couple from McIntyre. Others here came from Eschenbach and also Douglas Keenan. The latter relate to Wei-Chyung Wang, and despite his being exonerated by SUNY, Keenan has not changed his web site since being told the result by SUNY!

The paleo data requests have all been to Keith, and here Tim and Keith reply. The recent couple have come from McIntyre but there have been at least two others from Holland. So since Feb 2007, CRU is in double figures. We never get any thanks for putting things up – only abuse and threats. The latest lot is up in the last 3-4 threads on CA.

I got this email over the weekend – see end of this email. This relates to what Tim sent back late last week. There was another one as well – a chatty one saying why didn’t I respond to keep these people on CA quiet. I’ve ignored both. Finally, I know that DEFRA receive Parliamentary Questions from MPs to answer. One of these 2 months ago was from a Tory MP asking how much money DEFRA has given to CRU over the last 5 years. DEFRA replied that they don’t give money – they award grants based on open competition. DEFRA’s system also told them there were no awards to CRU, as when we do get something it is down as UEA!

Since he and Mann and the others have already deleted their emails, looks like David Palmer (the “FOI person”) was a bit too late with his excellent advice … however, I did get a “Mentioned In Dispatches” from Phil, at least …

I also like the sly way he tells Ben how to illegally delete emails, just do it as part of ‘normal’ deleting to keep emails manageable! Yeah, right, that’s the ticket.

Phil Jones to Raymond Pierrehumbert, 16 Jan 09 (1200493432):

Cc: Michael Mann , Gavin Schmidt

Ray,
…

I have had a couple of exchanges with Courtillot. This is the last of them from March 26, 2007. I sent him a number of papers to read. He seems incapable of grasping the concept of spatial degrees of freedom, and how this number can change according to timescale. I also told him where he can get station data at NCDC and GISS (as I took a decision ages ago not to release our station data, mainly because of McIntyre). I told him all this as well when we met at a meeting of the French Academy in early March.
…
Cheers, Phil

This is a very clear statement of what he has done. He has refused to release the data, not because there is any logical reason to do so, but “because of McIntyre”. This is shameful, and the fact that the FOI people, Dave Peters and Kitty Inglis and the Information Commissioner, went along with this is dereliction of duty.

“Thanks” Ben for this, hi all and happy new year. I had a similar experience– but not FOIA since we at Climatic Change are a private institution- -with Stephen McIntyre demanding that I have the Mann et al cohort publish all their computer codes for papers published in Climatic Change I put the question to the editorial board who debated it for weeks. The vast majority opinion was that scientists should give enough information on their data sources and methods so others who are scientifically capable can do their own brand of replication work, but that this does not extend to personal computer codes with all their undocumented sub routines etc. It would be odious requirement to have scientists document every line of code so outsiders could then just apply them instantly. Not only is this an intellectual property issue, but it would dramatically reduce our productivity since we are not in the business of producing software products for general consumption and have no resources to do so. The NSF, which funded the studies I published, concurred–so that ended that issue with Climatic Change at the time a few years ago.

This continuing pattern of harassment, as Ben rightly puts it in my opinion, in the name of due diligence is in my view an attempt to create a fishing expedition to find minor glitches or unexplained bits of code–which exist in nearly all our kinds of complex work–and then assert that the entire result is thus suspect. Our best way to deal with this issue of replication is to have multiple independent author teams, with their own codes and data sets, publishing independent work on the same topics–like has been done on the “hockey stick”. That is how credible scientific replication should proceed.

Let the lawyers figure this out, but be sure that, like Ben is doing now, you disclose the maximum reasonable amount of information so competent scientists can do replication work, but short of publishing undocumented personalized codes etc. The end of the email Ben attached shows their intent–to discredit papers so they have no “evidentiary value in public policy”–what you resort to when you can’t win the intellectual battle scientifically at IPCC or NAS.

Good luck with this, and expect more of it as we get closer to international climate policy actions, We are witnessing the “contrarian battle of the bulge” now, and expect that all weapons will be used.

Cheers, Steve

PS Please do not copy or forward this email.

Now, why would Dr. Schneider not want his email copied or forwarded … perhaps because he is saying don’t follow the spirit of the Freedom of Information Act, don’t release the code that shows the math and reveals how you got your results? He foolishly thinks that studies can be “replicated” by using different data and different codes … but that says absolutely nothing about the original study and whether it contains any mistakes. The only way to determine whether a study like a historical temperature reconstruction contains errors is to examine the scientists actual work You can’t just pick another different bunch of proxies, analyze them, and say “I’ve found mathematical errors in your reconstruction”. You can only find those errors if you examine the actual math the researcher used, and to do that you need access to “their” codes.

I put “their” codes, in quotes because, under the policies of the University of East Anglia (and many other Universities), the codes do not belong to Phil Jones. They were developed as a part of his employment, and as such they belong to the University, and not to Phil.

The researchers complain in various places that they do not want to reveal their “primary data” because it is available on the web. While this is often true, as I saw in my FOIA requests to CRU, it is not sufficient Just saying “I got the information from Website X” as CRU did is often totally inadequate to locate the data in question. Santer makes this charge, that anyone could go the CMIP website and get the data themselves … but unless he says exactly which data from which run of which model, the website address is meaningless.

The main impression that I get from the emails is that the various scientists think that I and other requesters are simply doing this to harass them. Nothing could be further from the truth. I respect actual scientists, I’m short of time myself so I understand time pressures, so I have no desire to put any scientist to any extra effort beyond providing what science requires – a full accounting of the data, the methods, and in some cases the computer code used to do the research. Anything more is harassment … but anything less is scientific obstruction. And if they would provide those things when they publish their results, they’d never hear from me. And if Nature Magazine and Science Magazine and the National Science Foundation and all of the journals and funders would just enforce their own existing rules on archiving and transparency, the problem would be solved. But noooo, for the select friends of Phil these bothersome transparency regulations are ignored and overlooked.

As I said, the issue is not Trenberth or the Nature “trick” or scientists talking smack about each other. It is the illegal evasion of legitimate scientific requests for data needed to replicate a scientific study. Without replication, science cannot move forwards. And when you only give data to friends of yours, and not to people who actually might take a critical look at it, care to guess what you might end up with?

218 thoughts on “The people -vs- the CRU: Freedom of information, my okole…”

A) How women matter in decreasing world population
B) The energy we need
C) Mining the impacts
D) Symbiotical relationship of religion and global life-support systems
E) Celebration of Life Day
F) The hidden agenda: China
G) Earth Government now a priority
H) The splitting of America into separate independent states living at peace for the
good of all
I) The war industry: the modern evil at work in the Middle East
J) Earth security
K) Earth governance
L) The Earth Court of Justice holds the people of the U.S.A. and Britain as criminals
M) Foundation for the new world order, Earth Government

Santer comes out of this looking like a big crybaby. And he clearly chose the wrong profession if he thinks that his work is always going to be beyond criticism. This guy is giving science a black eye.

Revkin is a shill. No more or less. I always gave him the benefit of the doubt… thinking that once there was solid evidence that things weren’t on the straight and narrow, he’d report it appropriately.

There are a number of historical threads covering the FOIA requests from Steve McIntyre, myself, Ross McKitrick, and others at Climateaudit. CA is loading slowly, the server is overloaded by people interested in the story.

Willis, what makes this by far the best expose on the emails so far is that you are able to reach into your own experiences and furnish the “glue” that allows the series of emails to form a coherent story line. No journalist, no matter how good at his trade, can do this, because he wasn’t part of the drama.

I hope this entry is published far and wide; it’s by far the most cohesive picture of the chicanery published yet.

Pseudscience has the first step as “Belief”, or “assumption based on a consensus” and not observation. No one has observed AGW occurring. This is a subtle but crucial difference, and one that also occurs in astronomy, archaealogy and geology. AGW is hence deductive science minus the initial empirical basis and climate science spends all of its efforts proving the initial assumption, not realising that no scientific theory can ever be proven, but they can be falsified. Hence AGW is the misuse of the scientific method for political purposes.

Interesting to see the interplay regarding FOI. It really helps to have your side as well. It’s sad that there are people who claim to be scientists who will function as apologists for this behavior – it’s wrong and they know it.

I especially love the comparison between McIntyre and McCarthy (!)…McIntyre is requesting that publicly funded data and research methods be made available as is consistent with the scientific method. And their response is that he’s being a totalitarian and a schoolyard bully?

It goes without saying that attempts to withhold data go against the spirit of scientific inquiry. Moreover, CRU staff seem to have behaved in a manner which, at first sight, appears to border on the illegal. A chronological chain of this kind does the public a not inconsiderable service by highlighting these issues to a broader readership. Both the obfuscation of science and the possibility illegality of their actions should be pursued with some vigour.

I can now see what realclimate was talking about when they said that these emails were made available without the proper context. Giving the full context is much clearer – these people’s behavior is despicable and no excuse can now be found for it.

I asked some questions in the comments section of Mr Revkin’s latest article. It will be interesting to see whether the moderator allow them. (I posted them under my real name, Informal. Yes, really.)

Does not the recent release of CRU e-mails demonstrate that much of the allegedly settled science on global warming is in fact bogus? Has not Mr. Revkin been mentioned in those e-mails as a tame spokesman of “the Team” which has organized a cabal of like-minded frauds to manipulate data and the world’s governments and media into fearing unlikely catastrophes? Should not Mr. Revkin disclose his interests and bias when reporting on alleged climate science? Is IPCC an acronym for Impossible to Predict Climate Change? Is it not an absolute certainty that I shall be calumniated as an heretical denialist for asking these questions?

It is the illegal evasion of legitmate scientific requests for data needed to replicate a scientific study. Without replication, science cannot move forwards.

Without replication, the results not science, they are no more valid than opinion or conjecture. As so many have others have said, until these studies can be replicated, HadCRUT3, and all studies and policy based on HadCRUT3, must be thrown out until the issue is examined in detail by independent parties.

Part of the difficulty with climate science is that, unlike all other physical sciences, it does not study things — instead it studies averages.

This is because climate by definition is the average of weather over a suitably long period of time (typically taken as a minimum of 30 years).

As a result, much of the study that goes on, and the papers that are written, deal almost exclusively with mathematics and statistics. This is the reason that access to the computer codes is so critical.

It’s simple in the physical sciences to describe an experiment, e.g. “I took three grams of carbon and subjected them to a pressure of 50,000KPa and a temperature of 500C. Unfortunately, the experiment did not succeed, I could not replace the diamond I had lost from my wife’s wedding ring.” Anyone can reproduce that experiment (and get the same results).

But when you say “I took the raw temperature data, variance-adjusted it, averaged it, gridded it, area-adjusted it, extrapolated results to data-free areas within 250 km, and made a global temperature record”, that’s far from enough information. In order to determine what was done, we need far more detailed information in climate science, because in general we are describing intricate mathematical operations. These are often very hard to describe clearly in spoken or written language.

And even a crystal-clear description is not enough. Despite what he says he has done, if the scientist has inadvertently used an improper procedure (e.g. the uncentered principal components analysis used in Mann’s Hockeystick), we’ll never be able to determine that the answer is demonstrably wrong unless we have the actual code that he used. Otherwise, we could spend years trying to guess where he went wrong, but we would never be able to show that he went wrong as science demands.

This is why the insistence of scientists that their computer codes are sacrosanct private secret documents best kept under Hermetic seal in a clandestine vault is lethal to good science. Without the codes, we can’t tell if what has been done is correct and free from hidden mathematical error. Of course, this may be unconnected with the reason that Jones et. al are hiding their codes …

Given that climate science is not the study of things but of the averages of things, and that as a result math and statistics are central to climate science, the findings of the Wegman Report are now seen to be even more insightful, trenchant, and valid. They said:

It is important to note the isolation of the paleoclimate community; even though they rely heavily on statistical methods they do not seem to be interacting with the statistical community. Additionally, we judge that the sharing of research materials, data and results was haphazardly and grudgingly done. In this case we judge that there was too much reliance on peer review, which was not necessarily independent. Moreover, the work has been sufficiently politicized that this community can hardly reassess their public positions without losing credibility. Overall, our committee believes that Mann’s assessments that the decade of the 1990s was the hottest decade of the millennium and that 1998 was the hottest year of the millennium cannot be supported by his analysis.

And presciently, that was written thee years ago, well before we got the CRU emails …

Can’t argue with Popper, Popper wins of Aristotle any day of the week. Indeed it was Aristotle’s views and methods that helped make the dark ages so dark; except there is no evidence that the dark ages were dark.
PC analysis has shown that using paving-stone wear patterns as proxies for intellectual development clearly indicate that the people in the dark ages had a profound understanding of the natural world, the scientific method and the dangers of too much atmospheric CO2. They deliberately didn’t burn fossil fuels to save future generations.

They are elitists. They are not to be questioned by YOU. And they enlist their elitist friends in helping them. Since they see themselves as THE elite scientists they believe themselves to be above the petty requirements of the FOI act, these acts don’t apply to THEM. We see this all the time in politics, in journalism, throughout academia, nothing special in it, it is as common as dirt. The really interesting part is that these idiots are frequently socialists, they really want socialism to take over the world. And once it does it is these same elitists that quickly end up in the gulags as slaves or dead. It is quite logical actually, these elitists are insufferable to be around so once the Marxist thugs take over they quickly get rid of them. It is true, look at the old Soviet Union, Cuba, China, anywhere socialism took over, the elitists help them to power then get purged. They are not really that smart, they do this over and over again, history means nothing to them. Their ego tells them they are above history. If they are lucky they live somewhere that never goes socialist and never have their wish fulfilled, and they live a full life honored by their equally elitist friends, writing Marxist drivel, living in moderately well off surroundings; bitter, clinging to their ideology and their tenure.
Keep pushing, these are brittle people, they’ll crack before they’ll bend.

It is very clear now that there were 100% valid reasons for these requests.

They lost all the raw data, they never had it the first place, the Chinese data was only a hardcopy that we can’t locate now, all the raw data is held at the NCDC, except for the 2% of stations that we got directly, that we can’t locate now, that we have no way of tracking in the database, that a programmer can’t make heads or tails out of, but the chart of our data clearly shows anthropogenic warming.

But now we still have to move forward. CRU data will not be released for some time now (unless they are serious about an independent investigation).

There are two agencies which might have escaped from the requests so far (actually I’m not sure about that but in case they have) …

The UK Met Office and the NCDC.

From reading the emails, it seems the Hadley Centre portion of the Met Office might be in possession of some of the data and the NCDC, well I have had little luck getting usable data from their websites but someone can try these two agencies next.

Willis, couldn’t agree more. This is what I said in Tips and Notes earlier:

David (12:31:24) :

…..In the HARRY_READ_ME.txt, a search for ’synthetic cloud grids’ brings up this particular piece of information ‘This means that the cloud data prior
to 1996 are static.’ If the climate models can reproduce past climate with static cloud figures, then they are quite obviously wrong.

It is past time for the climate models to become open source, so that everyone can replicate the results of these models to their hearts content. It is also imperative that the data used to model climate become open to the public as well, as the disturbing trend of silencing critics of the AGW theory revealed in the CRU leak is simply a reminder that these scientists are also human and subject to the same internal faults as those in the financial community regularly excoriated for their crimes against investors (which, by the way, led to the practice of auditing financial statements as we know it today.)
—————

If I understand HARRY_READ_ME correctly, isn’t that the GCM they are talking about having cloud data prior to 96 as static? That seems like a huge mistake if that is the case.

If they can’t identify the stations they’re collecting data from, then presumably they don’t know what it refers to. Its akin to going to the bank and depositing cheques, and the cashier putting them into random acounts. Just imagine 100 people sitting exams and the examiner allocating results based on alphabetical order..

It looks like a rather clever “trick” (to use Jones language) to circumvent FOI

If this was indicative of my systems : Dr. A Burns,I learned a bit of Fortran Programming
back in my undergrad days downloading statistics from a Porcupine study I “Volunteered” to be involved in.In my own very limited ability-that is indeed a “Crap-crap” mess! Now it’s there for the whole world to see.It reminds me of the type of work that someone does after a couple of joints and a beer thinking they are the next Einstein or Darwin. Then the morning comes and what you wrote is absolute,
gibberish, drivel,-and the paper is due at 2 pm….
(Gulp)

This really gets back to what I have been saying for over a year now, on as many blogs that will take 5000 letter comments:

There has been atmospheric cooling the last 8 years, and no new high global annual temperatures in the last 11 years. None of the computer models replicate this fact. Anthropogenic (or man caused) global warming is not proved.

The global warming adherents base their argument of proof on more than 20 different computer models called general circulation models (also known as global climate models or GCMs). Each computer model is composed of dozens of mathematical equations representing known scientific laws, theories, and hypotheses. Each equation has one or more constants. The constants associated with known laws are very well defined. The constants associated with known theories are generally accepted but probably some of them may be off by a factor of 2 or more, maybe even an order of magnitude. The equations representing hypotheses, well, sometimes the hypotheses are just plain wrong. Then each of these equations has to be weighted against each other for use in the computer models, so that adds an additional variable (basically an educated guess) for each law, theory, and hypothesis. This is where the models are tweaked to mimic past climate measurements.

Now we find that by deleting the Medieval Warm Period, minimizing the Little Ice Age, and using a “trick” on the last half of the 20th Century, that allowed them to use grossly overstated coefficients and independent variables in ALL of the GCMs. Then all the GCMs say we ‘may’ burn up in the year 2100.

The SCIENTIFIC METHOD is: (1) Following years of academic study of the known physical laws and accepted theories, and after reviewing some data, come up with a hypothesis to explain the data. (2) Develop a plan to obtain and analyze new data. (3) Collect and analyze the data, this may even require new technology not previously available. (4) Determine if the hypothesis is correct, needs refinement, or is wrong. Either way, new data is available for other researchers. (5) Submit results, including data, for peer review and publication.

The output of the computer models run out nearly 90 years forward is considered to be data, but it is not a measurement of a physical phenomenon. Also, there is no way to analyze this so called data to determine if any or which of the hypotheses in the models are correct, need refinement, or are wrong. Also, this method cannot indicate if other new hypotheses need to be generated and incorporated into the models. IT JUST IS NOT THE SCIENTIFIC METHOD.

The worst flaw in the AGW argument is the treatment of GCM computer generated outputs as data. They then use it in follow on hypotheses. For example, if temperature rises by X degrees in 50 years, then Y will be effected in such-and-such a way resulting in Z. Then the next person comes along and says, well, if Z happens, the effect on W will be a catastrophe. “I need (and deserve) more money to study the effects on W.” Hypotheses, stacked on hypotheses, stacked on more hypotheses, all based on computer outputs that are not data, using a process that does not lend to proof using the SCIENTIFIC METHOD. Look at their results, IF, MIGHT, and COULD are used throughout their news making results. And when one of the underlying hypotheses is proven incorrect, well, the public only remembers the doomsday results 2 or three iterations down the hypotheses train. The hypotheses downstream are not automatically thrown out and can even be used for more follow on hypotheses.

“Fascinating. Thanks for keeping me in the loop, Phil. I won’t pass it on but I will keep it in the back of my mind when/if Russ asks about appropriate responses to CA requests. Russ’ view is that you can never satisfy them so why bother to try?”

Tom P has it completely backward: they had never tried to satisfy CA requests before, so why start now?

That was an excellent summary, Willis. After following this on Climate Audit, and now here, a few things are apparent:

These people are on a crusade. They truly believe that gives them carte blanche to invent and unilaterally “adjust” climate records.

In many cases they do not have empirical, replicable data; they apparently fabricated it in numerous cases.

They say their climate cohort Wei Cyung Wang was exonerated, and they provide a link to show that. But the link goes to this pdf file, which is the complaint filed against Wang.

They have real contempt for the UK’s Freedom of Information Act. Also, the FOIA officer and her superior became their active supporters; the FOIA people never asked for the point of view of the complainants, but instead reached a decision after talking only with the CRU scientists, and thus became their advocates.

They have a siege mentality, which can be seen in their constant impugning of Steve McIntyre’s motives as their rationale for refusing cooperation, rather than following the scientific method — which requires full cooperation with those requesting their data and methodologies. That raw data and their methods of “adjusting” it appear to either not support their conclusions, or were fabricated in order to show AGW. We can see that their excuses for refusing to provide the data were simply a subterfuge that they invented as they went along, constructed by emailing each other.

The fact that literally trillions of dollars in added taxes are now being proposed, based largely on these scientists’ unverifiable pronouncements that catastrophic global warming will result from human CO2 emissions, requires a prompt, full and complete investigation of this glaringly apparent fraudulent perversion of science, by a prosecutor whose character is unimpeachable.

In the mean time, let the discovery and depositions begin in the civil arena.

More than anything, the series of emails from those people really reflect their personalities, and it is clear that they don’t have nice personalities or respect for others that don’t think as they do… shame on them.

This is the most damning evidence I’ve seen so far against the CRU scientists. If all of this is true, the warmers claims of “peer reviewed” science will be blown to pieces. And if this “science” falls apart what will happen to the politicians who’ve been promoting it? They will be shown to be either gullible dupes who were willing to waste hundreds of billions of taxpayer dollars on a wild goose chase, or criminals who were knowing accomplices in the fraud. Neither of those is a good qualification for re-election.

I think this will be the end of what was known as “the peer-review process” as we know it. I like the idea of putting things up for all to see and comment on. What we will need are websites that will compile and manage all that information… a wiki type of management of science… Wikijournals maybe?

Sorry, Willis, I was in a dead hurry because I was due to do a podcast interview, didn’t mean to be obscure. And thanks for recalling Wegman’s conclusions, because I think that’s exactly the process through which CRU got itself into this mess.

If a fiction writer made up this story, no one would believe him. Truth is stranger than fiction. I admire your patience and persistence.

What someone should now do is file an FOIA request for all communications (e-mails sent and received, letters sent and received, internal memos, etc.) regarding the avoidance of providing information requested in a actual or anticipated FOIA request. Specific examples should be given as to the dates, subject line, addressee, sender, that were in the “hacked file”, and ask for these and any others like them. The request should be for both those items which still exist as well as any items which were deleted. Any communications that involve someone outside of UEA as a sender or receiver could then begin another FOIA request to another organization. The replies (if any) could be interesting.

“The scaling of the data we used to produce the Crowley curve that formed one of the lines in our spaghetti diagram (that we put on the web site under my name and made available to
NGDC), was based on taking the unscaled composite he sent and re-calibrating against
April – Sept. average for land North of 20 degrees Lat., and repeating his somewhat
bazaar calibration procedure

(which deliberately omitted the data between 1900-1920 that did not fit with the instrumental data (remember his data are also decadal smoothed
values)

In fact , as we were using summer data we calibrated over 1881-1900

(avoiding the high early decades that I still believe are biased in summer)

Malcolm
honestly just a cross thread between Tom and I. I had been asked by Darrell whether we should use the Sidorova chronology – because of hassle by you know who – so asked Tom a while ago to ask you. I did not see your answer – sorry if you cc’d me in as I have not been
checking emails. I fully accept and would NEVER go behind your back to ask for the data. I understood that the chronology was published and so thought to compare our RCS version with it if we could produce it in time .

We are being accused of not using that chronology in the Science paper- so then asked Anders for it.

I am happy to send Darrell the single chronology if that is what Anders has sent. I am
having to start thinking about the Yamal crap and then this Darrell stuff suddenly arises. I just wanted him to consider including the Polar Urals reconstruction and the Sidorova series in his analysis before publishing a correction in Science-

apparently the selection criterion for inclusion of series was anything published north of 60 degrees and longer than 1000 years.

Alleged CRU Email – 1163771694.txt
At 11:57 Uhr +0000 16.11.2006, Keith Briffa wrote:
> Martin and all,
Another serious issue to be considered relates to the fact that the PC1
time series in the Mann et al. analysis was adjusted to reduce the positive
slope in the last 150 years (on the assumption – following an earlier paper
by Lamarche et al. – that this incressing growth was evidence of carbon
dioxide fertilization) , by differencing the data from another record
produced by other workers in northern Alaska and Canada (which incidentally
was standardised in a totally different way).
This last adjustment obviously will have a large influence on the quantification of the link between these Western US trees and N.Hemisphere temperatures.

At this point , it is fair to say that this adjustment was arbitrary and the link between Bristlecone pine growth and CO2 is , at the very least, arguable.

Note that at least one author (Lisa Gaumlich) has stated that the recent growth of these trees could be temperature driven and not evidence of CO2 fertilisation.

That is more than interesting. It is a perfect example of why the code needs to be released.

The “oops” is just an error message if the number of years in their value adjusting data doesn’t match the number of years in the record. That’s not an issue, I do the same. It’s just an error test, that’s good programming.

But look at what they are actually doing. They are fudging the numbers by an arbitrary amount to get the result they want. You probably understand this code, but let me see if I can explain it in English for those who don’t.

First, they put together a list of numbers that will be used to calculate the yearly adjustment. The numbers start at zero, get a bit smaller, and then gradually increase. This list is called “valadjust”.

At the end of their calculations, they use valadj to make a yearly adjustment string, yearadj, by interpolating (taking intermediate values) from the values in valadj. This gives a value for each year, by which each year’s data will be adjusted up or down.

densall=densall+yearlyadj

But of course, they’re fudging things, so it won’t come out right the first time. To control the process, they put in a “fudge factor”, a single number that they can use to change the size of all the data adjustments. This is the “fudge factor” referred to in the code, which at the moment is set to 0.75. This is the 0.75 at the end of the line for setting up the valadj values:

They are using arbitrary values plus a fudge factor to make the result just what they want … YIKES! Or as Hal said in 2001, I DON’T THINK I CAN DO THAT, DAVE!

Upon such solid well tested “simple physics” calculations is the lofty edifice built … this is definitely why we need access to the code. This is far worse than I had feared, and I feared that it would be bad. Somebody’s got some serious ‘splaining to do.

Brilliant Willis, you’ve created a perfect linked ‘audit trail’ of events, dates and texts which proves the hacked/leaked file is accurate at least in respect to the ones regarding evasion of FOI requests.

I’ve been trying to do the same with various bits, but it is difficult to find the links when you don’t have the context or rough time-line (good fun though :-)

Your saga needs to be given an airing in the mainstream media – Telegraph could be a good start-point unless, of course, a gag-order has already been issued? It seems to me that democracy has broken down and we are fast becoming a Pan-European totalitarian state.

Reading this email trail gave me a distinct sense of deja vu. In another life I had been party to meetings concerning the deregulation of the electricity industry. The subject was rife with political agendas that played out in these meetings while some of us old goats around the table were concentrating on keeping the lights on (we failed in August 2003 for much of north east America). As the discussions wallowed in the political mire I used to play with my car key fob that still has this quip imprinted upon it: “I’m trying to see things from your point of view, but I can’t get my head that far up my ass.” Good manners and a lack of cojones prevented me from reading it out loud at the appropriate moment. However, I feel the folks at CRU and elsewhere who are playing fast and loose with science for political gain should take note because I think I see their August 2003 on the horizon.

In the future, these cru-tape letters of climate-gate will provide fodder for research and case studies in many areas:

The politics of the symbiotic relation between government and $cience.
The ethics of hacking/whistle-blowing to expose scientific fraud.
Information stored electronically just makes it all that more insecure.
Using selective interpolation for predetermined extrapolation.
The scientific method amended to include “the end justifies the means.”
… and yes, man is incurably religious in his beliefs!

“Obama Science Czar John Holdren is directly involved in CRU’s unfolding Climategate scandal. In fact, according to files released by a CEU hacker or whistleblower, Holdren is involved in what Canada Free Press (CFP) columnist Canadian climatologist Dr. Tim Ball terms “a truculent and nasty manner that provides a brief demonstration of his lack of understanding, commitment on faith and willingness to ridicule and bully people”.

“Baliunas and Soon were authors of excellent work confirming the existence of the Medieval Warm Period (MWP) from a multitude of sources. Their work challenged attempts to get rid of the MWP because it contradicted the claim by the proponents of anthropogenic global warming (AGW). Several scientists challenged the claim that the latter part of the 20th century was the warmest ever. They knew the claim was false, many warmer periods occurred in the past. Michael Mann ‘got rid’ of the MWP with his production of the hockey stick, but Soon and Baliunas were problematic. What better than have a powerful academic destroy their credibility for you? Sadly, there are always people who will do the dirty work.”

Indeed, Holdren’s emails show how sincere scientists would be made into raw “entertainment”.

How the deed was done
“A perfect person and opportunity appeared. On 16th October 2003 Michael Mann, infamous for his lead in the ‘hockey stick’ that dominated the 2001 IPCC Report, sent an email to people involved in the CRU scandal; “

Here we show how Obama’s bully pushed the medieval warm period out of the way for Miichael mann.

I’d be very happy to discuss these issues with you tomorrow. I’m sorry that the tone of this letter is so formal, Tom. Unfortunately, after today’s events, I must assume that any email I write to you may be
subject to FOI requests, and could ultimately appear on McIntyre’s “ClimateAudit” website.

With best personal wishes,

Ben
***********
In my experience one would expect a letter like that to be a defense exhibit. I’ve even seen people write things like that for that express purpose.

Exactly what I see happened. And GHCN does not have the data for RED BLUFF, CA that HARRY_READ_ME.txt cites. In fact, GHCN does not even have the 1891-1931 data available on-site. I just checked.
The 1891 onwards data for RED BLUFF, CA station has only precip, no temps, and it stays that way until mid-1934. Totally out-of-character for a COOP station.
So, Harry has data that nobody else has. How special.
Where is it?

Willis:
This is an extraordinary tale – elegantly retold. As someone above mentioned you have provided the context necessary to pull together one of the primary dimensions of this entire drama: The FOI theme.

A number of things strike me. First, this story is clearly worth some major investigative reporting. It is a shame that so many players appear to be in the bag. It looks like Fox may be the only outlet that would pursue this story and apart from Stoessel, Reilly, Beck et al have the attention span of gnats. Will Andy Revkin want to do the right thing? Will he ask you to prepare a piece for dot.earth?

Second, one theme that emerges clearly in the FOI issue, is that this group has been infected with a belief that you and Steve and others are part of a conspiracy to disrupt the progress of climate science. They simply do not believe that you, Steve and others are a bunch of essentially puzzle freaks who have discovered an area where they can apply their considerable data analysis skills and who have strong feelings about the integrity of science. They appear to have taken whatever happened in tobacco research issues and defined you in the role of the tobacco companies. This belief is so strong that it becomes self-fulfilling. It is pretty close to paranoia – though I think the holes in their research makes the defensiveness somewhat self-serving.

The third point is that they grossly over-estimate their own cleverness and capabilities and even more obviously under-estimate the smarts and capabilities of those requesting information and analyzing the research papers.

I keep reading demands for scientists to be “transparent” and make every last detail of their research to the public.

Ok. But what about the obligation for skeptics to treat that evidence fairly and not twist its meaning?

Case in point: it is clear that “hide the decline” has nothing to do with doctoring evidence, as Willis admits here. Yet the skeptic community continues to trumpet it around as if it mattered.

Also, I like Willis’s diagram of the scientific method here, and he is correct that this is the method scientists use to improve models. But the problem with doing this publicly is that any time a model is found to be in need of improvement, skeptics claim that it proves models are wrong. Case in point: WUWT claimed a paper in Nature which argued that models UNDERestimate positive feedbacks was evidence that global warming wasn’t happening–the exact opposite of the truth. Here’s the paper: http://www.nature.com/ngeo/journal/v2/n8/abs/ngeo578.html

Or take the hockey stick controversy. McIntyre found an error, the error was corrected, and the fundamental shape of the graph was no different. Yet by publicizing the error, without mentioning that the error was non-material to the conclusion that the earth is warming, the public was irreversibly misled. Even yesterday, WUWT claimed that a line of computer code from the hacked CRU files was proof of tampering – before another commenter corrected him and he admitted “I’ll give Dr. Jones and CRU the benefit of the doubt, maybe these are not ‘untowards’ issues.” But people who read the headline will be misled.

This is why the CRU scientists are so loath (loathe?) to publicize all their data – becasue they know it will be twisted and misinterpreted by politically motivated opponents. And as long as the data is available for laypeople (such as myself) to see, it will always be misinterpreted by people without the qualifications to understand it.

So I agree, the data should be public – as long as skeptics agree to behave civilly, and to not DO anything publicly with the data until consulting with the authors and making sure apparent mistakes are actually mistakes.

Last point. While it is important to be skeptical and repeat experiments to make sure they are correct, at some point it’s time to move on. Especially when the results of the experiment are the basis for DECISIONS, there’s a point where you say, “we’re 95% certain we’re right, and it’s time to act.” From this point on, repeating long-debunked claims is counterproductive.

The problem with the CRU is that they believe in making it up first, then “Verification and Validation through Replublication” (the lie oft repeated, and all that)

Which is why they had to control the peer review process.

So,
(1) – controlling the data was necessary to provide support for their hypothesis.
(2) – controlling peer review was to shield their lies from exposure.

Maybe Ellis or someone else, based on the email and other revelations, could come up with a flow chart showing what they actually did, as opposed to what they should have done? It might go a long way to convincing those who can’t understand how they engineered the “consensus,” by showing how they did it.

…This continuing pattern of harassment, as Ben rightly puts it in my opinion, in the name of due diligence is in my view an attempt to create a fishing expedition to find minor glitches or unexplained bits of code–which exist in nearly all our kinds of complex work–and then assert that the entire result is thus suspect….”

Does Mr. Schneider understand how truly absurd that statement is? He writes that “minor glitches and unexplained bits of code” exist in nearly all their work. By his own admission then, the entire result IS suspect — or is sloppy work justified in his world as long as you get the desired results?

This is simply jaw droping stuff Willis. I never actually thought that there was anything conspiratorial going on with the AGW cabal. I thought that the only glue that guided their actions and direction was a commonality of agenda. Now it is only too clear that efforts to fool the public, avoid scientific review, and avoid their legal obligations were in fact directly and explicitly coordinated between them. And despite their protestations, there was far more to their work than just some scientist trying to do their science. You can tell from their references that they were looking forward to having their work converted to public political policy, and that “hide the decline” was most likely the kind of work that they did in many other areas.

“Science works by one person making a claim, and backing it up with the data and methods that they used to make the claim. Other scientists then attack the claim by (among other things) trying to replicate the first scientist’s work. If they can’t replicate it, it doesn’t stand.”

Exactly. And so, rather than give a couple of ‘buddies’ his work for peer review, Willis decided to show his work to the entire world (Steel Greenhouse). Everybody took a swing at him, but he remains standing. That is how science is supposed to work. Great job Willis.

A carefully laid trap, and perfectly baited. Obtain the digital data, doctor it, replace. Lather, rinse, repeat. Anyone now attempting to examine the ‘decline’ will find that the present data supports the cause in a rather weakened state, but warmer that reality nonetheless.
This was done in at least 2 distinct steps.
Witness the twice moved 1900-1999 US temps graph.
HARRY_READ_ME.txt indicates he was already working on version #1 of doctored US data. Data for stations beyond the range of present day “in place” data. Present day data with decades missing, in places necessary to support the agenda hypothesis.
Cooked books.
See, this is exactly the type of things Steven McIntyre was asking for. Where’s the data?
Where are the 2 previous versions of the data that made 1900-1999 US Temps graphs?
Ah, but there’s a problem. That would mean outing cooked books and doctored data.
Busted by HARRY_READ_ME.txt.

This post by Willis Eschenbach is marvelous, outstanding, and goes to the core of the problem better than anything I’ve seen. I think the preliminary charts about the scientific method etc should be taken out because, though well meaning, they may cause many people to miss it, thinking this is a theoretical essay on the scientific method. I almost missed it that way. (I also think you have to be careful with assuming there is ONE scientific method that fits all scientific inquiry, and oversusing the phrase, etc. -but I digress. In any case, everybody understands quite well, after reading this post, that these people are frantically, obsessively, almost paranoically preoccupied with hiding the most fundamental part of their work, namely HOW they obtained the results they say they obtained. It is the height of absurdity and cheek to claim you are exempt from providing the data and mechanism by which you arrived at your results, and at the same time expect that your results be accepted as valid simply on faith.

I think this piece should be widely distributed among informed people because, as someone mentioned, it puts in perfectly coherent context the content of those emails. What these ‘scientists’ are doing is plainly disgusting. They are behaving exactly like a kind of mafia, and this needs to stop. But it will not stop unless they are forced to change their behavior, and this will only happen through legal action.

Don’t count on the main media to do anything other than whitewashing work here. I suspect that the same hidden interests who helped create such pathetic figures as Jones, Mann, Hansen… through increasingly lavish funding (rendering them intellectually corrupt to the point where they themselves cannot see their own corruption) and who have managed to reduce climatology to the petty service of one quaint but not particularly compelling hypothesis — those same interests also control the media, and to a great extent the legislators.

There is a chance, I hope, this will slowly snowball into something impossible to ignore, and maybe some good will come of it. But it won’t happen automatically and the resistence will be enormous.

“Dave Britton, a spokesman for the Met Office which works with the UEA on climate monitoring, said: “We are utterly confident that there was no collusion or manipulation. All the data used was peer reviewed and we are certain it is fully reliable.”

That’s curious. Data is peer-reviewed? And how have they accomplished that, if all the excuses they have given for why data isn’t provided under FOI are true?

My word if these bureaucrats aren’t ringers for some of the unsavory denizens of Dickens’ “Little Dorrit”. If anyone has the time and can unstick their eyes from this potboiler, here’s a scene from chapter 10, which explains the dealings of famous Circumlocution Office:

WAG:
“Case in point: it is clear that “hide the decline” has nothing to do with doctoring evidence, as Willis admits here. Yet the skeptic community continues to trumpet it around as if it mattered.”

You seem to be misinformed on this WAG. “Hide the decline” has everything to do with doctoring the evidence. It is not an issue of plotting surface data and proxy data on the same chart. Mike’s trick was to take surface data and add it to proxy data and to call the entire thing proxy data. Then he claimed that the proxy data agreed with the surface data. This is clearly a fraudulent thing to do. Your site tries to make some kind of explanation about there not being a decline using surface data. This shows that you have completely missed the point. No one has tried to indicate that there was a decline in the surface data. There was a decline in the proxy data, and that decline was doctored to hide it. Your other explanation about the divergence being a known issue is also off base. The divergence of the proxy data is something that has only recently been admitted to by the AGW cabal. The claim that it is only a later 20th century phemomena is completely without merit. No one has given an explanation for why such a divergence should only happen in the later part of the twentieth century. Such divergences could well have happened all through history – basically making dendro data useless. For example, how do we know that the same divergence didn’t happen during the medival warming period?

“McIntyre found an error, the error was corrected, and the fundamental shape of the graph was no different.”

No, you are wrong again. The shape of the graph was different. There was still a kind of hockey stick, but the severity of the blade was diminished and the MWP and LIA were slightly larger. Only someone with an agenda would say that the correction made no difference. But the story didn’t end with Mann’s errors. McIntyre also found that Mann’s primary data source was cherry picked. Mann got his blade from Graybill’s data, and Graybill got his blade by cherry picking strip bark bristlecone pines. When new and more extensive bristlecone samples from the same areas were taken for Lenah Ababneh’s doctoral dissertation, as well as samples taken in Colorado my McIntyre himself, there was no blade in modern times. In other words, the dendro data was unable to reflect the warming that we are suppose to have had in the later part of the 20th century. For a long time the AGW cabal avoided using contemporary dendro data in order to cover up this split.

So you have it backwards. The twisting and interpretation was not done by the skeptics, but by the supposed peer reviewed climate scientists themselves. The solution is to get all the data out there to the public. Let the light of day decide who is twisting and who is telling the truth. Obviously the climate cabal is afraid of the light of day. If someone like McIntyre “twists” their data, they can always refute him and show him for a fool. But their real fear is that they will themselves be shown to be frauds.

“Dave Britton, a spokesman for the Met Office which works with the UEA on climate monitoring, said: “We are utterly confident that there was no collusion or manipulation. All the data used was peer reviewed and we are certain it is fully reliable.”
That’s curious. Data is peer-reviewed? And how have they accomplished that, if all the excuses they have given for why data isn’t provided under FOI are true?

Great point. Britton should be provided with a small list of the excuses, including the one that says they don’t even have a list of the sites currently being used, and that even if they had it, it would not be useful because they “would not be able to link the sites with sources of data.” Wow.

The “decline” is in tree-ring data, not instrumental. At the risk of being rude, you really have no idea what you are talking about.

The issue about the decline is that tree ring data, which they were using to “get rid of the medieval warm period”, showed a temperature decline in the late 20th century where the instruments showed an increase.

If the tree-rings do not match thermometers now, how can they be used to evaluate temperatures up to 1,000 years ago?

That temperatures increased in the latter part of the 20th is accepted by everyone (although Anthony makes a good point that the UHI effect is not properly accounted for).

If that is your site you might save yourself an eventual Emily Litella moment by deleting that post.

“Finally, I know that DEFRA receive Parliamentary Questions from MPs to answer. One of these 2 months ago was from a Tory MP asking how much money DEFRA has given to CRU over the last 5 years. DEFRA replied that they don’t give money – they award grants based on open competition. DEFRA’s system also told them there were no awards to CRU, as when we do get something it is down as UEA!”

Gloating about hiding their funding from Parliament because the requesting Tory MP didn’t know that they are listed under UEA and not CRU… How quaint!

Thanks for posting that. I tried to explain that also, but the internet ate my post.

WAG, the specific data that diverged was the tree-ring data which was the backbone of MBH99 and 99. The tree-ring showed a decline in temperature (if one assumes that it reliably shows temperature at all) when the instruments were showing an increase (it really did get warmer from 1970 to 1998, most everybody knows that). The issue was using the tree-ring data to “get rid of the Medieval Warm Period” if the tree-rings disagreed with the instruments.

But Tilo, there is an explanation for the divergence. CO2 and NOx both fertilize bristlecones, hence the wider rings.

Sallie Baliunas is a wonderful person. She is careful with her science. She has not deserved the persecution she has gotten from the AGW wolves.

I know. That’s why I posted the video of her talking about the old witch-hints. They are so reminiscent of the modern ones, only now they call us “climate criminals” instead of “witches.” Same small minded wolves as in the past, same craving of power and control based on the “religion” of “scientific consensus” they cooked up themselves.

I have been desperately trying to spread word of this story to everyone I know by linking to your outstanding site, Anthony et. al. However, this being a very technical blog, the story is being lost in the jargon for those not in the field of climate science or at least meteorology. Most people I send information to tell me something along the lines of “well I can tell there was some hiding the bad science going on, though I don’t really understand any of the details…” or thereabouts.

We need a non-technical description of the charges the skeptic community is bringing on the University of East Anglia and CRU. We need to take off the scientist hats and put on the public relations hats or we’re never EVER going to get enough people angry to force the powers that be to listen. I don’t suppose you could drum up some sort of summary for the non-insiders? Something that tells the story in the e-mail and program evidence without quoting too heavily from the jargon? Because I don’t think the severity of these offenses is getting through to people outside of meteorology.

My thanks to all who have posted, and a big hat tip to Anthony for providing the site. This is a work in progress, I am re-writing this piece now, with a more information. It is now a Word document, I’ve posted it on my server here. Anyone is free to download it and send it around if they wish, post it on your own site, it’s public domain. It is important that the story get out.

Here’s some further citations to help explain the occurrences:

The Wegman Report, The definitive word on Mann’s mathematical errors in the Hockeystick paper.

Listing of Climate Audit posts on the Freedom of Information Act, Lots of interesting information on various requests and responses. Be patient, CA is loading slowly due to the number of people who are interested in the emails.

“Caspar and the Jesus Paper”, An example of what the “scientists’ did, what Amman and Wahl pulled off with the connivance of Phil Jones and the rest. This is likely the reason for Phil Jones wanting to delete some of the emails.

Fortress Met Office, The ugly story of how the Met Office hid from David Holland’s FOI requests.

Finally, a big thanks to Maurizio Morabito and his excellent blog, where I first posted this. He has been a gentleman about my crossposting it here, and is a good guy.

Sabr Matt. Good point, how can this be explained simply. I have just been watching Glen Beck. Brilliant I thought. I like these technical bloggers even though I am not a technician myself -it’s like looking in on the engine room of a vast ship. You have other people in different parts of the ship who have PR skills, and who feel equally strongly that the taxpayer has and is being ripped off, and are all too ready to support the boffins. Beaver away. Thank god for Anthony Watts et al.

Anthony can I be so bold as to make a addition to your scientific method, I know it’s probably assumed in your diagram, but when I did research we had to document everything we did at each stage. It was a long time ago (I can still see the quill pen and the inkpot on my desk) but before we did anything we documented in our day books exactly what we were doing and why. We then documented how we did it, and what the results were. Afterwards we would transfer the data in our day books to formal documentation for storage.

If anyone wanted to see what we’d done, how we’d done it and what the results were they were readily available. It is the ONLY way for scientific research to be pursued, but these guys, apart from being what the brits call “shifty” when asked for data, seem to have a complete haphazard methodology for producing scientific results.

I have had a couple of exchanges with Courtillot. This is the last of them from March 26, 2007. I sent him a number of papers to read. He seems incapable of grasping the concept of spatial degrees of freedom, and how this number can change according to timescale. I also told him where he can get station data at NCDC and GISS (as I took a decision ages ago not to release our station data, mainly because of McIntyre). I told him all this as well when we met at a meeting of the French Academy in early March.

However it seems to me the HARRY_READ_ME.txt is also critical. Look at the first line :

READ ME for Harry’s work on the CRU TS2.1/3.0 datasets, 2006-2009!

Now other comments in the file make it clear that “HARRY” was trying to recreate CRU’s published results FOR THREE YEARS AND FAILED. Do you see the significance of this?

It is absolutely fatal for CRU and anything they have produced.

What we have here is a documented THREE year effort by a CRU programmer, who has access to all the data, access to all the code, access to all the people who developed the code he could still NOT duplicate CRU’s OWN results. If he can’t it simply means the CRU’s methods can not be duplicated by ANYONE and so anything produced by CRU can not be reproduced.

In the Netherlands, the left wing Environment Minister, mrs. Jacqueline Cramer disapproves the hacking of the CRU files. She regards this as a serious criminal act.
Years ago, in a period that mrs. Cramer was just a mere president of an environment group, she signed a petition in which she approved stealing files frorm government officials and the publication of home adresses of civil servants who were involved in “controversial” studies, like new motorways, port expanding and all those things environmentalists do oppose.

you could do worse than point people at Andrew Bolt’s summary. It is written for a non-technical newspaper readership. I doubt if can be expressed more clearly and simply than this without losing meaning.

This is the first true “smoking gun” proving intent to deceive and withhold information under FOIA.

SPREAD THE NEWS FAR AND WIDE, FOLKS!

Willis is indeed correct to note that transparency (BLOCKED) and the need to permit repeatability is the key to sound science. Your demonstrative exposition, dear Willis, is the deepest and most important to appear so far, IMHO. Words fail me: HATS OFF!!

It is shockingly clear that a very deep seige mentality has been present at CRU for many years. From the email exchanges, your are either with us or against us. Critical scientists are labelled “contrarians”. Their request for data becomes “harassment”. It is easy to see how the first premise leads to the second.

The bad news is, it is very easy for the crew at CRU to spin the harassment argument to the world, as yesterday’s statement indicated. You can see how they duped the FOI officers at UEA. First paint CA as an organisation out to destroy the AGW agenda by trying to find minor flaws in papers. Next show how they have organised FOI campaigns to grind them down in a bureaucratic quagmire. How are the FOI officers to know any different? Only if they had a half hour session with Steve McIntyre. It is the old problem that happens when you only get one side of the story.

Now, unfortunately, they will be having “half hour sessions” with government ministers, convincing them of this campaign of harassment.

Whilst this is indeed a ‘hunt the fox’ jaunt of unparalleled complexity and dexterity in terms of evading tactics, delays, retrenchment and stand-offs, one key question does come into my mind.

‘How does a scientist in this day and age not simply hand over the fruits of 25 years hard labour to competitors who will jump on the bandwagon and trump them with the grant awarding bodies?’

I am minded to think of Bill Gates’ solution to programmes to develop HIV vaccines using his Foundation money. There, he saw that with a winner-takes-all attitude, all approaches would be reluctant to drop their trial hence trial data was likely to be distorted. Gates stated that all selected participants would benefit equally as all had as a goal the worthy aim of developing an HIV vaccine.

It does seem to me that primary credit via senior authorship on ‘seminal’ papers is likely to be a key block to progress here. Ditto membership of key scientific advisory boards.

What further appears clear is that, in all likelihood, CRU were early movers in this field, at a time when the research was relatively low-key and not intrinsic to enormous economic decisions of politicians. Under such circumstances, retaining data and methods in house is OK, if not optimal.

Once, however, CRU has become a global reference centre for data and predictions which lead to Kyoto and Copenhagen, the individual careers of the scientists, whilst important, are irrelevant in the wake of the mind-numbing changes being proposed globally. The sole thing which is important to justify spending trillions of dollars over the next 30 years is that the science is right. And if Jones et al can’t see that, I think they need to stand down.

Where people are coming from is a worry that the whole global warming story is an artefact of how the raw data is manipulated and how the overall constellation of raw data inputs evolves over time. I’m not saying it is or it isn’t, but that key objection MUST be overcome if this argument is to be resolved.

Releasing this data is not like releasing how to make a hydrogen bomb. Now that WOULD be amenable to FOI blocks! But given the studies carried out by McIntyre on previous statistical anomalies and given the examination of the maintenance of weather stations by Watts, it would appear that justifiable worry exists that the rigour of preparing, selecting and maintaining data reference points may (but may not) be insufficiently good for the conclusions drawn to be reliable or justifiable. Those are the reasons for FOI requests and they are justifiable ones.

Politicians must, however, realise that scientists are rather like bulls locking horns to win the cow’s mating rights. Laissez faire isn’t the way ahead here. There needs to be firm leadership and executive direction on this one and it must not only be free of factional interests, it must be SEEN to be free of them too. Only then, will all protagonists be prepared to accept what comes next.

Because what comes next is an impartial arbiter CO-ORDINATING the examination of the datasets and the data manipulations by multiple groups of independent groups. A bit like a public servant holding a public enquiry and soliciting evidence before weighing it up and coming to conclusions.

I do think, very strongly, that unless this takes place then climate science will continue to suffer from this wrangling in public, that the science, reliable or not, will continue to lose favour with the non-specialist public and that, in the case that the warming scientists are right, that the correct actions will not be taken.

I repeat, the egos of CRU scientists are irrelevant in this matter. This is a matter of the highest international stakes on the planet and the decisions which are to be taken must be taken knowing that the scientific community has examined the assumptions made in reaching conclusions and declared them to be sound.

Not to get their next grants or to curry favour with journal editors.

But because on the basis of the best scientific judgement of the day, that conclusion is the right one to be reached.

Now if Copenhagen were to initiate THAT sort of programme, then the carbon footprint left by all the scientists, journos and politicos in their jet planes etc etc etc will have been worth it.

I wonder what odds I would get at the bookmakers of THAT happening, eh?

Your conclusion that the modelling is all about interpreting various statistics of measured physical variables caused a heads-up reaction in me.

As far as the temperature statistic is concerned, it is computed from the intensive variable “temperature” and thus physically representative of nothing save perhaps the thermal state of a number of thermometers.

It doesn’t matter one iota what scientific methodologies are used to analyse this statistic , for it can be shown, absolutely, that it has no meaning physically. You might as well try to infer a climate signal by comparing cloud cover, rainfall measurements, land use parameters etc with the numbers in a telephone book.

At the risk of boring Anthony’s readers again, (Chris Essex and Ross McKitrick will understand), you cannot apply statistical computations on an intensive variable that is physically meaningful.

Temperature is a scale of relative thermal state and a specific temperature measurement is a measure of an objects ranking of that state.

Some years back I posed a question to two honours degree geophysical students – what does 3 Kelvin plus 3 Kelvin add up to.

In terms of physics, and thus the Earth system, the answer is 3K + 3K = 3K.

In terms of numbers, however, 3 + 3 = 6.

What the climate science people don’t seem to understand is that intensive variables, such as temperature, are actually measurements of some physical attribute of a physical object that are used to factor, or weight, an extensive variable of that object to produce a countable, and therefore a physically meaningful, statistic.

Objects?

Air temperatures?

Quite – a Stephenson’s Screen measures the thermal state of a calibrated measuring thermometer in equilibrium with another (physical) object, here a specific volume of air(atmosphere). The air enclosed by the screen? Has it’s volume been measured? Are these devices uniform in size so as to eliminate the sample – volume variance effect? Meaning that this device, as manufactured, is within a specific tolerance?

The whole AGW theory can be refuted on the basis of the methods used to aggregate the raw temperature data.

It isn’t necessary to criticise their analyses of the statistics when the basis of one of those statistics is plainly wrong.

Willis, thank you very much for this lucid and comprehensive summary. While I’ve avoided censuring individual scientists and others over this whole episode until the context is better understood, you have now provided such a context.

I’ve also noticed that people are questioning your motives etc (i think i saw this over at Pielke Jr’s site. I would like to make this public statement. In all the time that I have spent over at climateaudit, i have found Willis Eschenbach to fair, quite knowledgeable about statistics and data analysis, and unfailingly polite and respectful to others.

“I keep reading demands for scientists to be “transparent” and make every last detail of their research to the public.”

As one of the more vocal people doing the asking and as one guy who
clamored for hansen to “free the code” let me say this. The “demand”
is nothing more than asking scientists to do science. Hadcru publish a
claim: “the globe has warmed since 1850” That claim has a nice graph.
I am under no rational obligation whatsoever to accept their claim. very simply, if they don’t release the raw data and the methods ( code) then
I have no reason to believe them. None. The very epistemological foundations of science require transparency and reproduceability.

“Ok. But what about the obligation for skeptics to treat that evidence fairly and not twist its meaning?”

Anybody who wishes to take issue with science presented openly is under the same obligations. You will see me and those like me calling for skeptic papers to post their data and code. The process of open review will converge on the truth. Finally, just because someone may twist your data is no reason not to publish it.

“Case in point: it is clear that “hide the decline” has nothing to do with doctoring evidence, as Willis admits here. Yet the skeptic community continues to trumpet it around as if it mattered.”

Well, read my posts on RC where I say the same thing. “hide the decline” becomes a NON issue when the real data and methods are opened. When those who “hid” the decline explain forthrightly what they did and why.

“Also, I like Willis’s diagram of the scientific method here, and he is correct that this is the method scientists use to improve models. But the problem with doing this publicly is that any time a model is found to be in need of improvement, skeptics claim that it proves models are wrong. Case in point: WUWT claimed a paper in Nature which argued that models UNDERestimate positive feedbacks was evidence that global warming wasn’t happening–the exact opposite of the truth. Here’s the paper: http://www.nature.com/ngeo/journal/v2/n8/abs/ngeo578.html”

Your argument here is this: Scientists should NOT follow the scientific method because if they do, there is no assurance that others will do likewise. That is, you argue that the potential bad behavior by others is reason to engage in the very behavior ( hiding data) that you know to be wrong. In short, hide data because others might also do something wrong.

“Or take the hockey stick controversy. McIntyre found an error, the error was corrected, and the fundamental shape of the graph was no different. Yet by publicizing the error, without mentioning that the error was non-material to the conclusion that the earth is warming, the public was irreversibly misled.”

You categorically misunderstand the issue with the hockey stick. The issue with the hockey stick is NOT the current warming. It’s this:

A. The data is selected to diminish the MWP ( flat shaft)
B. The METHODS underestimate the confidence intervals.

” Even yesterday, WUWT claimed that a line of computer code from the hacked CRU files was proof of tampering – before another commenter corrected him and he admitted “I’ll give Dr. Jones and CRU the benefit of the doubt, maybe these are not ‘untowards’ issues.” But people who read the headline will be misled.”

Can you see how you defeat your very argument. Open code and open debate. One person made a mistake, the next corrected him. This is how
truth and consensus CAN be reached. But without access to the data and access to the code you cant get genuine consensus. You only get sham consensus. Consensus through limiting the ability of others to publish, consensus through kicking editors off journals. consensus through keeping papers out of AR4. consensus manufactured by rigging the publishing system.

“This is why the CRU scientists are so loath (loathe?) to publicize all their data – becasue they know it will be twisted and misinterpreted by politically motivated opponents. And as long as the data is available for laypeople (such as myself) to see, it will always be misinterpreted by people without the qualifications to understand it.”

Actually phil jones is on record saying he will not release data because he is afraid that people will find something wrong with it. And as you show above when one person twists something in an open forum it’s not too long before the mistake gets corrected. On the other hand dr. Mann used a temperature proxy upside down. he did this is the closed world of climate publication. Correcting that took years.

“So I agree, the data should be public – as long as skeptics agree to behave civilly, and to not DO anything publicly with the data until consulting with the authors and making sure apparent mistakes are actually mistakes.”

Well, I refer you to the mistake dr. mann made in one paper about the geographic location of a certain proxy. he was notified in writing. he ignored the correction. he published a second paper repeating the bookkeeping error. Still no correction. Finally, he silently changed the data. made no notice of the change. refused to acknowledge that others had found the error. data should be public. PERIOD. Regardless of whether idiots who don’t believe in AGW misuse it or not. This is not a negotiation about peoples behavior. This is a requirement of science. I have 3 numbers in my head. Their average is 2. Do you believe me? should you? Your argument is this. I should only release those three numbers if and only if the people I give those numbers to promise to behave properly. You are worried that if I tell people the numbers are 1,2 and 3 that some idiot may calculate the average as 6. well, some idiot may misuse my data. I’d say its certain that some idiot will misuse my data. That fact however has no bearing on my obligation to release the data. If I want to convince you with a RATIONAL argument I must release the data. You don’t want to release data unless I get irrational people to agree to use data rationally. That insures that nobody will be convinced of anything.

“Last point. While it is important to be skeptical and repeat experiments to make sure they are correct, at some point it’s time to move on. Especially when the results of the experiment are the basis for DECISIONS, there’s a point where you say, “we’re 95% certain we’re right, and it’s time to act.” From this point on, repeating long-debunked claims is counterproductive.”

yes, there is a point when one says “we are 95% certain”. we will only know if we reach that point when people release the data. I tell you that the average of 1000 data points is 1. Can you guess the confidence intervals?
Nope. not unless i release the data.

Yet all of these nonresponses manage to underscore what may be the most revealing truth: That these scientists feel the public doesn’t have a right to know the basis for their climate-change predictions, even as their governments prepare staggeringly expensive legislation in response to them.

In college we were always forced to display all of our work from beginning to end, including computer code, so it could be properly analyzed for grading. I would assume even stricter standards should apply for science which targets the worlds economics, politics, and energy production.

Some of the mentality you’re seeing at CRU and EAU comes out of the increasing trend to “corporatise” public institutions. To make them somehow be a competitive commercial enterprise operating on a market playing field producing assets rather than knowledge. Once corporatisation has taken hold, then the rationale develops to protect the productivity rather than share the discovery. This has been insidious and pervasive over the last decade, and seems to be characteristic of liberal governments (oddly counter-intuitive to the popular view of conservatives). Canada moved strongly in this direction a decade ago, and references to individuals going “corporate” are common. Its particularly unsettling in the context of departments charged with the responsibility for statutory enforcement, but no less so for technical institutions. Acting in the public interest doesn’t mean pulling the public wool over public eyes.

“My thanks to all who have posted, and a big hat tip to Anthony for providing the site. This is a work in progress, I am re-writing this piece now, with a more information. It is now a Word document, I’ve posted it on my server here. Anyone is free to download it and send it around if they wish, post it on your own site, it’s public domain. It is important that the story get out.”

Willis, while it is appreciated that you put the document up, please change the format to protected pdf or other (somewhat) more secure structure. Word docs are so easily manipulated; you may find that what wrote and what gets circulated may not be even close. Many here can help you with this if you don’t have the capability directly.

Just posted this at realclimate
“Hi Gavin,
Seeing as lots of people complain about the emails being “taken out of context” I would like to point you guys to one of the players in the email drama and his requests for information under FOI Act. His name is Willis Eschenbach. He lays out his emails and requests for information from CRU and their replies in light of some of the leaked emails. Very illuminating about the scientific method.

I see that the warmists’ line is going to be that these were only 3-4 rogue scientists and that the vast majority of scientists’ work still shows global warming (or climate change – well for the latter, how could it not?) Is there any explanation or analysis of the major contributing AGW scientists and who they are outside of CRU and GISS? To what extent did the IPCC science gain independent support, and where did this support come from? My impression is that the vast majority of scientific supporters are actually in the life sciences – biologists are particularly keen on seeing evidence of climate change in the biota. These individuals would be derided as not being climate scientists if on the skeptical side. However, it would be nice to see an analysis of where the most important work in AGW science was done and how reliant it was on Hadley CRU/GISS etc. This sounds like a major project, though.

Tilo Reber,
Thanks for your post about Mann’s hockey stick. I had assumed that the hockey stick was debunked by M&M and Wegman, so I was quite suprised, to say the least, when WAG said it was still the same after the corrections. Now I understand that there were 2 points. 1) After the correction to the algorithms there was a somewhat hockey stick shape, but this was due to 2) cherry picked data. After a wider dataset was used, the hockey stick disappeared altogether.

This is what I love about this site – you always learn something new. It shows how desperate the warmists are to try and resurrect it. The question is why are they doing this?

If anyone wanted to see what we’d done, how we’d done it and what the results were they were readily available. It is the ONLY way for scientific research to be pursued, but these guys, apart from being what the brits call “shifty” when asked for data, seem to have a complete haphazard methodology for producing scientific results.

Great point geronimo. At this point if Jones, Mann, et al had to start from scratch with just the raw data and their notes, could they even replicate their own work? Oops, no raw data, and apparently no notes. According to the emails, Jones doesn’t even seem to know exactly which stations he used to produce the “global average.”

Don’t count the Information Commissioner out just yet. I doubt he was appraised of the full facts of the situation when his advice was sought and he might actually be pretty pissed off he has been used as a smoke-screen.

At the very least one must expect the resignation of Phil Jones but surely questions must be asked of Government Ministers as to how Government Employees are involved in such a massive fraud.

Why is the BBC not covering this? Telegraph, Guardian and many others have written about the fraudulent science and unethical collusion by top “man-made” global warming scientists?

By chance, have you received instructions from the UK Government that the BBC are not allowed to cover this breaking story? (I see that all posts mentioning the content in the hacked emails have been deleted – in an effort to protect the people at CRU – in short it looks like a massive cover up is underway.)

Is the UK Government (which planned to spend 6 Million pounds on advertising that CO2 is a dangerous monster that will drown children’s pets) complicit in this whole endeavor to fabricate propaganda and control “accepted” science to the effect that CO2 is causing dire global warming (something which is patently false)?

Because what comes next is an impartial arbiter CO-ORDINATING the examination of the datasets and the data manipulations by multiple groups of independent groups. A bit like a public servant holding a public enquiry and soliciting evidence before weighing it up and coming to conclusions.

I do think, very strongly, that unless this takes place then climate science will continue to suffer from this wrangling in public, that the science, reliable or not, will continue to lose favour with the non-specialist public and that, in the case that the warming scientists are right, that the correct actions will not be taken.

You do realize that you’re talking multiple years here, and that vested interests will try mightily to skew the results?

Thanks Willis for this. Did a full read of this last night. I’ve been waiting for segments like this that put a chronological context to the story.

I’m struck by the behavior of the Team that most parents will recognize. They spend an immense amount of time, effort and emotion avoiding doing what they’re supposed to do when compliance would be much simpler and quicker. Really childish stuff.

That was a fascinating video about weather cooking by Sally Baliunas. I knew that storms much worse than the present occurred during the LIA, but I didn’t know that their science was so similar to ours. They reasoned that, since some extreme weather events were so unprecedented, they could not be natural. Therefore they must be man-made – more specifically, the work of witches. They called the act of human – or anthropogenic – weather modification ‘weather cooking’.

Today we have something similar. Except that today’s equally scientific theory might well be called ‘climate cooking’.

We know that AGW is misnamed, because greenhouses work by trapping warm and and not by trapping infra red radiation. With that in mind I suggest it should be renamed ACC: Anthropogenic Climate Cooking.
Chris

GISS are HadCRUT use the same raw data and different methods to clean up the raw data. I am not an expert on those methods used, but they aren’t identical. So in the sense that the GISS team and the HadCRUT team made use of independently derived methods to deal with some of the uncertainties and problems with the raw station data, they are indeed independent. However, they use the same raw data (there is only the raw data obtained from each nation in the WMO…it’s not like these countries keep different sets of data to release to different groups), and because James Hansen and the HadCRUT team suffer from samethink, there’s a good probability their methods are at least SIMILAR.

From “The Team’s” perspective, outing all the data and methods would make their lives more difficult. Paying higher taxes and suffering lower economic growth makes my life more difficult. Maybe we should do this the democratic way and have a show of hands to decide?

Let us not let Nature and other tainted journals off the hook. I believe these appalling revelations should impact on the complicit and exclusive journals themselves. Anyone have an idea for dealing with this piece of the calumny?

The way to deal with bad journals is to do what has already been done…and that is create alternative journals. If there are enough climate skeptics with credentials doing science of their own, then the HadCRUT and GISS folks can boycott those journals all they want, they will continue to function and produce quality science literature. The question of how to get governments to listen to the alternative literature is “above my pay-grade”…we are losing the PR war and we need to find a way to reverse that. But that’s a separate issue.

“Beyond honest errors and errors caused through negligence are a third category of errors: those that involve deception. Making up data or results (fabrication), changing or misreporting data or results (falsification), and using the ideas or words of another person without giving appropriate credit (plagiarism)—all strike at the heart of the values on which science is based.

“These acts of scientific misconduct not only undermine progress but the entire set of values on which the scientific enterprise rests. Anyone who engages in any of these practices is putting his or her scientific career at risk…”

going through that harry_read_me file it shows that errors in climatology, like in radiative physics are the maximum that error will allow. Like in radiative physics, they make up the equations as they go along, without any regard for actual measurement and this has been going on for over 100 years. It is this historical antecedent that allows them to continue it with impunity, probably unaware of how derived numbers relate to the physical process, or how temperatures, both absorbed and emitted in fact, relate to molecular composition, density and absorbtion wavelengths than a constant.. This methodology, permits some rather arbitrary extrapolations that can be applied to derive earth temperatures from the past as is seen desirable. If so much theoretical physics is in the wrong, and left unchecked, then likewise, so much climatology will be in the wrong for the same reasons

When a couple of the major media have published a little bit about the scandal I email them to remind them that the stench of this so-called science has been around for quite a while and I consider their coverage now too little and too late.

By their silence they have been complicit in the fraud. Just the lack of coverage of the Wegman Report is enough to indict them. They are being dragged kicking and screaming to cover this now – they’d rather eat green flies, but the public won’t stand for it.

Actually, Vincent, it’s a three part drama. After the Bristlcone pine cherry picking and divergence problem was resolved Mann’s original hockey stick was destroyed. But the assertion that Mann wanted to make with it was still alive. The dendro alarmists claimed that nothing had changed because the hockey stick could still be produced even when you took out the Graybill bristlecone data. They were right. But what basically happened was that the blade that was provided by that series was replaced by the blade that was created by the Briffa Yamal series. This is why McIntyre struggled so long to see why Briffa was able to get results that didn’t seem to be present in other series.

When Briffa was finally forced to release his data several things became clear. First, the sample set that Briffa used was pitifully inadequate for making statements about trends. Everything depended on a handful of trees. Second, the Russians that took the Yamal samples did not find a hockey stick. Third – and Briffa admitted this himself – the series that were used for reconstruction were based upon their agreement with the surface temperature record. So while individual trees were not cherry picked, series were cherry picked. The fourth problem had to do with the growth characteristics of the Larches that were used in the Yamal studies. Their growth tends to increase as they get older. Due to the way that the trees are selected and cored, this means that the rings representing the later part of the twenthieth century all came from old tree rings, while the rings that came from the earlier history were from a mixture of young and old tree rings. And last, there were other, more complete, more comprehensive series available in the same area that actually showed cooling.

When one looked at the various hockey sticks that were created by the hockey team, virtually all of them used either the Graybill series, or the Yamal series, or both. So now, not only is Mann’s hockey stick broken, but virtually all of the hockey sticks that were created by various memebers of the hockey team are broken – particularly those that make up the IPCC hockey stick spagetti graph. Proxies other than tree rings tell a very mixed story, depending on the particular proxy and the type of proxy. What is clear, however, is that the statement that the hockey team seeks to make – that the second half of the twentieth century is unique in the climate record of the past 2000 years – is not even close to accurate.

“We know that AGW is misnamed, because greenhouses work by trapping warm and and not by trapping infra red radiation. With that in mind I suggest it should be renamed ACC: Anthropogenic Climate Cooking.”

It depends what you mean by warm. Presently, physicists give an equal constant value to anything that has the same colour or apparent surface and are able to say that c02 traps any amount of radiation in equal measure. If this constant were the case then oxygen would trap as much heat as c02. The amount of heat absorbed and re-emitted by c02 is a function of its wavelength of absorbtion, as a result of its molecular structure and physical properties – its absorbtion which is between 13.5 and 16 microns – is anything but warm air. This corresponds to freezing temperatures subzero, as would be found in Antarctica or indeed the mid-upper levels of the troposphere. There., however, it tends to be -45C, so there is no way that such cold temperatures could cascade back to earth to make the surface warmer than the nominal 15C

“If all of this sounds incompatible with the process of scientific investigation, it shouldn�t. One of the biggest misconceptions the public has about science is that research is a straightforward process of making measurements, and then seeing whether the data support hypothesis A or B. The truth is that the interpretation of data is seldom that simple.

There are all kinds of subjective decisions that must be made along the way, and the scientist must remain vigilant that he or she is not making those decisions based upon preconceived notions. Data are almost always dirty, with errors of various kinds. Which data will be ignored? Which data will be emphasized? How will the data be processed to tease out the signal we think we see?”

Well, count me amongst those naiively harbouring such misconceptions, and I would probably include most MPs and reporters in that, particularly where such a basic function (from a layman’s point of view) as taking temperatures is concerned.
How can that data be dirty and be subject to various kinds of error? Why on earth would would you ignore some data and emphasise some other? There can only be one way to process data and that is to process it straightforwardly to produce a result and definitely not in such a way as to tease out a signal that we think we see.
The reading is what it is. Modifying and massaging it for whatever reason is dishonest, especially so when that teases out such a footling anomaly as 0.7C in a century, for which we are expected to stand civilisation on it’s head.

BTW, I am not trying to disparage all PhDs – most our very fine, outstanding and admirable people. It is this type you find huddled in the alarmist camps that really give all us propeller heads and geeks a bad rap.

HARRY_READ_ME.txt clearly says that someone has been manually revising the data.

I don’t think that GHCN is usable in it’s raw form so it is inevitable that there must be some “manual (or automatic) revision”. This though is why it is so important to have the raw CRU input data (post revision) in order to replicate the work.

Revision is perhaps the wrong phrase here, what you are looking at is decision. Given four duplicate entries for one year of a station record which will you choose? You may be lucky and they are identical, no problem. What if some agree and some are slightly different? Where a gap appears in the record do you leave it there for a process to deal with later? Discard the entire record? Split it into two? Discard the post gap data but keep the early record?

That is why directing Willis to this or that raw data is useless. Without knowing what decisions were made when cleaning up the record, the final result remains a mystery.

(no idea what WordPress will do to my formatting – apologies in advance)

(If anybody doubts this then I would encourage them to look at the GHCN data as an example. This isn’t any kind of criticism of the GHCN data, it is comprehensive, freely available to all and in the end is what it is.)

Willis, I can appreciate your frustration. Without the final data that goes into the ‘sausage machine’ you can never replicate. I suspect that there a two things going on with CRU.

The Data A long investment in cleaning up raw data from GHCN and elsewhere and they now see it as ‘their’ product. Even though it was funded by the taxpayer.

The Code HARRY_READ_ME.txt points me to the view that unlike GISS they just couldn’t publish a single ‘package’, at least not one that anybody could follow. There is probably much burning of midnight oil every time a HadCRUT update is due out.

Rhys Jaggar (02:11:56) :
‘How does a scientist in this day and age not simply hand over the fruits of 25 years hard labour to competitors who will jump on the bandwagon and trump them with the grant awarding bodies?’
I am minded to think of Bill Gates’ solution to programmes to develop HIV vaccines using his Foundation money.

The comparison is outlandish. If you are a chemist or a lab researcher trying to develop a certain product with certain characteristics, the validity of your work will be demonstrated by whether or not the product you have developed has the characteristics you claim it has. In order to ascertain this, they don’t need to know how you did it. The result validates itself, and how you obtained it is your intellectual property.

If you are processing raw data to create past temperature records, the records you have created do NOT validate themselves. The only way to validate it is to look at the data and how you processed it.

Somebody posted another useful analogy with numbers, that can be rephrased like this. Imagine somebody saying: “I am a scientist. I was given some numbers and I have calculated their average to be 2. But I cannot give you those numbers, or tell you what I did with them, because that would be handing over the fruits of my labor for you to trump it. You will just have to take it on faith that the average I am giving you corresponds to the secret numbers that I am not giving you.”

Say what??
Is the raw data dangerous to public health or to the security of the state? If not, why can’t it be disclosed?
These people are a joke.

Gavin over at realclimate keeps distracting people by saying the issue is the scientists being nasty to each other, and what Trenberth said, and the Nature “trick”, and the like

Dr. Judith Curry, GA. Tech Climatologist, is doing the same thing at CA. It’s a pitiful attempt, but there you are. She’s actually very easily outed and rebutted as to this tactic, and its irrelevance to the real issue. The question is whether she’ll actually do anything to restore real Science. I’m not holding my breath.

I wonder if, as a government organisation, the MET office is required to use data produced to an International Standard (although it seems the UN isn’t too interested in standards either given their use of Wikipedia!)? Obviously, the CRU temperature data relied on by the MET wasn’t produced to any kind of standard whatsoever. FOI request?

I have a somewhat regular critic of my blog who is also a AGW true believer. He only now just got around to commenting on my blog about this. Despite, taking so much time his reply only focused on the personal emails were illegally obtained, Climate Audit and others are pseudo/non-science blogs and Professor Mandia (who commented earlier) was correct.

This would only be hilarious If all this did not affect trillions of dollars of economic activity and the way of life of millions in several countries.

So the consensus is, if you control the data and the right people the science doesn’t matter. This is not my discipline but it’s seems obvious to me that if you can’t replicate your own work, because of lost or unidentifiable source data, you have nothing but a good science fiction novel.

Ask him he has read the Wegman Report and ask him to critique it for you. That’ll keep him busy for a while.

If he cannot understand it, which is likely, perhaps he’ll at least begin to understand that once one disallows the fudged dendrochronology the data does not show that the current warm spell is exceptional.

I would not attempt to debate anyone that has not at least TRIED to read Wegman (not saying I understand it all – I don’t understand PCA except in very broad terms).

Gavin over at realclimate keeps distracting people by saying the issue is the scientists being nasty to each other, and what Trenberth said, and the Nature “trick”, and the like

Dr. Judith Curry, GA. Tech Climatologist, is doing the same thing at CA. It’s a pitiful attempt, but there you are. She’s actually very easily outed and rebutted as to this tactic, and its irrelevance to the real issue. The question is whether she’ll actually do anything to restore real Science. I’m not holding my breath.

J. Peden, what you say is not true at all, and is an attack on on an honest scientist. Dr. Curry is one of the few establishment climate scientists to defend her work at ClimateAudit. She is open and sharing about her work and her data. She openly states her reservations about others work. In my interactions with her, she has been consistently pleasant and open about her beliefs and concerns about the state of climate science.

More to the point, however, I posted on the Climate Audit thread you reference, inviting her to read my account. She did, and posted the following upstream:

Judith Curry (03:46:31) :
Willis, thank you very much for this lucid and comprehensive summary. While I’ve avoided censuring individual scientists and others over this whole episode until the context is better understood, you have now provided such a context.

I’ve also noticed that people are questioning your motives etc (i think i saw this over at Pielke Jr’s site. I would like to make this public statement. In all the time that I have spent over at climateaudit, i have found Willis Eschenbach to fair, quite knowledgeable about statistics and data analysis, and unfailingly polite and respectful to others.

So, quite contrary to your claim, she is one of the few true scientists in the field, who is willing to publicly change her position when new facts emerge. Thank you, Dr. Curry, I can only wish that all climate scientists were as honest and as gracious as you are.

I disagree with you, Willis. But the proof will be in the pudding: will she stand up and immediately reject all Climate Science which does not follow the Scientific Method, specifically as to the transparency involved in contemporaneous code, data, and method release, or will she not?

With some character development and a storyline…. This would be a best selling thriller Willis…. Fact is certainly stranger than fiction;-)

I agree. Your organization of these e-mails into a real story helps the rest of us who’ve been picking up a piece here and there. Better than a collation of disparate e-mails, the chronology helps to see the development. It’s been noted on another thread that whereas the recently-released e-mails may help expose implied malfeasance, intimidation, cover-ups, obfuscation, etc, the motive for e-mail statements can always be argued. E.g., “Of course I warned against answering e-mails from McIntyre, C.A. and their ilk – they had mounted a public campaign which was highly disruptive to the effective workings of our offices…” or whatever the legal defense is for “hiding” what probably should be public knowledge.

But those of you who can read this code must eventually find the evidence of code-manipulations occurring at the same time that these e-mails were being posted, identify the culprits and explain what effects these manipulations had on the graphs released to the public and for policy use.

Multi-dimensional stories of this sort will end this fairytale of AGW, IMO. Already, scientists have no choice but to re-examine their positions. The appeal to fairness and truth already seems to carry a bit more clout than it did a few weeks ago, and as the entrenchment of this bureaucracy is shown, the public will be less and less sympathetic to its claims. The scientists and journals that have colluded in this will be standing there holding their “smoking guns” with nowhere left to hide.

Maybe, too, you technicians who read the code can find in the archives some uncorrupted paleo-data that will help re-establish respectable, long-term records worthy of a new “consensus”.

I am a layperson, but can anyone reconcile the apparent conduct of Professor Mann with the provisions of Penn State’s Faculty Handbook? This seems to be advertent deceit as opposed to research misconduct involving “disputes regarding honest error or honest differences in interpretations or judgments of data”. See

Well as to the flow chart above; i would feedback the “inconsistent with Hypothesis” directly to the “hypothesis” block; after all, that is where hypotheses are dreamed up as a consequence of “observations”; and the outputs of experiments are certainly “observations”

Well, someone sent me a quote from Professor Trevor Davies, Pro-Vice Chancellor for Research and Knowledge Transfer of the EAU. He said:

The University [of East Anglia, home of the CRU] takes its responsibilities under the Freedom of Information Act 2000, Environmental Information Regulations 2004, and the Data Protection Act 1998 very seriously and has, in all cases, handled and responded to requests in accordance with its obligations under each particular piece of legislation.

Ooooh, I am ashamed to confess it, but I said bad words and forbidden epithets, it angrified my blood immensely. Now they’ve gone and done it. I expect crooks to be crooks, that’s no surprise, I’m not angry at Jones et. al. As long as there is a system, people will try to game it.

But I also expect the people in charge to do something when they notice the crooks, not just spout platitudes.

So I’ve just sent the following email:

Dear Professor Davies:

I see that you are quoted regarding the hacked emails as saying

“The University [of East Anglia, home of the CRU] takes its responsibilities under the Freedom of Information Act 2000, Environmental Information Regulations 2004, and the Data Protection Act 1998 very seriously and has, in all cases, handled and responded to requests in accordance with its obligations under each particular piece of legislation.”

If you believe this, I encourage you to read my account of my experience with my FOIA requests to CRU below.

I would greatly appreciate a response, particularly if (as I hope) you are actually taking action to insure that the FOI requests are handled properly in future.

Many thanks,

w.

I attached a copy of my story of my battle with the CRU, which is posted at

Excellent, truly excellent.
It seems to me this story is taking a twist besides climate. And that is the conduct of FOI organization. First, there are many many journalist who are sympathetic to this angle and have had their own severe frustrations with FOI and obtaining information that should rightly be public. Their instinctive reaction is that all information should be public (so they can get a story out of it). So, I think the FOI angle has legs. Official complaints can be made, calls for investigation, noting the similarities between this struggle and some similar problems that journalists have had.
PS Dont forget the National Post in Canada. Its business section, The Financial Post, is a long time climate sceptic.

Willis, you might try e-mailing Marc Morano; and he can probably put you in touch with Senator Inhofe’s assistant who now does what Marc used to do. I’m not at my office so I don’t have Marc’s e-mail handy, but I suspect that Anthony or ChasMod can steer you.

> chasrmartin (16:47:13):
>
> Okay, Steve, you may have found the first shill….

I anticipated this kind of prob the day I obtained the file, so for what its worth…

Note: due to a post on another blog describing the 68MB leaked file, this caution —

1. The original file made available on-or-about 20 Nov 2009, and still available via bittorrent can be identified as follows:
a. name FOI2009.zip
b. size 64,936,854 Bytes (61.93MiB)
c. MD-5 DA2E1D6C453E0643E05E90C681EB1DF4
d. tor Hadley_CRU_Files_(FOI2009.zip).5171206.TPB.torrent

2. The first meme of the Warmists was that the “hacker” probably faked some (or all) of the files in the archive. They have now had sufficient time to create and distribute fake alternatives to poison the well and potentially discredit people who fall for their deception. Take care that all this digging is done on the original data file, and not a “tricked up” version.

4. Most responsible websites have sanitized their posted email extracts to remove actual email addresses etc. to discourage feeding the spambots. Good practice, but it might lead to ambiguity in some cases. If the MD-5 sum of your archive doesn’t match the original you need to figure out why. If the MD-5 is correct, the probability of a doctored version is virtually nil.

In the UK there has been virtualy no television coverage of this story.
The BBC, which is a great champion of the climate change lobby had a 10 min slot on ‘Newsnight’ half of which was spent explaining what climate change was about and not even addressing the main issue.
We have nicknames for the BBC. ‘Pravda’ and ‘Aljabeeba’ being amongst them.
God bless you for that outstanding work.

Hmmmm!!!! seems to work both ways:
from CA
1.- The Team and Pearl Harbor
by Steve McIntyre on March 18th, 2007
2.-Nature “Discovers” Another Climate Audit Finding
by Steve McIntyre on May 28th, 2008
In an article in Nature today by Thompson, Kennedy, Wallace and Phil Jones claim:
Here we call attention to a previously overlooked discontinuity in the record at 1945,
Well, folks, the discontinuity may have been overlooked by Hadley Center, CRU, NOAA and NASA and by the stadiums of IPCC peer reviewers, but it wasn’t overlooked here at Climate Audit.

The bucket problem has been researched since at least 1995!!!!!!!!!!!!!!!!!!!!!!!!!!!

Alleged CRU Email – 1212276269.txt

of data switched from European fleets to almost exclusively US fleets (and
who tended to use engine inlet temperatures rather than canvas buckets).
This offset was large and dramatic and was identified more than ten years
ago from comparisons of simultaneous measurements of night-time marine air
temperatures (NMAT) which did not show such a shift. The experimentally
based adjustment to account for the canvas bucket cooling brought the sea
surface temperatures much more into line with the NMAT series (Folland and
Parker, 1995). (Note that this reduced the 20th Century trends in SST).

More recent work (for instance, at this workshop in 2005), has focussed on
refining the estimates and incorporating new sources of data. For
instance, the 1941 shift in the original corrections, was reduced and
pushed back to 1939 with the addition of substantial and dominant amounts
of US Merchant Marine data (which mostly used engine inlets temperatures).

The version of the data that is currently used in most temperature
reconstructions is based on the work of Rayner and colleagues (reported in
2006). In their discussion of remaining issues they state:

Does anyone know if the British FOI Act maintains the anonymity of requestors?

I am very familiar with the Ontario FOI Act here in Canada (actually called the Freedom of Information and Protection of Privacy Act) and in in, the receiving body of the government does NOT know the ID of the requestor (this is kept only by the FOI office: the people providing the information do not know who is asking, and can’t ask the reason for the request).

I only ask that because the CRU crew seem to know precise details of each request…

Judy Woodruff (also) works for the Jim Lehrer News Hour, a highly respected, balanced, prime time news program on Public Television. She referred to the hacked e-mail story tonight on the News Hour, and the terse response from CRU (roughly quoted) that, “We are sticking with the facts that man-made global warming is an imminent threat to the Earth.”

Disappointing on several fronts, but it’s encouraging to me that the story got some coverage. Lehrer’s program has been a regular forum for pro-global warming stories, including the segments put out by Climate Central.

The hacked e-mail stories are just beginning to be written, as skeptics pour through them, and find out what each has to say about the politicking that went on. There’s time for many more to be written before Copenhagen, and before spring when U.S. and world leaders will meet again.

It’s pretty obvious that this campaign was neither a sincere search for information nor even a fishing expedition.

You had all the information ( see your language above) you requested in your possession. The only rational for your behaviour is an simple attempt to wast the tine and money of the CRU.

John, this is absolutely untrue. We still do not have the raw data, right up to today. Before the poolpah broke around their heads, the most recent excuse for not releasing was that it was “lost”. But that was bullshit, too.

Phil Jones (1255298593)

The original raw data are not lost either. I could reconstruct what we had from some DoE reports we published in the mid-1980s. I would start with the GHCN data. I know that the effort would be a complete wate [waste] of time though. I may get around to it some time.

So no, I’m not trying to “wast the tine and money of the CRU” as you say. I’m just trying to get access to taxpayer funded data.

It will be interesting to know what responses you received from the list above.

A note to John McManus: Agencies which collect data, such as the UN, World Bank, government agencies usually publish data without being asked. If there is something that you don’t find then they’re quite willing to find it.

John McManus is missing the point.
That as an exercise in proving the existance of shady scientists obstructing any alternate research other than their own, Willis Eschenbach has proved it.
Along with the champions from the Left (who historicaly have never been right about anything) we can now prove the scientists who they support are not beyond reproach either.

“Anthony can I be so bold as to make a addition to your scientific method, I know it’s probably assumed in your diagram, but when I did research we had to document everything we did at each stage. It was a long time ago (I can still see the quill pen and the inkpot on my desk) but before we did anything we documented in our day books exactly what we were doing and why. We then documented how we did it, and what the results were. Afterwards we would transfer the data in our day books to formal documentation for storage.

If anyone wanted to see what we’d done, how we’d done it and what the results were they were readily available. It is the ONLY way for scientific research to be pursued, but these guys, apart from being what the brits call “shifty” when asked for data, seem to have a complete haphazard methodology for producing scientific results.”

I second this statement Anthony. I worked in industry managing Quality Control Labs and yes we did have computers in the various plants where I worked. However ALL data was written in bound Lab notebooks each day. At the end of the day all entries were signed and dated and then counter signed and dated by me as manager. Routine internal calibration of the equipment and “official” external calibration was also documented and retained as well as samples of everything we made.

As a lowly BS Chemist, I am stunned to find a UNIVERSITY, full of PhDs, has a much shoddier “Scientific Method” than the various industries where I worked. The whole blasted University needs to be shut down if this is an example of the “Scientific Method” they are teaching. It is worse than having ex-Cons teaching ethics courses!

“…What further appears clear is that, in all likelihood, CRU were early movers in this field, at a time when the research was relatively low-key and not intrinsic to enormous economic decisions of politicians. Under such circumstances, retaining data and methods in house is OK, if not optimal….”

Actually Jaggar, the politics came first. Global warming is just a political whip being used to herd people in the direction wanted. This is why the news media will try to ignore “Climategate”

“The common enemy of humanity is man. In searching for a new enemy to unite us, we came up with the idea that pollution, the threat of global warming, water shortages, famine and the like would fit the bill. All these dangers are caused by human intervention, and it is only through changed attitudes and behavior that they can be overcome. The real enemy then, is humanity itself.”
Book – The First Global Revolution Club of Rome

Maurice Strong, a member of the Club of Rome ran with the above ideas by first organizing the 1972 Stockholm Conference on the environment and more recently by chairing the 1992 Rio summit on the environment.

Maurice Strong “has never stopped pressing for a world where UN resolutions would be enforced as law all over the Earth….

Compare the Club of Rome statement to what Elaine Dewar wrote in Toronto’s Saturday Night magazine about Strong

“It is instructive to read Strong’s 1972 Stockholm speech and compare it with the issues of Earth Summit 1992. Strong warned urgently about global warming, the devastation of forests, the loss of biodiversity, polluted oceans, the population time bomb. Then as now, he invited to the conference the brand-new environmental NGOs [non-governmental organizations]: he gave them money to come; they were invited to raise hell at home. After Stockholm, environment issues became part of the administrative framework in Canada, the U.S., Britain, and Europe.”,/i>http://www.mail-archive.com/ctrl@listserv.aol.com/msg106963.html

Greenpeace is one of the groups whose e-mails were released by the CRU whistleblower. Greenpeace, a very small organization in 1972, was one of the groups invited to the first environmental conference. Greenpeace is now very well funded by the Rockefeller foundations and Maurice Strong is one of the Rockefeller foundation trustees.

Political Conspiracy? — Oh yes in spades. Do you really think all the money spent on “Global Warming Science” for the past forty years would be available if it was not useful to some very powerful people? The same is true of all environmental sciences. Environmentalism would have been squashed big time. Greenpeace would have received no press coverage and would be viewed by the world as a small fringe group of loonies to be laughed at. Our would be world leaders realized Political Activists need to have a ” common enemy of humanity” to rely behind so they could be controlled and used. So the powerful gave them one. Notice how power has been moved from individual voters to NGOs organized and lead by the United Nations.

The proof is in how US (and EU) laws are now being written“The harmonization of laws, regulations and standards between and among trading partners requires intense, complex, time-consuming negotiations by CFSAN officials. Harmonization must simultaneously facilitate international trade and promote mutual understanding, while protecting national interests and establish a basis to resolve food issues on sound scientific evidence in an objective atmosphere. Failure to reach a consistent, harmonized set of laws, regulations and standards within the freetrade agreements and the World Trade Organization Agreements can result in considerable economic repercussions.”International Harmonization – FDA http://www.cfsan.fda.gov/~comm/int-laws.html

The World Trade Organization and the United Nations are now writing the laws for individual nations and the internal governments are rubber stamping them despite the protests from the voters.

To the moderators, Science has to be view with an eye to the political climate just ask Galileo.

note: Stephen Hawking says, “Galileo, perhaps more than any other single person, was responsible for the birth of modern science.” Galileo pioneered “experimental scientific method”

You know the companies offering to buy and sell carbon credits. From what I have seen the balance sheet is not adding up. There are some recently bailed out banks involved what could be the largest ponzi scheme (by definition this fits) ever.

Interesting that ponzi schemes usually “collapse under their own weight”

I put it to the bloggers of the world to get to the bottom of the issue.

“While Gore, a politician turned environmental activist chairs Generation Investment, David Blood, a former CEO of Goldman Sachs Asset Management, is managing partner of the firm, which was founded four years ago.”

Phil Jones et al ought to read section 77 of the FOIA and be very afraid.

77
Offence of altering etc. records with intent to prevent disclosure .(1)
Where— .
(a) a request for information has been made to a public authority, and .

(b) under section 1 of this Act or section 7 of the [1988 c. 29.] Data Protection Act 1998, the applicant would have been entitled (subject to payment of any fee) to communication of any information in accordance with that section, .

any person to whom this subsection applies is guilty of an offence if he alters, defaces, blocks, erases, destroys or conceals any record held by the public authority, with the intention of preventing the disclosure by that authority of all, or any part, of the information to the communication of which the applicant would have been entitled.

(2) Subsection (1) applies to the public authority and to any person who is employed by, is an officer of, or is subject to the direction of, the public authority. .

(3) A person guilty of an offence under this section is liable on summary conviction to a fine not exceeding level 5 on the standard scale. .

(4) No proceedings for an offence under this section shall be instituted— .
(a) in England or Wales, except by the Commissioner or by or with the consent of the Director of Public Prosecutions; .
(b) in Northern Ireland, except by the Commissioner or by or with the consent of the Director of Public Prosecutions for Northern Ireland.