Willis vs. The CRU: A History of (FOI) Evasion

(a guest blog by Willis Eschenbach, originally posted to the Climate Sceptics mailing list. Published almost completely as-is).

An excerpt for those without time to read it all

the issue is not Trenberth or scientists talking smack. It is the illegal evasion of legitmate scientific requests for data needed to replicate a scientific study. Without replication, science cannot move forwards. And when you only give data to friends of yours, and not to people who actually might take a critical look at it, you know what you end up with? A “consensus” …

Freedom of information, my okole…by Willis Eschenbach

People seem to be missing the real issue in the CRU emails. Gavin over at realclimate keeps distracting people by saying the issue is the scientists being nasty to each other, and what Trenberth said, and the Nature “trick”, and the like. Those are side trails. To me, the main issue is the frontal attack on the heart of science, which is transparency.

Science works by one person making a claim, and backing it up with the data and methods that they used to make the claim. Other scientists attack the work by (among other things) trying to replicate the first scientist’s work. If they can’t replicate it, it doesn’t stand. So blocking the FOIA allowed Phil Jones to claim that his temperature record (HadCRUT3) was valid science.

This is not just trivial gamesmanship, this is central to the very idea of scientific inquiry. This is an attack on the heart of science, by keeping people who disagree with you from ever checking your work and seeing if your math is correct.

As far as I know, I am the person who made the original Freedom Of Information Act to CRU that started getting all this stirred up. I was trying to get access to the taxpayer funded raw data that they built the global temperature record out of. I was not representing anybody, or trying to prove a point. I am not funded by Mobil, I’m an amateur scientist with a lifelong interest in the weather. I’m not “directed” by anyone, I’m not a member of a right-wing conspiracy. I’m just a guy trying to move science forwards.

The recent release of the hacked emails from CRU has provided me with an amazing insight into the attempt by Steve McIntyre, myself, and others from CA and elsewhere to obtain the raw station data from Phil Jones at the CRU. We wanted the data that was used to make the global temperature record that is used to claim “unprecedented” global warming. I want to give a chronological account of the interactions. I will reference the email numbers so that people can see the entire emails if they wish. While we don’t know if all of these emails are valid, the researchers involved such as Gavin Schmidt and Michael Mann that clearly indicate that they think they are authentic.

The story actually starts with Warwick Hughes, a climate researcher who had previously been in cordial contact with Phil Jones, the lead researcher of the CRU. I find only one email in the archive (0969308954) where Phil emails Warwick, from 2000. This is in response to some inconsistencies that Warwick had found in Phil’s work:

Warwick Hughes to Phil Jones, September ’04:

Dear Phillip and Chris Folland (with your IPCC hat on),
Some days ago Chris I emailed to Tom Karl and you replied re the grid cells in north Siberia with no stations, yet carrying red circle grid point anomalies in the TAR Fig 2.9 global maps. I even sent a gif file map showing the grid cells barren of stations greyed out. You said this was due to interpolation and referred me to Phillip and procedures described in a submitted paper. In the last couple of days I have put up a page detailing shortcomings in your TAR Fig 2.9 maps in the north Siberian region, everything is specified there with diagrams and numbered grid points.

[1] One issue is that two of the interpolated grid cells have larger anomalies than the parent cells !!!!?????
This must be explained.

[2] Another serious issue is that obvious non-homogenous warming in Olenek and Verhojansk is being interpolated through to adjoining grid cells with no stations, like cancer.

[3] The third serious issue is that the urbanization affected trend from the Irkutsk grid cell neare Lake Baikal, looks to be interpolated into its western neighbour.

I am sure there are many other cases of this, 2 and 3 happening.
Best regards,
Warwick Hughes (I have sent this to CKF)

Phil to Warwick, same email:

Warwick,
I did not think I would get a chance today to look at the web page. I see what boxes you are referring to. The interpolation procedure cannot produce larger anomalies than neighbours (larger values in a single month). If you have found any of these I will investigate. If you are talking about larger trends then that is a different matter. Trends say in Fig 2.9 for the 1976-99 period require 16 years to have data and at least 10 months in each year. It is conceivable that at there are 24 years in this period that missing values in some boxes influence trend calculation. I would expect this to be random across the globe.

Warwick,
Been away. Just checked my program and the interpolation shouldn’t produce larger anomalies than the neighbouring cells. So can you send me the cells, months and year of the two cells you’ve found ? If I have this I can check to see what has happened and answer (1). As for (2) and (3) we compared all stations with neighbours and these two stations did not have problems when the work was done (around 1985/6). I am not around much for the next 3 weeks but will be here most of this week and will try to answer (1) if I get more details. If you have the names of stations that you’ve compared Olenek and Verhojansk with I would appreciate that.

Cheers
Phil

OK, so far we have a couple of scientists discussing issues in a scientific work, no problem. But as he found more inconsistencies, in order to understand what was going on, in 2005 Warwick asked Phil for the dataset that was used to create the CRU temperature record. Phil Jones famously replied:

Subject: Re: WMO non respondo
… Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it. …
Cheers Phil

Hmmm … not good. Science can only progress if there is a free exchange of scientific data. The scientific model works like this:

A scientist makes claims, and reveals the data and methods he used to come to his conclusions.

Other scientists who don’t agree attack the claim by (inter alia) seeing if they can replicate the result, using the first scientist’s data and methods.

If the claims cannot be replicated, the claim is adjudged to be false.

Obviously, if the data or the methods are kept secret, the claims cannot be verified. Attacking other scientist’s claims is what what scientists do. This adversarial system is the heart of science.

When I found out about this, I couldn’t believe it. I thought, a scientist can’t do that, can they? So I wrote to CRU on September 8, 2006, saying:

I would like to obtain a list of the meteorological stations used in the preparation of the HadCRUT3 global temperature average, and the raw data for those stations. I cannot find it anywhere on the web. The lead author for the temperature average is Dr. Phil Jones of the Climate Research Unit.

Many thanks, Willis Eschenbach

I got no response from Phil Jones or anyone at CRU. So I filed a Freedom of Information act request

Now at this point, let me diverge to what was happening at CRU during this time. The first reference to Freedom of Information in their emails is from 2005, before they had received a single request. Immediately, they start to plan how to evade requests should some come in:

Tom Wigley, Former Director CRU, to Phil Jones, 21/01/2005

Phil,
…

I got a brochure on the FOI Act from UEA. Does this mean that, if someone asks for a computer program we have to give it out?? Can you check this for me (and Sarah). [snip]
Thanks,
Tom.

Phil replies to Tom:

Tom,
…
On the FOI Act there is a little leaflet we have all been sent. It doesn’t really clarify what we might have to do re programs or data. Like all things in Britain we will only find out when the first person or organization asks. I wouldn’t tell anybody about the FOI Act in Britain. I don’t think UEA really knows what’s involved.

As you’re no longer an employee I would use this argument if anything comes along. I think it is supposed to mainly apply to issues of personal information – references for jobs etc.

[snip]
Cheers
Phil

So the coverup starts immediately, even before the first request. “I wouldn’t tell anyone about the FOI act in Britain”.

Tom to Phil

Phil,

Thanks for the quick reply. The leaflet appeared so general, but it was prepared by UEA so they may have simplified things. From their wording, computer code would be covered by the FOIA. My concern was if Sarah is/was still employed by UEA. I guess she could claim that she had only written one tenth of the code and release every tenth line.
…
Tom

You can see how they plan to observe the spirit of the FOI Act.

Phil to Tom

Tom,
…
As for FOIA Sarah isn’t technically employed by UEA and she will likely be paid by Manchester Metropolitan University. I wouldn’t worry about the code. If FOIA does ever get used by anyone, there is also IPR to consider as well. Data is covered by all the agreements we sign with people, so I will be hiding behind them. I’ll be passing any requests onto the person at UEA who has been given a post to deal with them.
Cheers
Phil

Phil Jones has just gotten the news, and immediately he starts to plan how he is going to hide from an FOI request.

The next email (1109021312) is later in 2005:

At 09:41 AM 2/2/2005, Phil Jones wrote to Michael Mann :

Mike,
…
Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites – you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? – our does ! The UK works on precedents, so the first request will test it.

We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it – thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who’ll say we must adhere to it !

….

Phil

So now we have two more ways for Phil to hide from the FOI Act … along with a threat to delete the data rather than release it.

Mann replies to Jones:

Thanks Phil,

Yes, we’ve learned out lesson about FTP. We’re going to be very careful in the future what gets put there. Scott really screwed up big time when he established that directory so that Tim could access the data.

Yeah, there is a freedom of information act in the U.S., and the contrarians are going to try to use it for all its worth. But there are also intellectual property rights issues, so it isn’t clear how these sorts of things will play out ultimately in the U.S….
mike

Next, from February 05. Jones to Mann, cc to Hughes and Bradley (co-authors of the “hockeystick” study)

Leave it to you to delete as appropriate !
Cheers
Phil
PS I’m getting hassled by a couple of people to release the CRU station temperature data. Don’t any of you three tell anybody that the UK has a Freedom of Information Act !

The first rule of the Freedom of Information act … nobody talks about the Freedom of Information Act.

With that as a prologue, let me return to my FOI request. On February 10, 2007, I received my reply from Mr. Dave Palmer of CRU:

Dear Mr. Eschenbach

FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04)

Your request for information received on 28 September now been considered and I can report that the information requested is available on non-UEA websites as detailed below.

This page is where you can get one of the two US versions of the global dataset, and it appears that the raw station data can be obtained from this site.

Datasets named ds564.0 and ds570.0 can be found at The Climate & Global Dynamics Division (CGD) page of the Earth and Sun Systems Laboratory (ESSL) at the National Center for Atmospheric Research (NCAR) site at: http://www.cgd. ucar.edu/ cas/tn404/

Between them, these two datasets have the data which the UEA Climate Research Unit (CRU) uses to derive the HadCRUT3 analysis. The latter, NCAR site holds the raw station data (including temperature, but other variables as well). The GHCN would give their set of station data (with adjustments for all the numerous problems).

They both have a lot more data than the CRU have (in simple station number counts), but the extra are almost entirely within the USA. We have sent all our data to GHCN, so they do, in fact, possess all our data.

In accordance with S. 17 of the Freedom of Information Act 2000 this letter acts as a Refusal Notice, and the reasons for exemption are as stated below

Exemption Reason
s. 21, Information accessible to applicant via other means Some information is publicly available on external websites

I was outraged. So the next day, I made a second request:

Dear Mr. Palmer:

Thank you for your reply (attached below). However, I fear that it is totally unresponsive. I had asked for a list of the sites actually used. While it may (or may not) be true that “it appears that the raw station data can be obtained from [GHCN]”, this is meaningless without an actual list of the sites that Dr. Jones and his team used.

The debate about changes in the climate is quite important. Dr. Jones’ work is one of the most frequently cited statistics in the field. Dr. Jones has refused to provide a list of the sites used for his work, and as such, it cannot be replicated. Replication is central to science. I find Dr. Jones attitude quite difficult to understand, and I find your refusal to provide the data requested quite baffling.

You are making the rather curious claim that because the data “appears” to be out on the web somewhere, there is no need for Dr. Jones to reveal which stations were actually used. The claim is even more baffling since you say that the original data used by CRU is available at the GHCN web site, and then follow that with the statement that some of the GHCN data originally came from CRU. Which is the case? Did CRU get the data from GHCN, or did GHCN get the data from CRU?

Rather than immediately appealing this ruling (with the consequent negative publicity that would inevitably accrue to CRU from such an action), I am again requesting that you provide:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and

2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.

I find it somewhat disquieting that an FOI request is necessary to force a scientist to reveal the data used in his publicly funded research … is this truly the standard that the CRU is promulgating?

Thank you for your cooperation in this matter.

Willis Eschenbach

On April 12, 2007, I got my second reply:

In regards the “gridded network” stations, I have been informed that the Climate Research Unit’s (CRU) monthly mean surface temperature dataset has been constructed principally from data available on the two websites identified in my letter of 12 March 2007. Our estimate is that more than 98% of the CRU data are on these sites.

The remaining 2% of data that is not in the websites consists of data CRU has collected from National Met Services (NMSs) in many countries of the world. In gaining access to these NMS data, we have signed agreements with many NMSs not to pass on the raw station data, but the NMSs concerned are happy for us to use the data in our gridding, and these station data are included in our gridded products, which are available from the CRU web site. These NMS-supplied data may only form a very small percentage of the database, but we have to respect their wishes and therefore this information would be exempt from disclosure under FOIA pursuant to s.41. The World Meteorological Organization has a list of all NMSs.

That didn’t help one bit. Without knowing which data was used, it was meaningless. They’ve tried s.21, they’ve tried s.41, neither exemption applies. So the next day, I replied:

While it is good to know that the data is available at those two web sites, that information is useless without a list of stations used by Jones et al. to prepare the HadCRUT3 dataset. As I said in my request, I am asking for:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and

2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.

Without knowing the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service), it is not possible to access the information. Thus, Exemption 21 does not apply – I still cannot access the data.

I don’t understand why this is so hard. All I am asking for is a simple list of the sites and where each site’s data is located. Pointing at two huge piles of data and saying, in effect, “The data is in there somewhere” does not help at all.

To clarify what I am requesting, I am only asking for a list of the stations used in HadCRUT3, a list that would look like this:

etc. for all of the stations used to prepare the HadCRUT3 temperature data.

That is the information requested, and it is not available “on non-UEA websites”, or anywhere else that I have been able to find.

I appreciate all of your assistance in this matter, and I trust we can get it resolved satisfactorily.

Best regards,

I received another letter, saying that they could not identify the locations of the requested information. I wrote back again, saying:

Dear Mr. Palmer:

It appears we have gone full circle here, and ended up back where we started.

I had originally asked for the raw station data used to produce the HadCRUT3 dataset to be posted up on the UEA website, or made available in some other form.

You refused, saying that the information was available elsewhere on non-UEA websites, which is a valid reason for FOI refusals.

I can report that the information requested is not available on non-UEA websites as detailed below.

Your most recent letter (Further _information_ letter_final_ 070418_rev01. doc), however, says that you are unable to identify the locations of the requested information. Thus, the original reason for refusing to provide station data for HadCRUT3 was invalid.

Therefore, since the information requested is not available on non-UEA websites, I wish to re-instate my original request, that the information itself be made available on your website or in some other form. I understand that a small amount of this data (about 2%, according to your letter) is not available due to privacy requests from the countries involved. In that case, a listing of which stations this applies to will suffice.

The HadCRUT3 dataset is one of the fundamental datasets in the current climate discussion. As such, it is vitally important that it can be peer reviewed and examined to verify its accuracy. The only way this can be done is for the data to be made available to other researchers in the field.

Once again, thank you for your assistance in all of this. It is truly not a difficult request, and is fully in line with both standard scientific practice and your “CODE OF PRACTICE FOR RESPONDING TO REQUESTS FOR INFORMATION UNDER THE FREEDOM OF INFORMATION ACT 2000”. I am sure that we can bring this to a satisfactory resolution without involving appeals or unfavorable publicity.

Further to your email of 14 April 2007 in which you re-stated your request to see

“a list of stations used by Jones et al. to prepare the HadCRUT3 dataset” I am asking for: 1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and 2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.”

In your note you also requested “the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service)”,

I have contacted Dr. Jones and can update you on our efforts to resolve this matter.

We cannot produce a simple list with this format and with the information you described in your note of 14 April. Firstly, we do not have a list consisting solely of the sites we currently use. Our list is larger, as it includes data not used due to incomplete reference periods, for example. Additionally, even if we were able to create such a list we would not be able to link the sites with sources of data. The station database has evolved over time and the Climate Research Unit was not able to keep multiple versions of it as stations were added, amended and deleted. This was a consequence of a lack of data storage in the 1980s and early 1990s compared to what we have at our disposal currently. It is also likely that quite a few stations consist of a mixture of sources.

I have also been informed that, as the GHCN and NCAR are merely databases, the ultimate source of all data is the respective NMS in the country where the station is located. Even GHCN and NCAR can’t say with precision where they got their data from as the data comes not only from each NMS, but also comes from scientists in each reporting country.

In short, we simply don’t have what you are requesting. The only true source would be the NMS for each reporting country. We can, however, send a list of all stations used, but without sources. This would include locations, names and lengths of record, although the latter are no guide as to the completeness of the series.

This is, in effect, our final attempt to resolve this matter informally. If this response is not to your satisfaction, I will initiate the second stage of our internal complaint process and
will advise you of progress and outcome as appropriate. For your information, the complaint process is within our Code of Practice and can be found at: http://www1. uea.ac.uk/ polopoly_ fs/1.2750! uea_manual_ draft_04b. pdf

Yours sincerely David Palmer Information Policy Officer University of East Anglia

I loved the story line in this one “we do not have a list consisting solely of the sites we currently use”. Say what? How do they produce updates that change the temperature all the way back to 1870? But I digress …

So I advised him that I was appealing. His letter was passed to a Ms. Kitty Inglis, who replied

Following David Palmer’s letter of 27th April 2007 to you regarding your dissatisfaction with our response to your FOI request of 25th January 2007, I have undertaken a thorough review of the contents of our file and have spoken with both Mr. Palmer and Professor Jones.

As a result of this investigation, I am satisfied that we have done all we can to fulfil [sic] your request and to provide you with the information you require where it is possible for us to do so.

I confirm that we are able to make available on the Climatic Research Unit website a list of stations, including name, latitude, longitude, elevation and WMO number (where available).

We are unable to provide a simple list of sources for these stations as we do not hold this information. Nor do we hold the raw (i.e. unadjusted) station data, as you describe it, at UEA. As stated in prior letters to you, raw station data are available on the NCAR and GHCN websites and gridded data are available on the Climatic Research Unit website. If these data are insufficient for your requirements, you will need to contact the NMS for the country in which the station is located to obtain the information you require.

I hope you are able to accept this response. We have contacted the Information Commissioner’s Office in relation to this matter and their advice is that if you are still dissatisfied with this response, you can, at this time, exercise your right of appeal to the
Information Commissioner by contacting them at:
Information Commissioner’ s Office
Wycliffe House

At that point, I let it go. I had a small victory, we got a list of the stations. It took me a couple more letters to actually get them to post the list. But nothing else.

Meanwhile, behind the scenes at CRU, I now find out that they were circling the wagons … what follows are their internal discussions about a series of FOI requests from myself, Steve McIntyre, Doug Keenan and others to CRU for various data. Phil Jones to Tom Keenan and Wei-Chyung Wang, 6/19/2007:

Wei-Chyung and Tom,
…
1. Think I’ve managed to persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.
2. Had an email from David Jones of BMRC, Melbourne. [EMAIL NOT FOUND IN CRU EMAILS – Willis] He said they are ignoring anybody who has dealings with CA, as there are threads on it about Australian sites.
3. CA is in dispute with IPCC (Susan Solomon and Martin Manning) about the availability of the responses to reviewer’s at the various stages of the AR4 drafts. They are most interested here re Ch 6 on paleo.
Cheers
Phil

Well, that explains a few things … they’ve managed to “persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.” I hadn’t noticed that exemption in the FOI documentation I’d seen. Call me crazy, but I don’t think that’s legal, and it definitely isn’t ethical.

I note that they are circling the wagons in Australia as well …

Phil Jones to Thomas Peterson of NOAA, 6/20/2007 AM (1182342470) :

Tom P.

Just for interest. Don’t pass on.

Might be a precedent for your paper to J. Climate when it comes out. There are a few interesting comments on the CA web site. One says it is up to me to prove the paper from 1990 was correct, not for Keenan to prove we’re wrong. Interesting logic.
Cheers
Phil

Wei-Chyung, Tom,
I won’t be replying to either of the emails below [FROM STEVE MCINTYRE AND DOUG KEENAN], nor to any
of the accusations on the Climate Audit website. I’ve sent them on to someone here at UEA to see if we
should be discussing anything with our legal staff. The second letter seems an attempt to be nice to me,
and somehow split up the original author team. I do now wish I’d never sent them the data after their FOIA
request!

Cheers
Phil

He obviously views sending data in response to an FOIA request as optional.

Thomas Peterson to Jones, same email:

Fascinating. Thanks for keeping me in the loop, Phil. I won’t pass it on but I will keep it in the back of my mind when/if Russ asks about appropriate responses to CA requests. Russ’ view is that you can never satisfy them so why bother to try?

Again, responding to an FOIA request is viewed as optional.

Phil Jones :

…

PS to Gavin – been following (sporadically) the CA stuff about the GISS data and release of the code etc by Jim. May take some of the pressure of you soon, by releasing a list of the stations we use – just a list, no code and no data. Have agreed to under the FOIA here in the UK.

Oh Happy days!

So … that’s why I only got the station list and not the data, just to take the pressure off. Thanks, Phil.

Jones to Bradley and Amman, 5/9/08 (1210341221):

Mike, Ray, Caspar,

A couple of things – don’t pass on either.
…
2. You can delete this attachment if you want. Keep this quiet also, but this is the person [DAVID HOLLAND – Willis] who is putting in FOI requests for all emails Keith and Tim have written and received re Ch 6 of AR4. We think we’ve found a way around this.

Finding ways around FOI requests seems to be a popular sport. This is in reference to Steve trying to get the review comments to Chapter 6 of the UN IPCC Fourth Assessment Report.

Next, here’s the brilliant way that they found around the FOIA, a bombshell of an idea, Jones to Michael Mann, 29 May 2008 (1212063122):

Mike,

Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment – minor family crisis.

Can you also email Gene and get him to do the same? I don’t have his new email address.

We will be getting Caspar to do likewise.

…

Cheers
Phil

Again, call me crazy, but deleting evidence in the face of an FOI request must be illegal. Gene is Eugene Wahl. Of course, what these guys don’t realize is that there are multiple copies of most emails floating around. In some ways, I hope they deleted them, so that it can be proven.

Have a look at Climate Audit. Holland has put all the responses and letters up. There are three threads – two beginning with Fortress and a third later one. Worth saving the comments on a Jim Edwards – can you do this Tim?

I’ve saved all three threads as they now stand. No time to read all the comments, but I did note in “Fortress Met Office” that someone has provided a link to a website that helps you to submit FOI requests to UK public institutions, and subsequently someone has made a further FOI request to Met Office and someone else made one to DEFRA. If it turns into an organised campaign designed more to inconvenience us than to obtain useful information, then we may be able to decline all related requests without spending ages on considering
them. Worth looking out for evidence of such an organised campaign.

Tim

Another thing to hide behind, a false claim of an “organised campaign” …

Gavin,
…
Keith/Tim still getting FOI requests as well as MOHC and Reading. All our FOI officers have been in discussions and are now using the same exceptions not to respond – advice they got from the Information Commissioner. As an aside and just between us, it seems that Brian Hoskins has withdrawn himself from the WG1 Lead nominations. It seems he doesn’t want to have to deal with this hassle.

The FOI line we’re all using is this. IPCC is exempt from any countries FOI – the skeptics have been told this. Even though we (MOHC, CRU/UEA) possibly hold relevant info the IPCC is not part our remit (mission statement, aims etc) therefore we don’t have an obligation to pass it on.

Cheers
Phil

So now the Information Commissioner is in on the deal, s/he’s advising them to use the same exceptions not to respond.

My personal opinion is that both FOI requests (1) and (2) are intrusive and unreasonable. Steven McIntyre provides absolutely no scientific justification or explanation for such requests. I believe that McIntyre is pursuing a calculated strategy to divert my attention and focus away from research. As the recent experiences of Mike Mann and Phil Jones have shown, this request is the thin edge of wedge. It will be followed by further requests for computer programs, additional material and explanations, etc., etc.

Quite frankly, Tom, having spent nearly 10 months of my life addressing the serious scientific flaws in the Douglass et al. IJoC paper, I am unwilling to waste more of my time fulfilling the intrusive and frivolous requests of Steven McIntyre. The supreme irony is that Mr. McIntyre has focused his attention on our IJoC paper rather than the Douglass et al. IJoC paper which we criticized. As you know, Douglass et
al. relied on a seriously flawed statistical test, and reached incorrect conclusions on the basis of that flawed test.

I believe that our community should no longer tolerate the behavior of Mr. McIntyre and his cronies. McIntyre has no interest in improving our scientific understanding of the nature and causes of climate change. He
has no interest in rational scientific discourse. He deals in the currency of threats and intimidation. We should be able to conduct our scientific research without constant fear of an “audit” by Steven McIntyre; without having to weigh every word we write in every email we send to our scientific colleagues.

In my opinion, Steven McIntyre is the self-appointed Joe McCarthy of climate science. I am unwilling to submit to this McCarthy-style investigation of my scientific research. As you know, I have refused to
send McIntyre the “derived” model data he requests, since all of the primary model data necessary to replicate our results are freely available to him. I will continue to refuse such data requests in the
future. Nor will I provide McIntyre with computer programs, email correspondence, etc. I feel very strongly about these issues. We should not be coerced by the scientific equivalent of a playground bully.

I will be consulting LLNL’s Legal Affairs Office in order to determine how the DOE and LLNL should respond to any FOI requests that we receive from McIntyre. I assume that such requests will be forthcoming.

I am copying this email to all co-authors of our 2008 IJoC paper, to my immediate superior at PCMDI (Dave Bader), to Anjuli Bamzai at DOE headquarters, and to Professor Glenn McGregor (the editor who was in
charge of our paper at IJoC).

I’d be very happy to discuss these issues with you tomorrow. I’m sorry that the tone of this letter is so formal, Tom. Unfortunately, after today’s events, I must assume that any email I write to you may be
subject to FOI requests, and could ultimately appear on McIntyre’s “ClimateAudit” website.

With best personal wishes,

Ben

Well, he got the last paragraph right, at least. He also thinks that an FOIA request must serve some “scientific justification”, as determined by … well … by the person receiving the request.

Ben Santer to Tom Wigly, 12 Dec 07 (1228330629):

At 01:17 03/12/2008, Ben Santer wrote:

Dear Tom,
…
One of the problems is that I’m caught in a real Catch-22 situation. At present, I’m damned and publicly vilified because I refused to provide McIntyre with the data he requested. But had I acceded to McIntyre’s initial request for climate model data, I’m convinced (based on the past experiences of Mike Mann, Phil, and Gavin) that I would have spent years of my scientific career dealing with demands for further explanations,
additional data, Fortran code, etc. (Phil has been complying with FOIA requests from McIntyre and his cronies for over two years). And if I ever denied a single request for further information, McIntyre would have rubbed his hands gleefully and written: “You see – he’s guilty as charged!” on his website.

You and I have spent over a decade of our scientific careers on the MSU issue, Tom. During much of that time, we’ve had to do science in “reactive mode”, responding to the latest outrageous claims and inept science by John Christy, David Douglass, or S. Fred Singer. For the remainder of my scientific career, I’d like to dictate my own research agenda. I don’t want that agenda driven by the constant need to respond to Christy, Douglass, and Singer. And I certainly don’t want to spend years of my life interacting
with the likes of Steven McIntyre.

I hope LLNL management will provide me with their full support. If they do not, I’m fully prepared to seek employment elsewhere.

With best regards,
Ben

Dr. Santer, here’s a novel idea. Put enough information out when you publish the work so that your work can be replicated. Put on the web whatever is necessary in the way of code, data, and methods to allow your work to be checked by someone else. If you do that, not only will you not be bothered, but you will be following the scientific method. None of us at CA are doing this to harass anyone, as you claim. We’re doing this because we cannot replicate your work, and thus your work is purely anecdotal rather than scientific.

Phil responds (same email):

Cc: mann , Gavin Schmidt, Karl Taylor, peter gleckler

Ben,
When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions – one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school – the head of school and a few others) became very supportive. I’ve got to know the FOI person quite well and the Chief Librarian – who deals with appeals. The VC is also
aware of what is going on – at least for one of the requests, but probably doesn’t know the number we’re dealing with. We are in double figures.

One issue is that these requests aren’t that widely known within the School. So I don’t know who else at UEA may be getting them. CRU is moving up the ladder of requests at UEA though – we’re way behind computing though. We’re away of requests going to others in the UK – MOHC, Reading, DEFRA and Imperial College.
So spelling out all the detail to the LLNL management should be the first thing you do. I hope that Dave is being supportive at PCMDI. The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian, saying that the email maligned his scientific credibility with his peers!

If he pays 10 pounds (which he hasn’t yet) I am supposed to go through my emails and he can get anything I’ve written about him. About 2 months ago I deleted loads of emails, so have very little – if anything at all. This legislation is different from the FOI – it is supposed to be used to find put why you might have a poor credit rating !

In response to FOI and EIR requests, we’ve put up some data – mainly paleo data. Each request generally leads to more – to explain what we’ve put up. Every time, so far, that hasn’t led to anything being added – instead just statements saying read what is in the papers and what is on the web site! Tim Osborn sent one such response (via the FOI person) earlier this week. We’ve never sent programs, any codes and manuals.

In the UK, the Research Assessment Exercise results will be out in 2 weeks time. These are expensive to produce and take too much time, so from next year we’ll be moving onto a metric based system. The metrics will be # and amounts of grants, papers and citations etc. I did flippantly suggest that the # of FOI requests you get should be another.

When you look at CA, they only look papers from a handful of people. They will start on another coming out in The Holocene early next year. Gavin and Mike are on this with loads of others. I’ve told both exactly what will appear on CA once they get access to it!

Cheers
Phil

Well, that explains why David Palmer and Ms. Kitty Inglis, the Chief Librarian, were so unsupportive. Took a couple of half hour sessions, but at the end of that, rather than being a representative of the FOI process, they were functioning as the personal representatives of Phil Jones. We have a new reason I hadn’t noticed in the FOI law for refusing a request, because the requester posts at CA.

Jones to Ben Santer again, 10 Dec 2008:

Ben,

Haven’t got a reply from the FOI person here at UEA. So I’m not entirely confident the numbers are correct. One way of checking would be to look on CA, but I’m not doing that. I did get an email from the FOI person here early yesterday to tell me I shouldn’t be deleting emails – unless this was ‘normal’ deleting to keep emails manageable! McIntyre hasn’t paid his £10, so nothing looks likely to happen re his Data Protection Act email.
Anyway requests have been of three types – observational data, paleo data and who made IPCC changes and why. Keith has got all the latter – and there have been at least 4. We made Susan aware of these – all came from David Holland. According to the FOI Commissioner’ s Office, IPCC is an international organization, so is above any national FOI. Even if UEA holds anything about IPCC, we are not obliged to pass it on, unless it has anything to do with our core business – and it doesn’t! I’m sounding like Sir Humphrey here! McIntyre often gets others to do the requesting, but requests and responses all get posted up on CA regardless of who sends them.

On observational data, there have been at least 5 including a couple from McIntyre. Others here came from Eschenbach and also Douglas Keenan. The latter relate to Wei-Chyung Wang, and despite his being exonerated by SUNY, Keenan has not changed his web site since being told the result by SUNY! [1]http://www.informat h.org/

The paleo data requests have all been to Keith, and here Tim and Keith reply. The recent couple have come from McIntyre but there have been at least two others from Holland. So since Feb 2007, CRU is in double figures. We never get any thanks for putting things up – only abuse and threats. The latest lot is up in the last 3-4 threads on CA.

I got this email over the weekend – see end of this email. This relates to what Tim sent back late last week. There was another one as well – a chatty one saying why didn’t I respond to keep these people on CA quiet. I’ve ignored both. Finally, I know that DEFRA receive Parliamentary Questions from MPs to answer. One of these 2 months ago was from a Tory MP asking how much money DEFRA has given to CRU over the last 5 years. DEFRA replied that they don’t give money – they award grants based on open competition. DEFRA’s system also told them there were no awards to CRU, as when we do get something it is down as UEA!

Since he and Mann and the others have already deleted their emails, looks like David Palmer was a bit too late with his excellent advice … however, I did get a “Mentioned In Dispatches”, at least …

Phil Jones to Raymond Pierrehumbert, 16 Jan 09 (1200493432):

Cc: Michael Mann , Gavin Schmidt

Ray,
…

I have had a couple of exchanges with Courtillot. This is the last of them from March 26, 2007. I sent him a number of papers to read. He seems incapable of grasping the concept of spatial degrees of freedom, and how this number can change according to timescale. I also told him where he can get station data at NCDC and GISS (as I took a decision ages ago not to release our station data, mainly because of McIntyre). I told him all this as well when we met at a meeting of the French Academy in early March.
…
Cheers, Phil

This is a very clear statement of what he has done. He has refused to release the data, not because there is any logical reason to do so, but “because of McIntyre”. This is shameful, and the fact that the Dave Peters and Kitty Inglis went along with this is dereliction of duty.

“Thanks” Ben for this, hi all and happy new year. I had a similar experience– but not FOIA since we at Climatic Change are a private institution- -with Stephen McIntyre demanding that I have the Mann et al cohort publish all their computer codes for papers published in Climatic Change. I put the question to the editorial board who debated it for weeks. The vast majority opinion was that scientists should give enough information on their data sources and methods so others who are scientifically capable can do their own brand of replication work, but that this does not extend to personal computer codes with all their undocumented sub routines etc. It would be odious requirement to have scientists document every line of code so outsiders could then just apply them instantly. Not only is this an intellectual property issue, but it would dramatically reduce our productivity since we are not in the business of producing software products for general consumption and have no resources to do so. The NSF, which funded the studies I published, concurred–so that ended that issue with Climatic Change at the time a few years ago.

This continuing pattern of harassment, as Ben rightly puts it in my opinion, in the name of due diligence is in my view an attempt to create a fishing expedition to find minor glitches or unexplained bits of code–which exist in nearly all our kinds of complex work–and then assert that the entire result is thus suspect. Our best way to deal with this issue of replication is to have multiple independent author teams, with their own codes and data sets, publishing independent work on the same topics–like has been done on the “hockey stick”. That is how credible scientific replication should proceed.

Let the lawyers figure this out, but be sure that, like Ben is doing now, you disclose the maximum reasonable amount of information so competent scientists can do replication work, but short of publishing undocumented personalized codes etc. The end of the email Ben attached shows their intent–to discredit papers so they have no “evidentiary value in public policy”–what you resort to when you can’t win the intellectual battle scientifically at IPCC or NAS.

Good luck with this, and expect more of it as we get closer to international climate policy actions, We are witnessing the “contrarian battle of the bulge” now, and expect that all weapons will be used.

Cheers, Steve

PS Please do not copy or forward this email.

Now, why would Dr. Schneider not want his email copied or forwarded … perhaps because he is saying don’t follow the spirit of the law, do as little as you possibly can? He also foolishly thinks that studies can be “replicated” by using different data and different codes … but that says absolutely nothing about the original study and whether it contains any mistakes.

The researchers complain in various places that they do not want to reveal their “primary data” because it is available on the web. While this is often true, as I saw in my FOIA requests to CRU, just saying “I got the information from X” is often totally inadequate. Santer made this charge, that anyone could go the CMIP website and get the data themselves … but unless he says exactly which data from which run of which model, that information is meaningless.

The main impression that I get from the emails is that the various scientists think that I and others are simply doing this to harass them. Nothing could be further from the truth. I have no desire to put any scientist to any extra effort beyond providing what science requires – a full accounting of the data, the methods, and in some cases the computer code used to do the research. Anything more is harassment … but anything less is scientific obstruction.

As I said, the issue is not Trenberth or scientists talking smack. It is the illegal evasion of legitmate scientific requests for data needed to replicate a scientific study. Without replication, science cannot move forwards. And when you only give data to friends of yours, and not to people who actually might take a critical look at it, you know what you end up with? A “consensus” …

0 Replies to “Willis vs. The CRU: A History of (FOI) Evasion”

You have to admit, hearing a self-admitted amateur scientist claim that his intent is legitimate peer review is rather humorous. =)

Speaking from a completely moral standpoint, and ignoring the information office’s claim that the policies it suggested earlier to the scientists involved were actually illegal after the fact – why should scientists produce data for cranks? Certainly there is very little valid review of the science going on in the denialist sphere.

I must say, after reading this, that I have much more sympathy with Jones et al. They were clearly being harassed by the likes of Eschenbach and McIntyre, who don’t actually demonstrate any understanding of the science or produce any scientific results themselves.

There is no demonstration that any laws were broken, and even if Jones handled it badly, I can understand why he would want to get on with his work and deal with scientists rather than boys.

I must say, after reading this, that I have much more sympathy with Jones et al. They were clearly being harassed by the likes of Eschenbach and McIntyre, who don’t actually demonstrate any understanding of the science or produce any scientific results themselves.

There is no demonstration that any laws were broken, and even if Jones handled it badly, I can understand why he would want to get on with his work and deal with scientists rather than boys.

Harrassed? I have harassed no one. I filed one FOIA request for his raw data. GHCN and GISS post their raw data on the web. Revealing your data and methods is PART OF EVERY SCIENTIST’S JOB DESCRIPTION. Yes, Jones would have been very happy not to deal with people asking for some kind of scientific support for his claims … but that’s what scientist’s do. They go around trying to knock down other people’s claims.

And since the “scientists” Jones deal with seem to do nothing but pat each other on the back, of course Jones would rather deal with them rather than us bad boys who actually want to see some substance.

Finally, before you say I’m not a scientist, you might want to take a look at my latest post at WUWT called The Smoking Gun at Darwin Zero. I think it is an interesting tale of scientific malfeasance.

I think that it might be instructive if less notice be taken of possible physical changes in the condition of Earth, and more notice taken of possible biological and chemical changes. It seems well documented that the Carbon Dioxide [CO2} content of the atmosphere has changed, is changing and will continue to change – in an upward direction. I make no comment on the cause of that change, but its biological and chemical effects, especially on Virus activity, may be much more important than any amount “warming” ??
Climategate,
While the Earth’s atmosphere may or may not be warming due to Mankind, and Vicky Pope – whom I have known since she was but a gleam in her father’s eye – thinks so, there may well be a much more immediate and universal peril confronting us. More to do with Chemistry and Biology than Physics.

I suggest that one aspect of Science in connection with Climate Change has received scant attention so far seehttp://www.hm-treasury.gov.uk/d/climatechange_grunhausproject.pdf especially this
Effects of CO2 on Mammalian Organisms Report of a Workshop, 5-6 June 1980, Bethesda, Maryland USA, published Dec 1982 by the US Dept of Energy, ref CONF-800249 (with disclaimer) 24 US and one European scientist attended. (Undersea Medical Society Inc, 9650 Rockville Pike,Bethesda, Maryland, 20014, USA). Page 10-3 possible effects on enzyme systems which are pH dependant ? Pages 10-6 to 10-13 Malignancies reference to lymphoma and mammary gland lymphoma effects. (12
references). Page 12-2 possible changes in blood pH.

Virus’s don’t replicate on their own – nor do they replicate in the atmosphere. They only replicate once they are inside a biological organism’s cells.

In other words, the atmosphere has no effect on virus replication, nor the enzymes involved. Those are all within the biological organism, all of which have incredibly robust mechanisms to control pH.

As to the Ukraine situation, even from the link you provided, the issue is mutation of the virus, not any atmospheric effect. Mutations occur as the virus replicates – again, within cells within highly pH controlled biological organisms.

To try to extrapolate a supposed atmospheric acidification (where did you get that stretch??) to changes in viral replication reminds me of a story I saw once that is very indicative of a gross misunderstanding of science. A study was done putting sperm into test tubes with a particular herb. When the sperm didn’t fare well and started dying, the article trumpted how the herb may reduce human fertility. EXCUSE ME??? The conclusion is utterly absurd and not at all supported by tne experiment itself. First, god knows what proportions were in that test tube – levels of something that can be optimal for growth can be highly deadly if too high, or also deadly if too low. Next, when a person ingests a herb, you first get huge transformations chemically, then thru digestion, then who knows what you have that actually gets into the bloodstream but its sure not the same as the raw herb. Then thru the bloodstream into select parts of the body and again a transformation there… then there’s the whole sperm generation process which the ingested herb may not have any effect on whatsoever… You can NEVER extrapolate in vitro (out of the body, test tube, etc) results with in vivo (in the body) results. NEVER. They’re totally different environments and processes.

Same with the idea of viral replication – its in vivo. Atmospheric effects would be in vitro, so to speak.

The very premise of any atmospheric acidification from higher CO2 levels is awfully questionable, though. Our earths atmosphere is incredibly complex and there are all sorts of mechanisms that work to mainatin homeostasis – e.g., no significant change. There are a million different things that would interact and work to negate any pH changes. I’m quite certain that its far too complex for us to even accurately describe, let alone assume that there’s some linear or direct link between increased CO2 and atmospheric acidification.

Remember one of the key rules of thumb in science: Correlation is NOT causation. If it were, then all we’d have to do to cut the human suicide rate in half would be make everyone drink at least two cups of coffee a day. Yes, there was a study many years ago that found that people who drink two or more cups of coffee a day are half as likely to commit suicide as people who drank less than 2 cups. There are all sorts of other, probably even better, examples of correlations that are pretty clearly not indicative of any real tie to the cause.

Meanwhile, when it comes to CO2 there’s the question of whether CO2 has anything to do with temperature increases or whether the reverse is the situation – e.g., increased temperatures cause increased CO2 levels. The latter actually makes far more sense. Why? Because as temperatures rise, the oceans will off gas more CO2. Reconstructions also seem to show CO2 lagging temperature increases, rather than leading them. Also, ask yourself this – if increased CO2 causes significant temperature increases, how did temperatures in the past turn around and drop, even into ice ages? What, historically, would have allowed temperatures to drop during high CO2 levels? Or, another scenario, what histrorically would have caused CO2 levels to drop, if the sensitivities and forcings are all such that we risk runaway global warming? How did we get the huge swings in the past before man was emitting any CO2 to speak of??

Until these sorts of questions can be answered with some accuracy and robustness, scientifically, its absurd to try to say there is any substantial or meaninful scientific basis to this entire issue. Sure, there are various research papers, with various levels of merit – but there is nowhere near the body of scientific evidence to even begin to go down this road – let alone speeding along it far down the line at 200 mph as we’ve been with AGW in the last number of years.

From the USA government ALONE, $79 BILLION has been spent on AGW research and so on. I don’t think that even includes the $6 or $7 Billion given to one of the major car companies this year to retool to be CO2 and green friendly, and somewhat lower sums to the other brand. That’s insanity; Just think what could have been directly accomplished to the good of huge numbers of people with that money. Not to mention the large numbers harmed, even killed, by food prices being doubled in many of the poorest countries because of the agricultural products diverted to bio-fuels – which, as it turns out, if you actually calculate the CO2 footprint, aren’t any better than just burning the fossil fuels!!

Not to mention that CO2 levels have been vastly higher in the past than they are today – much of our evolution was under higher CO2 levels. Current day plants grown in greenhouses grow far better when they increase the CO2 levels up tp 1000 ppm – above that they still grow very well, just not any better than at 1000 ppm, so there’s no point in pumping the level up higher. What does this imply? That there’s no problem with higher levels, far higher levels than we are in today, in terms of biological organisms. Which makes sense, considering our evolutionary history. Historically, some of the most lush, most biologically abundant, diverse, and robust times were when both temperatures and CO2 levels were far higher than today.

In the recent history, some of the most rapid advancement in agriculture, technology, living standards, nutrition, were all during times when temperatures were HIGHER than they currently are today.

The IPCC is a political body that is agenda based – what agenda? To move environmentalism and themselves into a position of world power and global governance. You can easily find this information if you just research the foundation of these groups and how the IPCC works. Non-scientists write the summaries and often significantly change what the scientists have prepared. That’s NOT peer reviewed, and that’s NOT science.

These are the same scurrilous tactics used by the gun grabbers here in the US. And the global warmers. And whatever other Far Left cause you can name. Their data always “got destroyed in a fire” or “washed away in a flood.”

These people do not play by the rules. They are convinced they’re above the rules, smarter than the rest of us and snootily beyond our reach. Anyone who refuses to share data paid for by the taxpayers should go to jail.

I cannot even believe all the data and models aren’t in the public domain already since the politicians are rushing to get global taxation in place. It is amazing it has all been on faith. I never believed the models could work but I can’t believe even that data has been a secret. What a bunch of baloney. Is that a scientific term? These guys remind me of that old comedian Professor Erwin Correy who used to use fake words and mumbles to confuse people.

quote:
“The first reference to Freedom of Information in their emails is from 2005, before they had received a single request.”

That would make sense, since the UK FOIA was first introduced in January of (you guessed it).. 2005. Back then, every scientist in the country was talking about the FOIA and how they could protect their research from it.

Ryan, thanks for the question. I wish it were that easy. What you have linked to is the result of their (unknown) mathematical operations on the raw data. What we are still missing is the raw data, and the computer code that they used to convert the raw data into the finished products that you reference above.

Well, someone sent me a quote from Professor Trevor Davies, Pro-Vice Chancellor for Research and Knowledge Transfer of the EAU. He said:

The University [of East Anglia, home of the CRU] takes its responsibilities under the Freedom of Information Act 2000, Environmental Information Regulations 2004, and the Data Protection Act 1998 very seriously and has, in all cases, handled and responded to requests in accordance with its obligations under each particular piece of legislation.

Ooooh, I am ashamed to confess it, but I said bad words and forbidden epithets, it angrified my blood immensely. Now they’ve gone and done it. I expect crooks to be crooks, that’s no surprise, I’m not angry at Jones et. al. As long as there is a system, people will try to game it.

But I also expect the people in charge to do something when they notice the crooks, not just spout platitudes.

So I’ve just sent the following email:

Dear Professor Davies:

I see that you are quoted regarding the hacked emails as saying

“The University [of East Anglia, home of the CRU] takes its responsibilities under the Freedom of Information Act 2000, Environmental Information Regulations 2004, and the Data Protection Act 1998 very seriously and has, in all cases, handled and responded to requests in accordance with its obligations under each particular piece of legislation.”

If you believe this, I encourage you to read my account of my experience with my FOIA requests to CRU below.

I would greatly appreciate a response, particularly if (as I hope) you are actually taking action to insure that the FOI requests are handled properly in future.

ali baba (19:40:55), I fear you are not understanding the problem. You say

While it may (or may not) be true that “it appears that the raw station data can be obtained from [GHCN]“, this is meaningless without an actual list of the sites that Dr. Jones and his team used.

Exactly what I said, “an actual list of the sites that Dr. Jones and his team used” are unnecessary to independently replicate the results. Do you know what what replication is? It doesn’t mean to duplicate, i.e. “audit”, which is only useful to confirm fraud.

In science, “replicate” means to do the experiment over again. This does not mean to take different data and do different things to it. It means to take the same data, and do the same things the original scientist said he did to it, to see if the results are the same.

Unfortunately, without knowing either what data CRU used, or what they did to the data, that is not possible. Since their results can’t be verified, they are not scientist, merely anecdote.

Even if your claim were correct, and what we are talking about were an audit, which is “only useful to confirm fraud”, what on earth do you think we are dealing with here? Innocent error???

In science, “replicate” means to do the experiment over again. This does not mean to take different data and do different things to it. It means to take the same data, and do the same things the original scientist said he did to it, to see if the results are the same.

No, this is high school stuff, you have to independently reproduce the data, which takes work but is obviously much more powerful than your proposal to look over the original scientist’s shoulder. Science does not advance by zero stand deviations.

encs – you have still failed to explain why verification and reproduction would be mutually exclusive. Besides, is there a way to independently reproduce several decades of measurements, without a good time machine at hand to go back and re-take those measurements?

Re: replication, though, perhaps I’m not understanding something important, but why do you need a list of sites Dr. Jones and his team used, when you can compile your own list from the same primary data Jones used, or at least 98% the same? Seems to me if you did that but obtained a different result, then that would be a useful scientific result, provided the data’s statistical treatment was sound.

The data sucks. Much of it wasn’t very good to begin with, but it’s all been so massaged and corrupted without oversight or explanation, it’s now totally unreliable.

The models suck. Even with faked-up data and parameters added to force results, they can’t replicate the past. And the actual code is kept secret.

Any sane person has to look at this and find that there is NO WAY it can justify public policy.

We need responsible, honorable scientists, with forensic auditors by their side, to go back to original data readings and thoroughly lay out their strengths and weaknesses, and discuss any adjustments in a fully transparent process.

We need new modellers to review every line of code to ensure the models properly implement good science. All parameters need to be justified, all results validated.

we need a serious program to collect new data, review it, make adjsutments as required, and have custody. Not just CO2, but EVERYTHING that we can conceive might affect climate.

Then we need honest, competent scientists and technicians to load the validated data into the validated models and see what we get. And develop programs to improve teh models and supplement the data, based on the results.

This will be expensive and time consuming. But if we think this subject worth understanding, there is no alternative.

My guess is in about 3-5 years we could be on track with a real program.

Anyone who says “It’s too late for that, we can’t wait, the danger is too grave and too imminent,” needs to be reminded that we now know NOTHING, we have to unlearn the falsehoods we have been fed, and get it right the next time.

And, obviously, Mann and Jones and most of the rest of these guys need to be defending themselves in court and before administrative panels and academic disciplinary hearings, while all this is going on.

There’s more than one definition of replication, or reproducibility. This Wikipedia entry on reproducibility is helpful. See especially the section on “Reproducible research” as defined by Jon Claerbout of Standford; He recommends sharing the full computational environment. But I don’t think that definition is universally accepted.

One of the documents (the HARRY_READ_ME.txt file) is a THREE YEAR journal of a CRU programmer describing everything he tried with the data and models in an attempt to reproduce existing results CRU had published. Comments in the file make it clear that “HARRY” tried FOR THREE YEARS to recreate CRU’s published results AND FAILED.

Do you see the REAL significance of this because it is absolutely fatal to the credibility of anything CRU has produced.

What we have here is a documented THREE year effort by a CRU programmer, who had access to all the data, access to all the code, access to all the people who developed the code and the models and still HE could still NOT duplicate CRU’s OWN results. If he can’t it simply means the CRU’s results cannot be reproduced even by themselves and so there is no point anyone else even trying — CRU themselves have proven it’s a waste of time and so they themselves have proven their own results are plain rubbish. That means any “peer reviewed” document CRU produced along with any other papers that cited the CRU papers are based on data the CRU themselves can’t verify.

Besides, the absolutly sorry state of affairs in the data handling and software managment the HARRY_READ_ME.txt reveals, the utter and total mess of CRU data and software this document reveals is WHY CRU has not released its data and model software.

Given the CRU is one of the most, if not the most cited sources of climate data — upon which trillions of dollars of economic policy is being set, the importance of what the HARRY_READ_ME.txt file reveals becomes scary.

A very nice layman’s summary of some of the issues in the HARRY_READ_ME.txt can eb found here

Taking the 5th as it appears the CRU folks have been doing as it concerns their data does not mean we cannot assess their guilt of innocence based on their silence like in a court of law … In science silence=guilt or fraud …

BTW, it is debatable that the replies to your original request were “unresponsive.” You were in fact directed to a source of data that was entirely adequate to replicate the results of the paper, which is why people normally request scientific data. The poor soul who responded to your original request was naive to the CA mindset of auditing “fraud” (because working scientists are reliant on a network of trust and on replication rather than auditing) and charitably inferred a scientific intention behind your request. You escalated this cultural difference into a war. You think these emails speak for themselves. Obviously they don’t, in fact academic careers are made and lost in the interpretation of seemingly innocuous text, and anyway if they spoke for themselves you wouldn’t have to annotate them in the self-serving way that you do.

While it may (or may not) be true that “it appears that the raw station data can be obtained from [GHCN]“, this is meaningless without an actual list of the sites that Dr. Jones and his team used.

In the climate science arena, this is called “handwaving”. This means a vague explanation that explains nothing. Waving your hand at a huge stack of data and saying “I got my data from there” is no help in trying to find out if CRU has done their work accurately.

I also pointed out:

You are making the rather curious claim that because the data “appears” to be out on the web somewhere, there is no need for Dr. Jones to reveal which stations were actually used. The claim is even more baffling since you say that the original data used by CRU is available at the GHCN web site, and then follow that with the statement that some of the GHCN data originally came from CRU. Which is the case? Did CRU get the data from GHCN, or did GHCN get the data from CRU?

and

I had originally asked for the raw station data used to produce the HadCRUT3 dataset to be posted up on the UEA website, or made available in some other form.

You refused, saying that the information was available elsewhere on non-UEA websites, which is a valid reason for FOI refusals.

I can report that the information requested is not available on non-UEA websites as detailed below.

Your most recent letter (Further _information_ letter_final_ 070418_rev01. doc), however, says that you are unable to identify the locations of the requested information. Thus, the original reason for refusing to provide station data for HadCRUT3 was invalid.

As you can see (I hope), these are non-responsive answers. First they said the information was available on non-UEA websites. When pressed, they said that they couldn’t actually say which non-UEA websites had the data. Thus, the information was not available at all.

While it may (or may not) be true that “it appears that the raw station data can be obtained from [GHCN]“, this is meaningless without an actual list of the sites that Dr. Jones and his team used.

Exactly what I said, “an actual list of the sites that Dr. Jones and his team used” are unnecessary to independently replicate the results. Do you know what what replication is? It doesn’t mean to duplicate, i.e. “audit”, which is only useful to confirm fraud.

This is not good — the existing program produces a serious error when it’s run on what is supposed to be the old, working data. Harry presses on, finding a solution to that bug, going through many more issues as he tried to recreate the results of these runs for the data from 1901 to 1995. Finally he gives up. He has spoken to someone about what should be done:

AGREED APPROACH for cloud (5 Oct 06).

For 1901 to 1995 – stay with published data. No clear way to replicate process as undocumented.

encs (16:44:54): The whole point of an having an independent assessment is to “replicate” (or fail to!) the original findings. What’s ironic is that you would find this ironic.

Please re-read my comment. What I find ironic is the call for independent assessments, especially when done by people that before ClimateGate would not have done a thing about getting independent assessments.

Willis – you were asking for information and data that CRU cannot provide. Read the harry read me files. A CRU researcher spent three years on the code trying to reproduce the temperature history. He failed, and fell back to going with the published data all the up through 1995. You want the data to reproduce CRU work. CRU cannot reproduce CRU work. What a mess, and an embarressment.

The HARRY_READ_ME.txt file is a 700kbyte text file personal note history of someone that appears to be trying replicate/reproduce/validate the existing code. The first line of the file reads :READ ME for Harry’s work on the CRU TS2.1/3.0 datasets, 2006-2009!”

The writer chronicles his many, many problems and intense frustration at poor recordingkeeping, at poor documentation, and the many bugs and erroneous operations he has found in the programs.

It is clear that the writer of the harry file was unsuccessful in his attempt to rerun the programs that generated the historical temperature record that is the most commonly cited one in various journals and IPCC reports.

——————-

Schadenfreude is a good description of my state of mind while reading the file. Another poster described it as a Monty Python episode tripping on acid.

Lots of comments like

I am seriously close to giving up, again. The history of this is so complex that I can’t get far enough
into it before by head hurts and I have to stop. Each parameter has a tortuous history of manual and
semi-automated interventions that I simply cannot just go back to early versions and run the update prog.
I could be throwing away all kinds of corrections – to lat/lons, to WMOs (yes!), and more.

This whole project is SUCH A MESS. No wonder I needed therapy!!

============================

The above comments are from just the most recent few pages!!!

Also of interest – how did the program find a 2000-2009 station when the previous update was to 2008?

So it worked fine until the 0904151410 run, when it went crazee.
So.. what happened? Why did it behave differently? No idea. It was the same for pre though!

What about the ones I used for the CRUTEM3 work with Phil Brohan? Can’t find the bugger!! Looked everywhere,
Matlab scripts aplenty but not the one that produced the plots I used in my CRU presentation in 2005. Oh,
FUCK IT. Sorry. I will have to WRITE a program to find potential duplicates.

Excellent post by Willis Eschenbach and some very good comments too. Re Alexander Harvey’s comment (20:29:07): “The Act gives everyone both in and outside UEA a right of access to ANY recorded information held by UEA”, this is crystal clear (here’s the link.) What part of it wouldn’t anyone understand?

Dave Palmer, Information Policy and Compliance Manager at UEA, will have plenty of interesting e-mails and letters to field, in the days to come, it seems.

My question, though, is why all this subterfuge and evasion, in the first place? If the work was able to stand up to anything that a sceptical auditor could throw at it, it would surely emerge stronger than before.

Willis – You write –
“2. Had an email from David Jones of BMRC, Melbourne. [EMAIL NOT FOUND IN CRU EMAILS – Willis] He said they are ignoring anybody who has dealings with CA, as there are threads on it about Australian sites.”

The email you seek is 1182255717. Short text is

1. Think I’ve managed to persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.

2. Had an email from David Jones of BMRC, Melbourne. He said they are ignoring anybody who has dealings with CA, as there are threads on it about Australian sites

Willis – You write –
“2. Had an email from David Jones of BMRC, Melbourne. [EMAIL NOT FOUND IN CRU EMAILS – Willis] He said they are ignoring anybody who has dealings with CA, as there are threads on it about Australian sites.”

The email you seek is 1182255717. Short text is

1. Think I’ve managed to persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.

2. Had an email from David Jones of BMRC, Melbourne. He said they are ignoring anybody who has dealings with CA, as there are threads on it about Australian sites

Writer of email? P Jones.

Geoff, good to hear from you. Sorry for my lack of clarity. The email I couldn’t find was the email from David Jones explaining the Australian boycott of CA.

You know, a week before this story broke somebody accused me of being a “conspiracy theorist”, and I strongly denied it … not sure I’d do the same today.

With no attempt to decry your efforts, I note that the Warwick Hughes report was presented to the Tasman Institute in 1991. It was a think tank in which I was invited to help select projects to fund. Warwick showed us the data discrepancies that made him uneasy. We were jointly a bit slow to recognise the severity of the issue as it was a bit novel.

A few years later I revived the subject as shown at http://www.climateaudit.org/?p=1545 in March 2006. At the time I was catigated by other scientists who thought it outrageous that I criticise the marvellous CRU – apologies were demanded.

“I am responding to the criticism of my letter in ‘The Australian’ of 15 February 2006 which read:

“THERE is an excellent argument for curbing the public statements of scientists like those from CSIRO, a former employer of mine. Scientists, like the public, cover a spectrum of beliefs, some of which are based on emotion rather than science. There are greenie scientists in CSIRO and there are honest ones. Human nature being what it is, there are private agendas pushed by CSIRO people that would make your jaw drop. An example is the selection of Australian weather recording sites used to construct the temperature measurements of the continent, which play a big part in southern hemisphere weather models. From the beginning, most sites that showed little or no temperature rise or a fall from, say, the 1880s to now were rejected. The few sites selected to represent Australia were mainly from capital cities and under suspicion for “heat island” effects. I could give example after example as it was one of my employment functions to distill the best results from the bogus on many matters related to energy/greenhouse/nuclear etc. I found few truly objective submissions among those masquerading as science.”

Now we find Phil Jones quoted in one email as saying he had destroyed the early data and in another saying he had destroyed nothing.

With you, Willis, and a few others, Warwick and I were helping hold the flag for the Southern hemisphere. But the early credit goes to Warwick, fair and square, for being the first globally (so far as I know) to question the validity of what is now, nearly 2 decades later, a visible, sorry mess of deception.

Hope you are reading these comments on your excellent chronology of events on the whole FOIA fiasco.

I’ve just read CRU’s pathethic press release and I quote

“One particular, illegally obtained, email relates to the preparation of a figure for the WMO Statement on the Status of the Global Climate in 1999. This email referred to a “trick” of adding recent instrumental data to the end of temperature reconstructions that were based on proxy data. The requirement for the WMO Statement was for up-to-date evidence showing how temperatures may have changed over the last 1000 years. To produce temperature series that were completely up-to-date (i.e. through to 1999) it was necessary to combine the temperature reconstructions with the instrumental record, because the temperature reconstructions from proxy data ended many years earlier whereas the instrumental record is updated every month. The use of the word “trick” was not intended to imply any deception.

Phil Jones comments further: “One of the three temperature reconstructions was based entirely on a particular set of tree-ring data that shows a strong correlation with temperature from the 19th century through to the mid-20th century, but does not show a realistic trend of temperature after 1960. This is well known and is called the ‘decline’ or ‘divergence’. The use of the term ‘hiding the decline’ was in an email written in haste. CRU has not sought to hide the decline. Indeed, CRU has published a number of articles that both illustrate, and discuss the implications of, this recent tree-ring decline, including the article that is listed in the legend of the WMO Statement figure. It is because of this trend in these tree-ring data that we know does not represent temperature change that I only show this series up to 1960 in the WMO Statement.”

”

So they are saying they didn’t ‘hide the decline’ but rather just substituted instrumental temperature data after 1960 in place of proxy data because of the ‘divergence problem’. Now who ha sgone on record saying that they would NEVER deliberately ‘splice’ instrumental temperatue data onto to tree-ring proxy data? They would never do that would they? Yet here is the ‘Wizard of UEA’ himself finally admitting to the fact that they have done so and blames the WMO (‘The requirement for the WMO Statement was for up-to-date evidence…’) for doing so. After performing the ‘splice’, he then says breathtakingly pretends to the WMO (and the rest of us) that this is ‘up-to-evidence’. How can this be ‘evidence’? If it is evidence then all it is evidence of is that temperature reconstructions (particularly those that use tree-ring data) make for very poor thermometers i.e they are invalid.

If you can’t to any degree of confidence (as this admission shows) reconstruct the temperature of the past prior to before when instrumental records began, then what evidence is there that the claimed warming trend towards the latter part of 20th century (which itself is of the same size as the level of adjustments made to the ‘raw’ recorded data so is also subject to significant uncertainty) is anything but due to ‘natural climatic variability’? To claim as Michael Mann (and others in these released emails) have done that this period late 20th century warming is ‘unprecedented’ is quite frankly specious and dishonest if not fraudulent.

Phil Jones also says in the press release

“That the world is warming is based on a range of sources: not only temperature records but other indicators such as sea level rise, glacier retreat and less Arctic sea ice.
”

What ‘sea level rise’ – is that the one in Tuvala that hasn’t happened (yet?) or Al Gores’s fictional one?

What ‘glacial retreat’ – the one that enabled Hannibal to cross the Alps and sack Rome – or the one that isn’t happening in most the Himalayas at the moment?

What ‘less Arctic sea ice’ – is that the Arctic sea ice that has recovered since the publication of the IPCC FAR or for which we only have observations since late1970s/early 80s so we have no real idea of knowing if there was just as less Arctic sea ice during the earlier part of the 20th century as observed towards the end of the 20th century/early 21st century?

I am also a scientist. It wouldn’t occur to me to ask for another groups data or methods past the papers that they publish. If I really wanted to find out if what they saying is true I would do the following:

– read their papers to see if they make sense and whether there are obvious errors or things I didn’t agree with

– point these out

– if that didn’t satisfy then start my own project to replicate their stuff from scratch, get the data from source myself with my own agreements or a different data set, write my own code, …

I think it is a problem if you want to get ahold of other people’s work even if it is in some way publicly funded that people simply wouldn’t want to work any more.

Imagine that you are close to getting the genome for something and someone does an FOI, gets it, and completes the work and gets all the credit (kind of an extreme example but you can see the point).

So I can see why there would be a natural resistance to not sharing data or programs i.e. you life’s work, when publishing makes your career.

Might be against the FOI law though. We will see. This looks like the test case for the UK.

@Improbable Wizard (11:57:01) -SNIP– if that didn’t satisfy then start my own project to replicate their stuff from scratch, get the data from source myself with my own agreements or a different data set, write my own code, …

————

Getting the data yourself, when its temperature readings from around the world would be virtually impossible – and you wouldn’t be replicating their work to validate thier conclusions when you use a different subset of stations, different coding, different base assumptions, etc., all because they refuse to release the methods and data sources used for their already published papers

As they’ve already published, your analogy of unfinished work doesn’t apply to this situation. For arguments sake, however, I believe (but am not certain) that FOI laws provide for the very sort of situation you describe and in those situations allows withholding of only the information that affects the work directly until the work is completed and either published or at least submitted for publication.

Jones, Mann, et. al, on the other hand, are withholding crucial information related to papers and work they completed YEARS ago – but that has wound up forming the foundation of the AGW camp, IPCC reports, EPA findings, etc. All of which are affecting national policies around the world and costing literally billions if not trillions of dollars. To withhold the information is unconscionable under these circumstances.

Many years ago whilst studing a degree in Advanced Mathematics and Physics I formally studied the development of some of the greatest scientific theories. In all the cases I ever studied theories (including Relativity, Quantum Mechanics etc) they were laid bare/ exposed, scrutinised, criticised and even ridiculed by the scientific community and amateur observers alike. People said Einstein was mad, Einstein himself was abhorred by Quantum Mechanics but in all cases the scientists responsible for the theories – whilst they did not like it – knew they had to go through biased scrutiny of their work as part of the scientific process. In some cases it took years for a study to be verified as data was checked, experiments replicated and questions answered in what were very often very heated exchanges.

This is not about the hockey stick or whether climate change is happening (I still suspect it is) – it is about the CRU singularly failing to follow the scientific process and being arrogant enough to believe they are above the (FOI) law….it is a sad day for science.

“So since Feb 2007, CRU is in double figures. We never get any thanks for putting things up – only abuse and threats.”

FOI requests now into double digits in a period of 2 years is a long, long, long way from being called a ‘deluge’ – I was amazed it was so few!! And as part of the scientific process you shouldn’t expect ‘thanks’ – it is your duty as a scientist to make the data available. People will criticise your work endlessly but you should have the confidence and belief in it to let it stand on its own merits.

Imagine if you made an FOI request for all the data that CERN has on particle physics to make sure they are doing what they are supposed to do. It would shut them down and it would only cost you 10 GBP.

You make your own experiments, make your own data, and publish your own papers.

This is about peer-review, not original experimentation. There is a push to have all LHC raw data made available in ASCII form to anyone who wants it. It isn’t agreed yet and is a hotly debated topic (and not because it would ‘shut them down’ which is a ludicrous statement and it hasn’t required a FOI request to get them to this point) but it takes us nicely to the main point which you seem to have missed.

If, as a CERN scientist, I used the LHC raw data (whether it was in the public domain or not) and then published a particle physics paper without stating which subset of that data I had used to arrive at my results would I really be taken seriously? If I simply pointed at the billions of records of raw ASCII data and said “Well it came from in there somewhere, work it out” would that be accepted as sound scientific behaviour?

As soon as you publish a paper for peer review you must state the references used including any data sources you used that contributed to your conclusion and results (whether or not those data sources are in the public domain). To say “Here’s the conclusion but I’m not going to tell you how I reached it” is not in the interest of science.

NASA went quickly into raw-data-publishing-mode (with the usual caveats to give the right to the original researchers to do original research) after all sorts of crank theories started to appear about UFOs and Martian faces allegedly “airbrushed” by “evil” Governmental organizations.

It looks to me as if Mann was willing to share data at first but then when he saw what MM was doing with it, he realized they weren’t interested in replicating the research, but finding holes in it, however small. Given the obvious harassment of these scientists by people in the CA camp who really have questionable scientific credentials, I can understand why they were reluctant to comply. I’m sure there was a great deal of talk between scientists about CA’s harassment and FOI requests. It was clear to them what CA was up to, and apparently, it was obvious to the FOI officials. The requests were nuisance requests rather than those of serious scientists hoping to replicate the work and advance science.

greenaway, I appreciate your thoughts.

What “CA is up to”? What we are up to is auditing. We look at scientific papers for errors of any type. Math errors, theoretical and logical errors, proxy selection errors, any kind of errors. We’re not “against” anything but bad science. I didn’t set out to “harass” Phil Jones, I just made a polite request for his data. If he’d been smart, he would have said “sure”, pointed to the data, and kept on working. I didn’t want to file a freakin’ FOIA, it was a pain in the ass. It was much more of a hassle to me than to Phil, I had to do all the correspondence on my side, David Palmer and Kitty Inglis wrote the letters on his side, and Phil just shined it on. Your sympathy is misplaced.

The thing you’re missing in your explanation is that a scientist should give his data and methods to his worst enemy. Because if his worst enemy can’t punch holes in his theories, then he can bet that they are solid. That’s why I put my ideas out on the web at CA or WUWT. Not because they are my friends, but because I know for sure that if my paper can survive the pummelling it will get out here in the rude world, it is solid and unlikely to be overturned.

Finally, a scientist should never have to face an FOI as Jones and Mann and the rest have. Why not? Because making the data public is PART OF THE JOB DESCRIPTION. Science is not some namby pamby game. It is a tough blood sport, where the participants are expected and required to throw their data and their methods and their claims and their reputation into the arena, to subject them to the merciless glare of public view, and watch as other ravenous scientists try to rip their ideas to shreds. It’s ugly, I know, because I play the game from both sides. But that’s how science works.

Science doesn’t work when some wimpy scientist goes “Oh, I don’t like what those bad CA boys and girls are up to, they harass people, I’m not going to show my work to them, they just want to find holes in it, I’ll keep it secret.”

Damn right we want to find holes in it, that’s the whole point. The only way that science advances is by a scientist poking holes in someone’s favorite theory. Jones and Mann want to poke holes in other people’s work, but shield their own work from public examination. If you can’t stand it when it’s your turn, when it’s your theory that the bad scientist boys and girls want to find holes in, go become an accountant. Science doesn’t progress that way, when people hide the data.

First, Willis, thank you so much for putting together the timeline and series of emails. Its truly mind boggling.

Next, wrt this post of yours – Well said and dead on the money.

Ideally, and often, the process proceeds in a reasonably congenial fashion or at least a reasonably courteous fashion – but there are plenty of times where it can become contentious just because of human nature and the personalities involved, and that’s all a part of the expected process.

The entire point of our tool “science” is to get human bias and emotions OUT of the process to the greatest extent possible, and then to repeat, validate, and verify any experimental results. All while specifically looking for, and allowing any and all comers to look for, any defects in the experimental design, the methods used, the assumptions made, the confounding factors that were or were not addressed, simple math errors even, and so on.

Human nature what it is there will always be those in science as in any other discipline who try to game the system or who cheat or lie – the tool of science works also to ferret out both unintentional errors and intentional ones. The ONLY way that can be done is if the original work is open to thorough skeptical evaluation by others. The more thorough the better, and the more errors found the better for EVERYONE in terms of ensuring that final results and conclusions actually have meaning.

Any good scientist may roll eyes and sign and even perhaps b*tch a little about having their work torn apart by others, but at the same time they value that aspect highly and welcome the news of any errors that are found – that’s the only way they can correct problems, come to solid useful conclusions, and move the state of knowledge forward. That’s true even if it turns out there were fatal errors in their work – because that also moves the state of knowledge forward and lets the scientist know where to move from that point on. “Null” results are every bit as meaningful as positive results, and to actually conduct science, one has to be totally open to having one’s experiment turn out to disprove whatever theory one was trying to prove. No decent scientist wants flawed conclusions just so they can fool themselves and others that they were ‘right.’ Its anathema to the very idea of science.

Of course, if no serious flaws are found after hard core evaluation by a number of others who have been seriously trying to find real problems with your work, well, then, that’s the best news!!

That’s exactly right — as appropriate. The rest of the narrative you’ve spun around an incomplete set of emails is your side of the story. I think it’s lame. There’ll be an independent assessment of how UEA responded to a deluge FOI requests, we’ll see how its findings compare with your own.

encs, thanks for your comments. I have quoted from the emails. The full emails are on line.

You act like there is some possible innocent explanation for an email instructing Mann and Wahl and Amman to delete all emails involving AR4 Chapter 6.

I was watching as it went down. I know that they were deleting the reviewer’s comments that by IPCC rules are supposed to be public documents. I know that a polite request had been made for those documents, which was flatly refused. I know that an FOI had been filed for those documents.

So perhaps you can explain some possible innocent explanation for the email saying:

Jones to Michael Mann, 29 May 2008 (1212063122):

Mike,

Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment – minor family crisis.

Can you also email Gene and get him to do the same? I don’t have his new email address.

We will be getting Caspar to do likewise.
…
Cheers
Phil

Because I sure can’t. In fact, I can’t even think of an innocent explanation for not revealing them when politely requested, much less deleting them.

And while I hope that there will be an independent assessment, I’m not counting on it. After all, the IPCC was set up to be an “independent assessment” of the climate … and we know how that has gone.

All the best, I do hope we do get an independent investigation as you suggest.

Willis, you’ve had access to 98% or so of the data for two-and-a-half years. How are you getting on number crunching it? I’d love to know what you’ve come up with.

Reply

inks, that’s the problem. They have never released 2% of the data, much less 98%. They said that only 2% was covered by confidentiality agreements. I said fine, I can live with that, release the other 98% to me and I’ll slope off.

They said, no way, fool, FOAD.

I would have settled for the 98% and gone away happy. But noooo, it was a sekrit …

That’s exactly right — as appropriate. The rest of the narrative you’ve spun around an incomplete set of emails is your side of the story. I think it’s lame. There’ll be an independent assessment of how UEA responded to a deluge FOI requests, we’ll see how its findings compare with your own.

Since 2002, McIntyre has repeatedly asked Phil Jones, director of CRU, for access to the HadCRU data. Although the data are made available in a processed gridded format that shows the global temperature trend, the raw station data are currently restricted to academics. While Jones has made data available to some academics, he has refused to supply McIntyre with the data. Between 24 July and 29 July of this year, CRU received 58 freedom of information act requests from McIntyre and people affiliated with Climate Audit. In the past month, the UK Met Office, which receives a cleaned-up version of the raw data from CRU, has received ten requests of its own.

BTW, it is debatable that the replies to your original request were “unresponsive.” You were in fact directed to a source of data that was entirely adequate to replicate the results of the paper, which is why people normally request scientific data. The poor soul who responded to your original request was naive to the CA mindset of auditing “fraud” (because working scientists are reliant on a network of trust and on replication rather than auditing) and charitably inferred a scientific intention behind your request. You escalated this cultural difference into a war. You think these emails speak for themselves. Obviously they don’t, in fact academic careers are made and lost in the interpretation of seemingly innocuous text, and anyway if they spoke for themselves you wouldn’t have to annotate them in the self-serving way that you do.

E-Mails, even “deleted” e-mails, reside on various servers across the globe depending on the author, the addressees, the isp and the routing from the isp thereafter and server retention policies. With a proper plaintiff, US civil suit & a subpoena, there is enough information below to bypass the “gatekeepers” and gather any relevant deleted data which could reasonably lead to the discovery or presentation of admissible evidence at Trial.

JasonR – only thing, Dr Kelly was trying to get information out, not keep it secret. Ah, and Dr Kelly was fighting the Establishment, not steadfastly defending it. Ah, and Dr Kelly was left utterly alone by friends and colleagues, not the centre of a select band of scientific brothers. Ah, and Dr Kelly had control of nothing, not the receiver of dozens of millions of pounds.

In the US it is two, I doubt it is different in the UK – a husband a wife at the same location during an event that is at some later date deemed to have been a crime can be charged with conspiracy. In any event, in practice conspiracy is little more than a rider charge, an “enhancement,” to give a prosecutor more charges to work with.

A motivated prosecutor could make do with much less than has happened here. However, given that their actions were not blatantly criminal, no direct financial benefit was obtained, and they’re on the side of “good science”, I highly doubt any action will be taken. In light of the fact that the political consensus is against you and you are “practicing bad science,” I wouldn’t even ask for censure.

The best you can hope for is to publicly establish that they have acted in bad faith. In the future you may be better off drilling down on those issuing grants, asking for them for the data and models that are used. If they’re not forthcoming, you can tie grants issued directly to inadequate scientific reporting.

Gavin over at RC is spinning like a damned fool and you know what? Gavin is full of [snip] and himself – the same thing. Enough with this [snip]! Stop telling us we don’t know what we’re talking about. The emails are evidence of forgery, data manipulation, lying, cheating, moving money around, admission that the global warming people are wrong, nasty digs at each other, advice on how to evade FOIA, and there’s even one which talks about not worrying because their ‘friends upstairs’ will take care of the problem.
[snip]! What the hell does it take to shut these guys down! Gavin, who’s involved in this mess himself, ought to be stripped of any funding and hurled out into the street where he can learn just how cold it is sleeping under a bridge.

It looks to me as if Mann was willing to share data at first but then when he saw what MM was doing with it, he realized they weren’t interested in replicating the research, but finding holes in it, however small. Given the obvious harassment of these scientists by people in the CA camp who really have questionable scientific credentials, I can understand why they were reluctant to comply. I’m sure there was a great deal of talk between scientists about CA’s harassment and FOI requests. It was clear to them what CA was up to, and apparently, it was obvious to the FOI officials. The requests were nuisance requests rather than those of serious scientists hoping to replicate the work and advance science.

“…. I know that DEFRA receive Parliamentary Questions from MPs to answer. One of these 2 months ago was from a Tory MP asking how much money DEFRA has given to CRU over the last 5 years. DEFRA replied that they don’t give money – they award grants based on open competition. DEFRA’s system also told them there were no awards to CRU, as when we do get something it is down as UEA!”

There’s bound to be one rather unhappy MP when he is informed of this little evasion scheme to his simple question! I’m sure he will be really enthusiastic about voting on renewing their budgets.

This is a great post which puts a lot of the principals’ actions in context. I have a PH. D. in astronomy, and worked many years in the telecom industry. I am amazed that we are so far down the road of diverting trillions in wealth based on so little actual good science and analysis. This is a black mark that it will take all branches of science decades to recover from.

This is one of the best pieces on the CRUd CRUsade that I’ve seen. Thank you Willis and Omni. Let Bishop Hill know, it should go into his forthcoming book – especially as this fanatical, cultist approach of Jones et al clearly goes back to MM challenging Mann’s Hockey Stick. “Global Warming Iiiiiiisssss Haaaaaaaaaapening and we have to be able to adjust the data if necessary to prove it and this is urgent and any challengers are only out to obstruct”

God almighty, another Jim Jones? or rather, a whole set of JJ clones?????

Given that Jones and CRU have been using the “can’t get all the permissions from some of the national met services” line for quite some time, might I suggest an FOI request to the CRU/UEA asking for the correspondence between them and all the non-co-operating NMSs. Surely they couldn’t claim that this would breach confidentiality agreements too?

I rather suspect that the CRU haven’t moved very fast on this, if at all, judging from the constant re-iteration of the same line by Phil Jones.

“The vast majority opinion was that scientists should give enough information on their data sources and methods so others who are scientifically capable can do their own brand of replication work, but that this does not extend to personal computer codes with all their undocumented sub routines etc. It would be “odious” requirement to have scientists document every line of code so outsiders could then just apply them instantly.”

Alex – funny thing is, there is no requirement in the FOI Act that “outsiders” should be able to “apply the code instantly”. The “odious requirement” is a bogus claim. Perhaps all the collection of “hacked” e-mails will end up with multiple references in an encyclopedia of logical fallacies…

That aspect really struck me also – there’s zero requirement to add code documentation, let alone ‘line by line’ – just release the code in whatever form its in. That simple.

The horrifying thing to me, however, is how clearly these ‘renown authorities’ of climate clearly have either zero understanding of the most basic tenants of science, or have zero respect for science period. Reproducibility, validation and verification, etc.

Gross perversion of science with utterly amazing fallout in terms of the literally billions of dollars that have been wasted and the effect on god knows how many people throughout the world as a result of the various policies that have been implemented already, not to mention whatever went lacking as funds were diverted to climate/CO2 issues rather than other endeavors.

Please bear with me while I quote from the UEA’s FOI pages: After each bullet point the anotation will be mine.

Guidance for staff

5 key facts that all staff should know about Freedom of Information

•The Act gives everyone both in and outside UEA a right of access to ANY recorded information held by UEA

Everyone, everywhere to ANY information. This is all UEA information, repeat it is all UEA information none of it is owned in any meaningful way by its author.

•A request for information must be answered within 20 working days

Standard stuff.

•If you receive a request for information which mentions FOI, is not information you routinely provide, is unusual, or you are unsure of, you should pass the request to your FOIA contact or the Information Policy and Compliance Manager

Again standard stuff.

•You should ensure that UEA records are well maintained and accessible to other staff, so that they can locate information needed to answer a request when you are not there

This is good point. UEA information is not author specific, there is no implied right for the author to even be informed of the request, if it is available and relevent it can be dispatched to the requestor by any appropriate menmber of staff.

•As all documents and emails could potentially be released under the Act, you should ensure that those you create are clear and professional

Say no more!

Quote from web pages over.

Look the request is to the body (UEA in this case), not to the author. In theory at least the FOI bod or anyone else could come along parcel it up and send it to the requestor. Job done.

There is other policy stuff on the pages and elsewhere that deal with publication schemes. These deal with pre-emptive actions (nice ones) that should be undertaken to ensure likely requests can be undertaken speedily and effectively.

I am disappointed by the FOI bods stance, were it I, I would have wished to put together a package that was clear and simple for the next request (including what they could release and detailing what they couldn’t an why), which they were aware would come. But were it I, I would have been re-assessed for suitability I suppose. But there is a point here, the FOI bod and in particular the FOI commissioner’s office are meant to facilitate. Aren’t they?

A secondary duty I had in the Canadian military was acting as the “Access to Information” link for the units I worked for. Saw some rather strange and time consuming requests, and did have to sever some information out for security reasons, but don’t recall ever thinking that I could just blow off the requesters and get away with it.

This is very serious stuff, heck the entire Canadian Military shut down for a day in the 90’s to search for anything written or concerned with Somolia, “On April 9, 1996, Gen Boyle issued a message to all Canadian Forces members to “stand down all but essential operations to conduct a thorough search of all their files, to identify and forward to NDHQ/SILT any Somalia-related document not previously forwarded…”. As a result, SILT received an additional 39,000 documents totalling more than 200,000 pages. This Inquiry did not receive final delivery of these additional documents until September 27, 1996. ” http://www.forces.gc.ca/somalia/vol1/v1c14e.htm

Breaking the law is breaking the law, I don’t care who you are or what you do for a living. Of course, this hack makes it clear that CRU really did not have what you were asking for, so I guess they were being at least partly honest.

I’m a long-time lurker and rare contributor on other sites. As a former academic chemist and provider of expert opinion to the legal and insurance industries, I have some modest experience of scientific matters. I have been following this saga for about 18 months and I was familiar with some of the above. I would like to echo what Viv Evans has just said. This an excellent and damning account of what has been going with respect to CRU, the FOI Act and their treatment of requests from Steve McIntyre, Willis Eschenbach and others.

I would strongly suggest that not only should this post be widely disseminated to as many sites, sceptic and alarmist, as possible, but copies should be sent to all MSM sites, the BBC (in the UK), television networks in USA, and, most importantly, to the Vice -Chancellor of UEA.

In the past, I have been a referee for the Chemical Society and I would add what little weight I can to the concern over the way the peer-review process has been corrupted by these so-called climate “scientists”. I would never have recommended a paper to be published that did not have the necessary and sufficient information in it for the work to be replicated by others. Again I echo the previous contributor, without the transparency and availability of information in a scientific publication there is no science.

take a way quote:This says that the GISS climatologists somehow believe that a thermometer in Africa or Europe needs as much as 8x more correction than a thermometer in China. Wow. What a discovery. And since the only way that the max correction can be both in the northern hemisphere and between 0 and 81 deg longitude is for the bad thermometers to be in Europe. All of the European thermometers read too cold and must have heat added to them.

I think what he actually said was that he received the one string of emails that involved their (Jones et. al) concerns about the “where’s the global warming” article that he (Hudson) wrote. He did not say that he’d received the 62 MB zip file that is now being analyzed.

Thank you for this astounding report.
I’m not a climate scientist, but have followed the debate on the relevant sites, like CA and WUWT.
I am, however, a zoologist – retired now – and that I am flabbergasted by these evasions of providing data to other scientist, as you describe in horrendous detail above, is saying it mildly.

What happened to that vital part of a scientific paper called ‘Material and Methods’?
That part, in my time, provided all the raw data and what one did with them. Without that – no publication.
This, for climate science, is now hidden? Lost? Inaccessible? Unnecessary? Fudged? Anything goes?
Unbelievable!

As you point out – without these data, there is no reproducibility.
Without reproducibility, there can be no science.

The problem with getting other data and writing differing programs is that you will only be able to show what you produced, you won’t be replicating the other scientist’s work. In this case, if someone gathered their own data, and wrote their own program and came up with a different conclusion, the scientists who refused to provide their data for replication purposes would be able to argue that the new data was wrong, falsified, etc.

In this instance, I am beginning to think that even if they did release their data and there were problems found with their methods, they would still claim that the researchers checking their work was biased and incorrect.

the UEA allegation that “there are persistent and motivated groups or individuals whose applications create a volume of work which is extremely demanding” is a grave one and it should be either substantiated, or withdrawn

“The VC is also aware of what is going on – at least for one of the requests, but probably doesn’t know the number we’re dealing with. We are in double figures.”

Tens of requests he had to deal with. Oh the horror.
Torturous workload. Can you imagine the sweat, the sheer drudgery of saying no ten times!
No wonder he bauks.

On a serious note, notice how few times Keith Briffa’s name is mentioned in this plot.
Lubos has a working theory that Briffa had been holding his nose at the stink given off by his colleagues for some time. Then when the Yamal recon came under scrutiny he was sucked into the sewer.
That’s when Briffa bauked, only he went with the scientific method and humanity, rather then the team.

Just a theory. Maybe if you want you can cross reference Briffa emails versus Eschenbach’s timeline. Look for comparison or contrast of tone.
My guess is you will see a growing pattern of disgust with his mates.