Climategate: An Autopsy

Climategate: An Autopsy

How did emails stolen from climate scientists snowball into a global news story in less than 48 hours?

“A lot is happening behind the scenes. It is not being ignored. Much is being coordinated among major players and the media. Thank you very much. You will notice the beginnings of activity on other sites now. Here soon to follow. ~ ctm [Charles Rotter, moderator on WattsUpWithThat.com]”

‘Climategate’, or ‘Swifthack’ was a media story about a set of hacked emails that was pushed by a group of avid climate skeptics, including bloggers Steven Mosher, Steve McIntyre, Ross McKitrick, Patrick Condon, Lucia Liljegren, Charles Rotter and Anthony Watts. Collectively, they took a mountain of stolen material, condensed it into a well-packaged pitch, and sparked a scandalous story that reached virtually every major news outlet in the world.

Brief Overview of Events:

In November 2009, an unidentified hacker stole a large amount of data from the Climatic Research Unit at the University of East Anglia. The data contained thousands of private emails, internal documents and climate data sets dating back to 1996. The hacker created a 61 mb file from this data and gave it to a small group of climate skeptics and ‘lukewarmers’, who pored over it to pick out potentially controversial snippets. The bloggers sifted through the data to find quotes in private emails which could be used to condemn the scientists and question their research.

They collectively developed a narrative asserting that climate scientists lied to the public, manipulated data and suppressed dissenting views. They implied that the contents of the emails cast doubt on the whole of climate science. The timing of the release of this story was perfect, coming less than a month before the highly-anticipated Copenhagen climate talks. Such a scandal had the potential to dampen public support for global efforts to mitigate global warming.Background: November 2009

Steve McIntyre has used Freedom of Information (FOIA or FoI) requests to gain access to CRU data since 2007. McIntyre maintains climateaudit.com and, with co-author Ross McKitrick, has published two articles in minor journals that point to flaws in the original hockey-stick graph. McIntyre has spent his carreer working for and running Canadian oil and mineral companies.

McIntyre filed or coordinated the filing of dozens of FOIArequests to climate scientists at East Anglia beginning in 2007, with the pace escalating significantly in the months leading up to the leak. According to a 2009 interview with Phil Jones in the TimesOnline, “in July alone the [CRU] unit received 60 FoI requests from across the world.”

“We were clearly being targeted,” says Phil Jones of East Anglia University. “Only 22% of the FoI enquiries were identifiably from within the UK, 39% were from abroad and 39% were untraceable.” What irked him was that the foreign applicants would all have had sources closer to hand in their own countries.

[Update:] Steve Mosher, writing in the comments below clarifies:

The first FOIA was in 2007. This was an FOIA done by Willis Eschenback. Willis filed the FOIA on his own initiative and said nothing about it to McIntyre for 5 months. Upon finding out that WIllis had done an FOIA, Mcintyre did one. Mcintyres was granted, 50 of Willis was granted.

In 2009 there were as you note a flurry of FOIA. Essentially CRU misrepresented the existence and wording in CONFIDENTIALITY agreements. The fact of their misrepresentation was documented in an letter sent by the FOIA appeal office to Mcintyre Nov 13th, 2009. The “60” or so FOIA were all combined into a single request and that request was answered by CRU posting 4 agreements on the web. Jones spent less than the 18 regulated hours on the request. My request which was in the same pile of requests was denied because CRU determined that responding to my request would take more than 18 hours. I requested their FOIA guidelines.

Scientists did not fill those requests, stating that much of the information was already publicly available, fulfilling some of the requests would be a huge time burden and, in some cases, would potentially violate the terms of agreement between CRU and the original sources of the data. [Update: Mosher clarifies that there were 4-5 requests for specific data, which were denied because some was covered by confidentiality agreements. The non-confidential data was then requested, which was already supplied to Peter Webster but denied to McIntyre, even though the confidentiality argument is inconsistent here on the part of CRU] (Ironically, some of the raw climate data the bloggers requested was actually found months earlier on unsecured links at UEA due to network glitches.) [update: According to Mosher, similar but older data was found, not the data that McIntyre requested.]

The emails that made up the now infamous “FOIA2009.zip” file that the hacker sent to the bloggers in late November were compiled in three batches, from Sept 2009 to Nov 13, 2009. Steve Mosher speculated that the emails were sorted using fairly simple search functions to weed out the truly inane (out of office replies, etc) so that only emails likely to be relevant to skeptics’ arguments remained. He also speculated the hacker understood enough about climate skeptics’ previous arguments to know what types of information to highlight, but did not spend enough time with the file to find all of what would be considered ‘juicy’ evidence.

Steve Mosher is an open-source software developer advocate and one of the moderators of a prolific commentor on The Blackboard, as well as ClimateAudit and WattUpWithThat. He also studied English in grad school at UCLA, specializing in statistical analysis of word frequencies. According to Patrick Courrielche at BigJournalism.com, Steve Mosher is “an open-source software developer, statistical data analyst, and thought of as the spokesperson of the lukewarmer set…”

Initially the hacker was suspected to be a highly sophisticated person or team, but analysis by the Guardian shows that the hack and the posting likely did not involve a high level of sophistication. Downloading the files, sorting them and uploading them onto a Russian or Turkish FTP server are not difficult tasks technically, as long as someone had access to the East Anglia servers, which had been proven to be less than secure. The next step of hacking into the backend of RealClimate.org to upload the .zip file could be considered moderately difficult, requiring familiarity with the weaknesses of the WordPress blog platform.

Hacker Action:

The hacker used several methods to anonymously send the file to a small set of skeptical bloggers. He first hacked into RealClimate.org (a site maintained by climate scientists Gavin Schmidt, Michael Mann, and others) and uploaded the FOIA.zip file. The hacker then attempted to write a post on RealClimate, which was not successfully published. He then commented on ClimateAudit.org listing his name as “RC” at 10:54 GMT on November 17th with the text “a miracle just happened” and linking to Realclimate.org/foia.zip.

About twelve hours later, a comment linking to the same file but located on a Russian FTP server was posted to WattsUpWithThat.com stating “We feel that climate science is, in the current situation, too important to be kept under wraps” and listing a “sample” of 19 email file names and snippets. Charles Rotter, also known as Charles the Moderator or CTM, did not approve the comment but he did notify Anthony Watts immediately. Anthony Watts runs WattsUpWithThat.com. He is a former TV meteorologist who started by blogging about locations of surface weather stations. His site is regarded as the largest and most active of the climate skeptic sites.

The hacker then placed a brief comment on a more obscure climate skeptic site called the Air Vent maintained by Patrick Condon (aka Jeff “Id”) and a long comment on Climate Skeptic identical to the WUWT comment, both linking to the Russian FTP site. These actions occurred between 11:20am on the 17th and 3:47am UMT on the 18th.

Rotter is the head moderator of WattsUpWithThat, a volunteer position. He only revealed his last name after heightened scrutiny following the leaked emails. He made a CD copy of the file and gave it to Steve Mosher around 9:36pm PST on November 17th. (Rotter and Mosher are flatmates in San Francisco.) Watts was traveling in Europe and says he didn’t look at file. Rotter felt that Mosher was far more qualified to look at it than he was.

Mosher started analyzing the emails for their validity. He and/or Rotter first called McIntyre late in the evening of November 17th to confirm some of the emails, which included conversations between Phil Jones, Michael Mann and McIntyre regarding FOIA requests. McIntyre confirmed the emails involving him were genuine, and said he was intrigued, but claims he didn’t get the actual file until the 19th. Mosher apparently spoke to McIntyre frequently over the next 36 hours.

On November 19th Paul Dennis, an isotope specialist and employee of CRU, emailed McIntyre to notify him of the security breach. McIntyre shared discussed the email with Mosher, providing further assurance that the FOIA2009.zip files were genuine.

“Much is being co-ordinated among major players and the media”The last known activity by the hacker happened just after Mosher and Rotter began distributing the story, about 60 hours after the first known activity of hacking into RealClimate.org.

[Updated, based on Mosher’s comments] Mosher says he became sure of the validity of the files when he learned of UEA’s security measures via McIntyre.He then looked for the files elsewhere online, found the comment on Air Vent, which had gone unnoticed since the 17th. He said in an interview that, “My first reaction to the link [on Air Vent] was relief. I didn’t want to be the only person who had these files…”

Mosher also emailed Tom Fuller, a blogger at Examiner.com and sent a facebook message to Andy Revkin at the NYTimes. We don’t know what other media people Mosher notified. While he has written extensively about the incident, he rarely mentions notifying Fuller, and I only found one reference to contacting Revkin (a note Revkins said he saw five hours later, after he had already seen the story elsewhere.)

An hour after these first comments from Mosher and Rotter, the hacker made a comment on WUWT which was held in moderation. According to Mosher, it said roughly “ ‘what the eff’ why isnt anybody talking about this”. Rotter, as moderator, replied via the comment forum at 1:16pm CST, or 70 minutes after Mosher’s first post on Liljegren’s site.

“A lot is happening behind the scenes. It is not being ignored. Much is being coordinated among major players and the media. Thank you very much. You will notice the beginnings of activity on other sites now. Here soon to follow. ~ ctm”

By the time Liljegren made her short post, Mosher had almost two days to pore over the files. Despite Liljegren’s apprehension about posting the contents of any emails, Mosher proceeded to post four of the emails as comments within 30 minutes. Lucia Liljegren studies fluid dynamics at Argonne National Lab and teaches university in Ames, IA. She began her site, The Blackboard, in Nov 2007 to document ‘teaching herself R’ (a statistical method) and using the European Climate Assessment database as her test set. She says “I firmly believe CO2 will tend to cause global warming and the effect is sufficiently large to worry about. That said, I’m also interested in reading the questions and arguments presented by skeptics.”

In addition to posting to Liljegren’s site, Mosher also posted three of the emails as comments on an unrelated thread on McIntyre’s site. Three of the six total emails were part of the ‘random’ selection posted by the hacker in the original comment, and the other three were new. One of the new ones contained the now-famous ‘hide the decline’ phrase that became and stayed the top-line message on the scandal among the right wing media. The ‘hide the decline’ email was the only email Mosher posted on both sites.

Watts posted on the story on his site from Dulles airport on his way back from Europe within 90 minutes of Liljegren’s post on the afternoon of the 19th, according to the time-stamp on the first comment.

Only 10 minutes after the first comment on Watts’ post, Ian Wishart in New Zealand made a brief post, linking to Watts and stating that he had emailed Phil Jones to confirm. Wishart maintains a blog called The Briefing Room and also writes for or edits Investigatemagazine.com, also known as TGIF. Wishart spoke to Jones in an ‘exclusive’ interview sometime in the next 12 hours and posted a story on TBR and TGIF at 9:30pm NZ time, (8:30am on the 20th UK time), who confirmed that the files were in fact real and the CRU had in fact been hacked. The TGIF article is referred to as proof of the validity of the stolen emails in most early stories.

5 1/2 hours after Liljegren’s initial post (now about 8:50 pm EST on the 19th), Tom Fuller posted on the Examiner.com that files were stolen, other people were writing about them. “I’m not going to publish the files. I hesitate to even write about what’s in them, other than in the most general terms.” Fuller posted two more blogs over the following twelve hours with more complete details. (Fuller also went on to write the Climategate book with Mosher.)

At 8:53 EST on the 19th, an hour after Fuller, Examiner writer Terry Hurlburt posted a long blog with full details and essentially the same messaging that would be used throughout the news cycle. His post is rarely mentioned in other accounts of Climategate, but at the time many other blogs linked back to his. He made several updates to that post, and it’s possible he could have altered his initial post to make it more appealing and fleshed out, but the writing and circumstances to indicate that he didn’t alter his original text. The Examiner’s editorial and business model allows for thousands of writers to keep their own blogs and who are given free rein to post as they please, with editors then selecting and promoting pieces. It is likely that when his post began to generate traffic from bloggers, the editors noticed and began to promote it as well.

Possibly after seeing Wishart’s post, and two hours after Hurlburt’s post, Andrew Bolt in Melbourne wrote another comprehensive post at 4:36am UMT which, similar to Hurlburt’s, captured the story as it would later be written by many larger outlets.

Marc Morano at Climatedepot.com first linked to Andrew Bolt’s account with the headline: “Update: ‘CRU director admits emails seem to be genuine’ – Climatic Research Unit Hacked? ‘Warmist conspiracy exposed?’ Morano doesn’t write posts, only publishes headlines and links to other sites, and his links don’t have time-stamps so it’s difficult to determine exactly when he became involved. However, first linking to Bolt and following links backwards from there is a plausible way he arrived at the story. He then links to Liljegren, Fuller, Hurlburt and then Condon and posts regularly on the story from there. Many of Morano’s headlines were short, attention-grabbing quotes from emails or commentary from bloggers. He also quickly and repeatedly used headlines criticizing the mainstream media for not covering climate skeptics.

At 6:40am EST (11:40am UMT) on Friday morning, UK-based reporters began emailing Michael Mann in the US for confirmation and reactions to the story. This was less than 16 hours since the first public comment by Mosher on Liljegren’s blog. Three more requests for comment arrived in Mann’s inbox by 9am EST and they kept coming throughout the day, as well as phone calls, as US media started picking up the story.

By late afternoon on Friday the 20th, Competitive Enterprise institute had issued a press release on the scandal, complete with a quote from Myron Ebell saying in part: “Some of the e-mails that I have read are blatant displays of personal pettiness, unethical conniving, and twisting the science to support their political position.”

Spreading the Story Far and Wide:

Six hours after Wishart confirmed the leak in TGIF, James Delingpole published the story in a Telegraph blog, with the meme ‘the final nail in the coffin of anthropogenic global warming’. Minutes later the BBC posts its first story at 2:13pm, UMT time. Four hours after that, Fox News started posting it online in the US with the hook ‘skeptics see smoking gun in leaked climate science emails.’ At 1:48pm on Friday the Drudge Report first found the story, spreading it to an even broader network.

By the time Bolt had the story in Melbourne there were likely dozens of bloggers downloading the file and scanning it for quotes to share. Based on the number of comments that day and the average readership of the skeptic blogs, they had an army poring over the files in a race to find the next ‘juicy’ comment from the climate scientists. However, the media headlines were all shaped by the relatively small number of comments discovered and featured by Mosher.

On Monday the 23rd, Glenn Beck did a segment, and on Tuesday Pat Michaels was interviewed on Fox News. A mid-western group even produced a video on the 24th called ‘hide the decline’ which rose to 500,000 views on Youtube after it was promoted by Rush Limbaugh.

Steve Mosher and Tom Fuller went on to write a book detailing the series of events around the hack. Collectively, the group of skeptic bloggers have written extensively about the link and refer to it frequently. Police have made little progress in identifying who stole or leaked the files. The skeptic bloggers were able to take a security breach and present it to the PR and media organizations invested in opposing action to reduce global warming emissions.

Previous Comments

WRT this bit: In November 2009, an unidentified hacker stole 61 megabytes of data from the Climatic Research Unit at the University of East Anglia.

My understanding is that the hacker took everything on the server he gained access to - ie the whole email database back to the mid-1990s. The files in FOIA.zip were a careful selection - probably done by keyword searches (as the Guardian says) on a computer with its time zone set to the east coast USA see here. Someone with a reasonable degree of knowledge of the main themes discussed at Climate Audit would have been required to produce that key word list.

I presume that none of the parties mentioned in your story were involved in making the selection of mails that went into FOIA.zip, though several would have been ideally qualified to do so. Their diligence in getting the story out is admirable, but the real story lines were set by the Bolts, Becks and Moranos, with considerable assistance from Ebell and the rest.

Thats interesting and worth further exploration. Could you cite a source for your understanding that the entire content was downloaded and the FOIA zip created from the content retrieved? I see a “see here” but no link.

And.. why cant I use apostrophes? If its to resolve the inject problem Id be happy to help with a workaround.

As I read this this post, Ill just note down some comments and corrections. Just as a note,
Charles thinks this is the best researched piece on the timeline and puts the guardian to shame.
Ill add my comments going a few paragraphs at a time to correct any errors I find.

1. Swifthack. My suggestion is that this is a bad frame for those of us who believe in AGW (as I do) to use.
One only need think of who it was who betrayed Kerry and their relationship to him to understand why this is a bad metaphor.

2. You wrote:

“In November 2009, an unidentified hacker stole 61 megabytes of data from the Climatic Research Unit at the University of East Anglia. The data contained thousands of private emails, internal documents and climate data sets dating back to 1996. The hacker gave the file to a small group of climate skeptics, who pored over it to pick out potentially controversial snippets. The bloggers sifted through the data to find quotes in private emails which could be used to condemn the scientists and question their research.

They developed a narrative asserting that climate scientists lied to the public, manipulated data and suppressed dissenting views. They implied that the contents of the emails cast doubt on the whole of climate science.”

A. I am not a blogger
B. I did not sift through the mails to find quotes. I read each and every email several times over. My initial goal in the first 48 hours to was ascertain if they were true or not. I used several means to do this.
C. I emphatically state in the book, Climategate: the Crutape letters, that the mails change nothing
in the science. Ive stated this on numerous occassions.
D. WRT telling lies. I have stated on numerous occasions that the only person who came close to telling
a lie was Jones and specifically in matters relating to FOIA and then again to parliament after the fact.
( see my post on WUWT caled “the last straw”
E. Manipulating data and supressing alternative views? Id yes to the latter and the former needs very careful qualification. They were definately as not as forthcoming as they should have been.

In short I accused them of noble cause corruption.

Since Ive repeatedly argued that I believe in AGW, in fact all Lukewarmers do, its a bit puzzling why you put me in the skeptic camp.

First, let me apologize for the long delay. I’m new to the web-platform here and wasn’t sure how to approve comments. Second, I appreciate your diligence, it made researching this relatively straightforward, and in hindsight, I should have contacted you for details. Third, I made several corrections in-line. A few of your comments were outside the scope of the post text (but valuable to have recorded in the finally-approved-comments), and a few I’ve left.
For example, the fact that ‘Steve Mosher is a member of a group of bloggers’ is accurate, since you write regularly on blogs and have authored at least one post about this. The fact that the scientists were accused of ‘manipulating data’ is a charge made by numerous commentators who are not as precise as you.

“McIntyre filed or coordinated the filing of dozens of FOIA requests to climate scientists at East Anglia beginning in 2007, with the pace escalating significantly in the months leading up to the leak. According to a 2009 interview with Phil Jones in the TimesOnline, “in July alone the [CRU] unit received 60 FoI requests from across the world.””

The first FOIA was in 2007. This was an FOIA done by Willis Eschenback. Willis filed the FOIA on his
one initiative and said nothing about it to McIntyre for 5 months. Upon finding out that WIllis had done an FOIA, Mcintyre did one. Mcintyres was granted, 50 of Willis was granted.

In 2009 there were as you note a flurry of FOIA. Essentially CRU misrepresented the existence and wording in CONFIDENTIALITY agreements. The fact of their misrepresentation was documented in an letter sent by the FOIA appeal office to Mcintyre Nov 13th, 2009.
The “60” or so FOIA were all combined into a single request and that request was answered by CRU posting 4 agreements on the web. Jones spent less than the 18 regulated hours on the request. My request which was in the same pile of requests was denied because CRU determined that responding to my request would take more than 18 hours. I requested their FOIA guidelines.

“Scientists did not fill those requests, stating that much of the information was already publicly available, fulfilling some of the requests would be a huge time burden and, in some cases, would potentially violate the terms of agreement between CRU and the original sources of the data. (Ironically, some of the raw climate data the bloggers requested was actually found months earlier on unsecured links at UEA due to network glitches.)”

You are confusing two sets of requests.

A. there were 4-5 requests for specific data. These requests were DENIED. They were deniedNOT because of time constraints. The time constraint per request is 18 hours. They were denied
because CRU argued that SOME of the data was covered by confidentiality agreements. So the NONCONFIDENTIALDATA was requested. This too was denied.

B. CRU had various conflicting arguments about the agreements. The data requested was the SAME data that had been provided to peter webster. that is what Mcintyre requested. CRU denied this arguing that Mcintyre was not an academic and that the agreements precluded release to non academics. Upon recieving this denial 4 academics requested the data. They were denied and CRU changed the argument. they said the agreements precluded ALL release. This made no sense as Webster had received the data.
So, we FOIAd the agreements.

C. The climate data requested WASNOT found months earlier. A 2003 version of the dataset was found the same month as the requests. Mcintyre did not request the 2003 version. he requested the data sent to Webster.

“Steve Mosher is an open-source software developer and one of the moderators of The Blackboard. He also studied English in grad school at UCLA, specializing in statistical analysis of word frequencies. According to Patrick Courrielche at BigJournalism.com, Steve Mosher is “an open-source software developer, statistical data analyst, and thought of as the spokesperson of the lukewarmer set…”

Im not an open source software developer. Im an open source advocate, but my main focus in the past has been on both hardware product development and software. I stopped writing code ages ago. While I started statistics in grad school, most of my statistics work was done in Aerospace.

“On November 19th Paul Dennis, an isotope specialist and employee of CRU, emailed McIntyre to notify him of the security breach. McIntyre shared the email with Mosher, providing further assurance that the FOIA2009.zip files were genuine.”

Mc NEVER shared the mail with me. We talked on the phone. He indicated that he had gotten a mail from somebody at UEA ( not CRU) the mail indicated that the university were looking into a brwch and that some mails had been posted on the internet somewhere. I asked Mc if he thought this guy was the source. He said, “no.” He commented on the guy as being an rare commenter on Climateaudit and a guy who was working on isotopes. I did not ask for the name of this guy and only found out after he revealed himself on Bishops blog.

“Mosher says he became sure of the validity of the files right around the same time as he found the comment on Air Vent, which had gone unnoticed since the 17th. He said in an interview that, “My first reaction to the link [on Air Vent] was relief. I didn’t want to be the only person who had these files…”

On the morning of the 19th when I learned that UAE had alerted the staff I knew three things.

1. the files were real
2. Nobody was trying to entrap Anthony
3. I didnt hold the only copy.

Up until the morning of the 19th I was under an obligation to not pass on the files. they came into WUWT
and they were passed to me on the agreement that I would not release them until Anthony was back in the country. When I heard that the files were “out there” it was easy to find them on the air vent. I then informed Charles that my agreement with him and anthony was OBE.

“Mosher also emailed Tom Fuller, a blogger at Examiner.com and sent a facebook message to Andy Revkin at the NYTimes. We don’t know what other media people Mosher notified. While he has written extensively about the incident, he rarely mentions notifying Fuller, and I only found one reference to contacting Revkin (a note Revkins said he saw five hours later, after he had already seen the story elsewhere.) ”

here is an exclusive. On the evening of Nov 18th I called Tom Fuller, a friend. I informed him that I had come into possession of certain files. I asked him for legal advice from a journalistic perspective. I informed charles that I had talked to Tom about the legal aspects only. This is the “major media” he refers to.

Ive commented several times on contacting Andrew. I gave Andrew the lede. I told him to follow the FOIA. he arranged to call me in the days after the event. I called him several times. he never returned my calls. Charles and I also did a long interview with other Mainstream media. We told them the ENTIRE chronology. They did not run with the story. On Nov 29th after watching people butcher the story and get many things wrong ( quotes out of context and attacks on the science ) Tom asked me to write a book with him

That once this story broke it was a ready-made media hook, and few reporters were actually concerned with the details of the files or exactly what CRU said and did. The ‘scandal’, ‘hack’, ‘doubt on the whole theory of global warming’ was so much louder than numbers of FOIA requests or data storage methods.
I wish the MSM you mention had run your long interview, to counter the ‘butchering’ you mention.

I have to concur with Charles. WIth a few minor corrections here and there this is a very nice piece of detective work. You seriously could have asked Charles or me for an interview and Tom as well.
We have all been open book with regard to the timelines and who knew what when. Tom like me believes in AGW. Charles Im not so sure as we dont talk about our views on the matter very much. I think he might be a skeptic. He just criticizes me for not proofing my comments or posts. ha. Anyways, good job.. but again a few of the people at the core of this actually believe in AGW. go figure.

Democracy is utterly dependent upon an electorate that is accurately informed. In promoting climate change denial (and often denying their responsibility for doing so) industry has done more than endanger the environment. It has undermined democracy.

There is a vast difference between putting forth a point of view, honestly held, and intentionally sowing the seeds of confusion. Free speech does not include the right to deceive. Deception is not a point of view. And the right to disagree does not include a right to intentionally subvert the public awareness.