IEA: Hadley Center “probably tampered with Russian climate data”

Recently rejected two papers (one for JGR and for GRL) from people saying CRU has it wrong over Siberia. Went to town in both reviews, hopefully successfully. If either appears I will be very surprised, but you never know with GRL.

Climategate emails show that Phil Jones of CRU, acting as a reviewer of the CRU data used in the HadCRU gridded temperature, “went to town” to block the publication of criticisms of his handling of Russian data.

On Dec 15, 2009, it was reported that the Moscow-based Institute of Economic Analysis (IEA) issued a report “claiming that the Hadley Center for Climate Change based at the headquarters of the British Meteorological Office in Exeter (Devon, England) had probably tampered with Russian-climate data.” (h’t Jeff Id) There is an online technical report dated Dec 15, 2009, which states that it considered data released by the UK Met Office on Dec 8, 2009 in response to “increasing public pressure”. The Google translation needs volunteers. [Update: here iea1]

Russian version here. (h/t Anastassia) [Note: The Institute for Economic Analysis is not the Russian equivalent of the UK Met Office; the Russian Met Office may have a different view.] [Further note: maybe even akin to Cato Institute or CEI. Comments on data need to be cross-examined before relying on them.]

An email from Jones to Mann in March 2004 stated:

Recently rejected two papers (one for JGR and for GRL) from people saying CRU has it wrong over Siberia. Went to town in both reviews, hopefully successfully. If either appears I will be very surprised, but you never know with GRL.

Climategate has already affected Russia. On Tuesday, the Moscow-based Institute of Economic Analysis (IEA) issued a report claiming that the Hadley Center for Climate Change based at the headquarters of the British Meteorological Office in Exeter (Devon, England) had probably tampered with Russian-climate data.

The IEA believes that Russian meteorological-station data did not substantiate the anthropogenic global-warming theory.

Analysts say Russian meteorological stations cover most of the country’s territory, and that the Hadley Center had used data submitted by only 25% of such stations in its reports.

Over 40% of Russian territory was not included in global-temperature calculations for some other reasons, rather than the lack of meteorological stations and observations.

The data of stations located in areas not listed in the Hadley Climate Research Unit Temperature UK (HadCRUT) survey often does not show any substantial warming in the late 20th century and the early 21st century.

The HadCRUT database includes specific stations providing incomplete data and highlighting the global-warming process, rather than stations facilitating uninterrupted observations.

On the whole, climatologists use the incomplete findings of meteorological stations far more often than those providing complete observations.

IEA analysts say climatologists use the data of stations located in large populated centers that are influenced by the urban-warming effect more frequently than the correct data of remote stations.

The scale of global warming was exaggerated due to temperature distortions for Russia accounting for 12.5% of the world’s land mass. The IEA said it was necessary to recalculate all global-temperature data in order to assess the scale of such exaggeration.

Global-temperature data will have to be modified if similar climate-date procedures have been used from other national data because the calculations used by COP15 analysts, including financial calculations, are based on HadCRUT research.

An intellectually honest person will evaluate the data and come to a conclusion. The only time the agenda of original researchers comes into play is if your conclusion is different from theirs, or if they hide their data and say “trust us.”

As we long suspected and have since confirmed with the emails from ClimateGate, climate scientists have a clear agenda. This particular release suggests another example of climate scientists mishandling data to further their agenda. Once the data is reviewed and the claims of the release are confirmed, this will be yet another example of unscientific behavior on the part of several leading AGW protagonists. As a scientist I find handling of data by those at the Hadley Center and CRU unforgivable.

If you look at the first three items on their website (not a great translation, but the idea comes across) you get a clear feeling that they are the Russian equivalent of the CEI or Heartland Institute. Because of this their credibility drops to almost 0 as far as I am concerned. Plus, they are a bunch of economists for god’s sake, they are doing analysis (of the physical climate record) which is way out of their area of expertise.

(Sorry about the double post, I clicked the reply link in the wrong box).

Your complaint based upon a comparison like that is no more logical than Viscount Monckton accusing the “warming” community of being communists. That is the kind of thinking that lead to this contretemps in the first place. The CRU emails paint a very clear image of paranoid researchers convinced their work and their papers were being sand bagged by special interests (Big Oil). The real question here is whether that paranoia combined the associated political polarization has lead to the basic data becoming untrustworthy. The science is important. The data is important. The operators are not. We will always have new crops of operators coming along. Historical raw data cannot be re-collected.

“CRU emails paint a very clear image of paranoid researchers convinced their work and their papers were being sand bagged by special interests (Big Oil).”

The emails are much more complex than that, they were being sandbagged but it was by people trying to get at their data and methods. People like McIntryre and McKitrick. The emails prove that far from being sandbagged by big oil they were in bed with big oil and happy to be so.

Until we get access to the raw data and analyse how they got the published temperatures we’ll not know if there has been malfeasance or not. If nothing else from the emails it looks like a bunch of people who have allowed poltics into science. The reason for this is I believe the iconic hockeystick diagram, once they had published that they were political conspirators who couldn’t couldn’t go back. I think Wegman hinted at that in his report. Their behaviour, stifling debate, refusing to provide data and methods, smearing opponents, arises in my from a very real problem, which is that they made a huge mistake with the hockeystick, then tried to cover it up with other hockeysticks instead of coming clean. Further evidence that they had been economical with the truth would have, and maybe will nowl, brought the whole house of cards down.

N.A. Pivovarova and A.N.Yllarionov, the authors of this paper, are not climatologists. They are clerks at the Russian Federation Patent Office in Moscow. Their paper must be rejected on this ground alone.

Economists are pretty good at statistical analysis
of data reporting schemes, and they do a pretty
credible job of showing that Hadley selectively
chose station data (they put grids on it and show
a systematic preference for particular stations
over nearby ones) that:

Had missing data

Had been moved

Were in urban areas and known “hot spots”

They also pretty credibly show that Hadley
ignored most of the data from stations with the longest uninterrupted record of observations, and
data from most of the Siberian stations.

“If you look at the first three items on their website (not a great translation, but the idea comes across) you get a clear feeling that they are the Russian equivalent of the CEI or Heartland Institute.” I can say th same about RealClimate, yet haven’t heard that from your mouth. Now, shall we concentrate on the message instead of the messenger?

I cannot fathom how CRU could possibly spin this. This is fairly damning when combined with the Jones et al. emails. Thanks to Steve for continuing to place into context the emails with these events. You, sir, would have made a fearsome lawyer. ( Ummm. Hope that does not offend ).

It would also be interesting to see where 2009 ends up in the ranking of warmest years. Its snowing in copenhagen, freezing in mainland Europe,record lows in Canada. The UK MET office prediction is looking permature, as was their UK barbecue summer prediction which ended up a complete washout.
snip – stop this sort of piling on

Do you think it is coincidental that the EAU/CRU leaked e-mails were originally posted on a file server in Russia? Global warming and green technology are important issues for Russia, a major exporter of oil. Lots of big shot political speakers at Copenhagen, but where have the Russians been?

As I have commented in other forums and this forum on another thread, the day the ftp site was posted I ran a google check for proxy dump websites, the IP of the russian server was listed among those of other open proxys on at least 2 of the proxydump websites. That IP was, as was the turkish proxy, open for anyone anywhere in the world, to use.

I have commented on this topic as well. I was one of the first to see the posting on Jeff Id’s site and immediately downloaded the .zip file from the Russian server. Additionally, I traced the IP addresses through ICANN. The IP address blocks that were used are extremely common proxy gateways. I block these very same IP address blocks from my own servers because they are very often used as transport for IP spoofing and DoS attacks. I see no way that the IP origins or location of the FTP server that was used, could be considered when attempting to identify the original posting source. Anyone on the planet could have used that particular server and there is nothing to suggest that the leaked CRU content originated from someone in Russia. It would have been equally absurd to say that, because they wrote their message in English, that the files must have been distributed by someone in Britain. There is absolutely nothing to support either hypothesis.

A lot depends on how those graphs were converted to anomolies. It looks like they were forced to be equal during the 1960-1990 period. There should be some difference during that period if they were baselined against a common average.

When I look at the other graphs in the report it appears the missing stations were cooler than the one’s used which means the late 20th rise would be much less when the data is incorporated into HadCRUT.

Raven,
I used Babel for a translation and it looks like they used 1961 to 1990 as the baseline for both plots.

For calculating the anomalies of the near-surface temperature of air above the territory Russia in comparison with the level of 1961-1990 (accepted in the contemporary climatology for the base) we produced calculations as according to all 152 cells of the five-degree reference grid (476 stations), so on those 90 cells, data on which they are represented on 121 station from [vyborki] of the center Of [khedli]. In both cases they were carried out the averagings of all accessible annual data over the cell, are calculated deviation from base level for each cell, is produced the calculation of the average significance of a deviation with respect to to all cells in each year.

Raven,
They set the average over the 1961 to 1990 for both series equal.

Nick,
1950 is generally considered the point where CO2 became significant. There clearly is a difference in the trends from 1950 forward. Also keep in mind this is just the raw data without any adjustments from CRU. It would have been nice if they would have plotted just the stations not used by the CRU as the trend difference would be even greater.

Russia do leverage their Gas against other countries, but actually only insofar as their allies in the past recieved it at less than world prices in the first place (Ukraine for example). This is what people mean by energy security. Less Oil/Gas use also affects the oil price, which when lower obviously impacts Russia’s rather undiversified economy. As lower fuel use in the West is generally a threat to Russian vital interests (selling fuel), it wouldn’t surprise me if they were set up to put as big a spanner in the works as possible.

That is not to say this isn’t justified on other grounds; it’s just that Russias interests coincide with those of the “deniers”. I’ve always thought that politicians would be on safer ground making arguments from the energy security angle than making them on dodgy science. Invest in alternative energy, yes, but there’s no need to destroy public trust in the integrity of the scientific process while doing so.

@Jason Lewis: The last time I looked at this stuff, the Asian Sector of the Russian Federation was the sixth largest contributor to the GHCN in terms of station ids.

My guess is, given that there are fewer stations per surface area (compared to, say, the U.S.) changes to data from stations in Siberia would have more of an impact on the data set but one cannot answer the question conclusively without actually crunching the numbers.

In the paper, they discuss how data covering 40% of the Russian surface area is completely removed from the record. Not massaged, not treated. Just ignored and replaced with estimates from CRU’s models.

Hmm, for me the source of criticism shouldn’t count (that’s something the Mannians of the world do too often). So, it doesn’t matter whether the institution critizing is scientific or just a layman (economists, though they are usually pretty knowledgable when it comes to data and data mining). If the criticism is good, then you have to respond in a likewise way.

EVERY side in a debate wants to be able to discount the genuineness of the arguments of the other side; it is part of the debating process. It is, of course, off topic, but it wins points.

Overall, THIS side of this debate has always – in my years of looking into it – shown it stays on topic, always discussing the data, the science, the process, the theory, the connections between the data and the theory and the science and the process.

It does not matter if the criticism is good. In fact, it is exactly because the criticism is good that it must NOT be responded to likewise. To respond to it with points about data is to lose the argument, because they KNOW it can’t stand up to scrutiny. The only winning strategy is to deflect attention elsewhere, hoping the discussion will move on to something else. And it always HAS, so why should they choose any other strategy? Even Wikipedia says this about Phil Jones:

“The content of the messages generated controversy, in part because Jones refers to a ‘trick’ used to ‘hide the decline’ in filling in the values of a chronological global temperatures diagram. Jones said in a statement[6] that ‘[t]he word “trick” was used here colloquially as in a clever thing to do. It is ludicrous to suggest that it refers to anything untoward.’ He did not address his use of the phrase ‘to hide the decline.'”

Notice that the devastating part was “hide the decline,” and he focused on the more easily sloughed-off “trick” quote. At least Wikipedia made mention of him ignoring the “hide the decline” quote.

snip – sorry. but no speeches on topical threads. Preferably no speeches at all.

We seem to have a pattern developing with the apparent selection of only the stations showing the most warming being chosen to represent locations even if one must narrow it down to a single station for an entire continent as was done in Antarctica. GISS does it, too. Four stations represent all of California, for example, and all four stations are on the coast.

You bring to light a recent question I have had. Since so many stations have dropped out of the GHCN network in the US, has anyone performed an in-depth analysis of how the record would or does differ from the complete record? I have always been very puzzled as to why so many stations have been dropped. And not like there are just a few, but the majority of the stations. This has always concerned me. Isn’t the gathering of data generally more accurate with the inclusion of more data sources rather than less? This has always smelled of bias to me.
Steve: USHCN is added later. They’ve just redone their USHCN methods and done entirely new adjustments tho.

On the whole, climatologists use the incomplete findings of meteorological stations far more often than those providing complete observations.”

This is a serious accusation. It would imply CRU prefers stations that need infilling to those that do not. And therefore the infilling algorithm is flawed or it empahizies warming in the same way Mannian algorithms create hockey sticks.

The same can be said for emptying gridcells completely so any warming that can be conjured in adjacent areas can be smeared over the empty cell. This is Steig all over again, and speaks to a common playbook of magic thermometers and transcontinental teleconnection. SM picked up on this long ago with “The Rain in Maine falls mainly on the Seine” or some such amusing title. The paleos and observational temperature reconstructionists are in symbiosis. The paleos need the right curves in their calibration periods so they get hockey sticks with decentered PCM/RegM/RCS etc. and the recons provide the goods. How convenient. Except for the email, the code, the divergence, and this historic website, which will ultimately get credited with revolutionizing how science gets done in the future.

I just got through the google translated document. The crux of the argument is that the CRU cherry picked data following the same methods that have been done everywhere else. They ignored data covering 40% of Russia and chose data that showed a warming trend over statistically preferable alternatives when available. They ignored completeness of data, preferred urban data, strongly preferred data from stations that relocated, ignored length of data set.

One the final page, there is a chart that shows that CRU’s selective use of 25% of the data created 0.64C more warming than simply using all of the raw data would have done. The complete set of data show 1.4C rise since 1860, the CRU set shows 2.06C rise over the same period.

The paper failed to go the final step and compare a strictly rural set of data vs the CRU data.

And they concluded by saying “if this is what the CRU did with Russia’s data, imagine what they must have done with the rest of the world.”

Clearly they are a political economic organization, but raw data is raw data.

When you read it for yourself, keep in mind that Russian uses double negative “not never including data” means “never including data.” And airborne weather stations just means stations that were moved at some point. The translation is rough, but not impenetrable.

On the positive side, they have nifty widgets for getting data. On the negative side, the google translator is not translating the fields and the written Russian is a bit beyond me. So it does look like the raw is there, just not easily accessible by the likes of me.

The “raw” data is on the UK Met site. (Technically, they explain that it is a mix of raw and adjusted data, but don’t identify which is which.)

It is easiest to understand the graphs and charts in the Russian paper if you read the translation along side the original Russian.

The claims are pretty standard and easily defended:
1) Russia is needlessly under-represented in the HadCRUT analysis.
2) Temperature data for 40% of the Russian land mass was ignored
3) Many of the longest temperature records were ignored
4) A preference for incomplete data series was demonstrated
5) A preference for stations that changed locations was demonstrated
6) A preference for stations in cities was demonstrated
7) Collectively this resulted in a 0.64C exaggeration of the Russian temperature rise compared to using all data with no filtering. The Russians chose not to publish a temperature history using only appropriate rural sites.

There is nothing fancy or technical in the paper. Just a look at which data was used and demonstration of how the selections impacted the results.

I was under the impression that only the subset that CRU was using was at the MET site. That doesn’t look like the full 400+ that the main paper is referencing (at least when I use the download station data function). Am I not looking in the right spot ?

I agree there is nothing fancy or technical in the paper and it is fairly easy to follow along, but for better or worse my trust in “climate studies without a second set of eyes checking things over” is pretty much kaput. So it should be a quick check to see if I get what they do 😉

“It all depends who and what the Moscow-based Institute of Economic Analysis realy is. Are they credible?”

This is a fallacy called an “argument from authority.” The ONLY thing important to science is the RAW data. That was not collected by anyone with a political agenda that included the word “climate.” Asking if they are credible is like asking if the Hadley lot were “communists” or accusing skeptics of being “right wing” or “conservatives.” It may or may not be true.

However, if the raw data is available, and you doubt their conclusions for any reason, then you can reanalyze the data yourself, add your own adjustments and come to your own conclusions. There’s nothing mysterious about the math involved. It’s just laborious. Multivariate simply calls for very careful consideration of the nature of your data and the actual character of the model you plan to use.

This is the crux of the battle over freedom of information and CRU raw data and methods. If the “Team” trusted their own work and were convinced it would stand scrutiny, then they should have released the data, published their methods in detail and then waited for the skeptics to do their work and either disappear in embarrassed silence or swing over to supporting geoengineering the climate.

As it is, in the CRU emails we are seeing evidence of convictions and data selection based upon whether it matched expectations or not. Kevin Trenberth’s complaint that the global data for the last 10 years “must surely be wrong” reflects this and also shows how far from disciplined scientific analysis the Team’s thinking had drifted. You don’t blame the data.

The founder of the Institute of Economic Analysis has a Wikipedia page. He was director till 2000 and is still listed as the blog writer. From 2000-2005 he was Putin’s senior economic adviser. He is an outspoken skeptic.

“We did not see so far any scientifically proven fact that there is any any clear relations between carbon dioxide emission of anthropogenic character and global warming. We do see a lot of confirmation that global warming goes due to natural forcing, not to anthropogenic forcing.”

I’ll add in the head post that the Institute of Economic Analysis is not the Russian equivalent of the Met Office. That’s relevant.

At this blog, let’s try to focus on what we can learn about the data sets themselves without editorializing on whether Russia exports oil (and Denmark imports oil.) There are many venues to do so but not many venues to discuss details of the station data sets. So let’s stick to knitting as much as policy.

No Syl,
It depends on whether the CRU can show us that THEY are credible and did not ignore 40 percent of the Russian data. Merely saying “We did not ignore Russian data.” is nowhere near enough given what we now know about the CRU handling of any temperature data.

Further I would ask that the CRU prove that they did not run it through their nuanced and complicated and sophistication hockey stick climate modeling software of dubious provenance.

Time to dig out all the analysis that Steve and others did on Russian loctions a few years ago. As I recall the general notion was that UHI was significant (increasingly poorly insulated steam pipes I seem to recall), micro-site location issues were significant and there was an odd pattern of station locations.
On the other hand the Russians sure have plenty of motivation to put their fingers on the scale.

Even in the US, not all station data are used, for various reasons. That in itself appears to be common, known and official practice, with the same kinds of skeptical doubts about this practice. And it is already known that there are different points of views on the handling of russian/siberian data, so again I think the information to substantiate such claims has not been presented in a way that addresses obvious questions.

Why are the editors of two journals sending papers critical of CRU’s data analysis in Siberia sending papers to Phil Jones for review? They must know that Jones is likely to give a negative review even if the papers have merit. The answer depends on who the editors chose as the second reviewer. With one negative and one positive review, the paper should be sent to a third reviewer along with the first two reviews. The third reviewer would be in an ideal position to provide the editor with an accurate assessment the merits of the paper in light of Jones’ rebuttal. On the other hand, the editor could have chosen a second or third reviewer whom he knew would be as likely as Jones to reject the paper. Is it fair that a paper critical of a powerful organization’s work can be published only after being critiqued by a representative from that organization? Certainly, but I wouldn’t expect the real world to work any differently. After all, the Russian authors probably don’t enter the review process with the same record of scientific accomplishment as CRU. Of course, the journal editor would never send a prominent scientist’s paper to be reviewed by the people whose work he was criticizing. The prominent author might learn of this outrage and ask someone on the editorial board to investigate. Hopefully the same editors will now be afraid of treating M&M papers in the same outrageous manner, fearing that the editor might end up testifying in front of Congress.

Do the editors of these journals properly moderate open scientific inquiry or simply provide an echo chamber for “scientific consensus”? They have the power to do either, usually with little oversight. I’d be much happier if journals and/or the scientific societies that oversee them didn’t have official positions on global warming and write editorials about the subject. Almost by definition, official positions conflict with the scientific method.

Frank, I think it is OK for the editor to send a paper critical of CRU to PJ, to allow him to defend himself, as long as the editor picks other unbiased reviewers as well and bears in mind PJ’s obvious bias when forming a judgement about whether to publish the paper.

I agree with you to some extent, but it is hard to tell without seeing the paper. If the paper is clearly an attack on CRU methods, then it might be appropriate to get factual information for consideration by independent reviewers from Jones. However, if the paper merely brings in data not considered previously or provides an alternative analysis and viewpoint, there is no need to involve Jones. For all the editor knows, Jones could be a competitor who is already be working on the same ideas as the Russians. Involving Jones in the review process would allow Jones to stall the review of the Russian’s paper and rush out a paper of his own.

It seems to me at this point in time, that there should be some sort of independent government commission setup to investigate how temperature data is collected, and processed. Not only the CRU data, but NASA/NOAA data as well. Is there any organized group, or online petition calling for this?

The link below is an eye opener. Is it true that the actual thermometer readings around the same areas as the Yamal trees used in the infamous hockey stick actually prove there is no hockey stick? If this analysis is correct, then why isn’t there a demand by the thousands of honest scientists all around the world to have the hockey stick team withdrawal their findings once and for all?

There are a lot of new people on this blog who would do well to dive into the archives. There are very few revelations coming out so far, that were not inferred by denizens of this site months or years in advance.

For instance, here is one that addresses your question from four years ago

I’m very curious as to who is behind the analysis of the Yamal data shown on the “greenworldtrust.org” site.
Is this formally published elsewhere where it could be cited academically? I cannot extrapolate going through the organisation’s site back to a site for 3 people involved in Transition Towns in the UK if there is a statistician or other academic at the heart of the matter.

The email at the BBC was from Richard Black who passed-on remarks from Mann regarding Paul Hudson later featured in the leak.
It was not the leaked package which does not appear to have been compiled at the time.

As I said the peer review process is broken, at least in the field of climate science research. Either it is repaired immediately or it will soon become the joke of the scientific community. Personally much of the theory that passes as climate science research is not better than astrology, and it should be treated as such.

Being asked to review the papers of your rivals in the field is almost unavoidable in most fields, where the number of experts in the subject matter can be counted on one or two hands. Likewise, when you submit a paper for review, it’s usually possible to tell which review was done by your rival by the sort of comments that they make. And, it’s not that unusual to really blast your rivals if for no other reason than to slow down their publication cycle.

But these deliberate attempts to squelch the publication of rival positions does seem to go well beyond the normal run of the mill scientific competitiveness and pettiness. I can’t imagine there is anyone in the academic community that isn’t made uncomfortable by such organized attempts to keep unpopular opinions from reaching publication. These are people’s careers and very lives that are on the line. I can’t imagine that anyone who publishes for a living isn’t more than a bit uneasy reading this sort of thing. If you are a grad student or post-doc working in any field, what lesson are you going to take from this about publishing results that go against the grain?

The only lesson I can see a student can take from this is that he/she either joins them or finds another avenue to have their work published. Eventually climate science will have to be “reborn” before it’s going to be taken seriously. snip – piling on

What strikes me about Jones’ email is that he asserts proudly that he shot down these papers without offering in the email a single scientific reason for their unacceptability. It sufficed as explanation that they disagreed with the preferred view.

Where is there any suggestion of openness to the possibility of contrary evidence?

It is not uncommon in the scientific realm to have researchers who are direct competitors reviewing your work. Sometimes you can request certain individuals or institutions not be considered in the jury pool but not always. Some areas of study are just that small that the population of reviewers are limited.

Translation needed – Anastassia? Also more info about the IEA and the author of the paper N A Pivovarova – googling gives a N A Pivovarova as the co-author of a physics paper on light scattering in 1987…

The SAO/NASA Astrophysics Data Service has a great archive, a very powerful search engine, softcopy versions of a surprising number of the items, citations statistics, and more. They are most complete in astronomy, but coverage of physics and related fields is also good.

The site is adsabs.harvard.edu/basic_search.html

They have 23 citations for N(atalie) B Pivovarova, although it is not clear that they all refer to the same person.

Would it be a fair assumption that most of the proxy studies that cover Russian territory use CRU gridded data for the instrumentation period? If so, and if the data has been modified in ways it shouldn’t have been, this would seem to call into question all of those studies. It will be interesting to see how this shakes out.

Speaking of letting the foxes frame the terms of the hen house debate. Is it any surprise there’s bickering about the meaning of sampling the air at random spots over very short geological time-frames?

For all the attractiveness of this release, I’d be wary of accepting this at face value immediately. The best approach I suggest is to examine the data again and investigate.

I do recall several other people having issues with the Siberian data before, wasn’t this data at the heart of the original “I won’t hand over my data because you’re only trying to find something wrong with it” complaint by Warwick Hughes ? Warwick did some work on Siberian records showing some quite puzzling records.

I commented elsewhere on Jones et al (1986) Northern Hemisphere Surface Air Temperature Variations 1851-1984 Jour. of Climate and Applied Meteorology 25, 161-179. I will comment here due to the long interest in Russian, particularly Siberian records.

While minimizing urbanization effects, the paper says, “Both van Loon and Williams (1976) and Jones and Kelly (1983) have shown that the twentieth century warming was strongest in high latitude regions (particularly over Greenland and northern Siberia)”

Look at the parenthetical part. That is not said at all in van Loon, and if it is laid out in Jones and Kelly, I did not detect it. Yet only those two sources are the authorities given for the first (to my knowledge) singling out of Siberia for exceptional warming

Looking at the Harry readme, it is full of problems resolving duplicates and merging updates into the existing record issues. Russia is specifically mentioned, as is several other countries including Australia. The updates come with unreliable metadata, and the update process is at best guesswork.

I think that the most likely reason lots of Russian is missing, is that they are having trouble getting the data in a usable form, ie a form compatable with thier tool kit.

The CRU database is basically broken. I assume the other databases are getting the same junk updates, and so are also in a sorry state.

If one can believe readmeHarry, the CRU databases are in a mess, but this is no reason to assume that the Russians’ (or anyone else’s) data is in similar bad shape.

Much available information from a very large area was left out of CRU’s climate reconstructions. It would help the CRU’s cause if they would detail why the chose which data sets and how they modified them after selection. Had they been doing this all along they would not be in the fix they are in.

Vincent GRAY had already mentionned the siberian thermometric discrepency in a (1999 ?) paper of which here is a 2003 updated version, available on the marvelous site of John DALY (whom death appear as a “cheering news” by Phil JONES).

Translation of the “Conclusions” section of that PDF (sorry, guys, it’s 21 pages and it’s way too late at night…)

Note:

1.) Things in [square brackets] are my notes, not in the original text.
2.) Please refer to the actual PDF for the charts (Figures 8 and 9).
3.) Yes, the language is *that* dry and scientific in the original. Also, Russian grammar allows for far longer sentence structure than English, so please be patient with the “run-on” sentences.

——————————————

9. Results

The analysis of the Hadley Center and the Climate Center of University of East Anglia’s usage of the meteorological data collected from Russian weather stations, shows that the data selection conducted by them leads to the following results:

• The statistical significance of Russian weather stations in the calculation of global temperature is lowered compared to the statistical significance of Russia’s territory in the geographical area of the world’s landmass.
• The selection of weather stations was conducted in such a way as to result in complete lack of coverage across over 40% of the country’s territory.
• Datasets with the longest continuous observation periods, which are the most valuable for the evaluation of temperature trends across 1.5 centuries, were used to far less than their full capacity.
• While selecting the datasets, preference was given to datasets which were missing measurements, while more complete datasets were ignored completely.
• Also while selecting the datasets, preference was given to stations whose location was changed, compared to stations whose location remained constant.
• When given a choice, the preference was given to stations which were located close to population centers, including cities with obvious “heat spots”.

In other words, the associates of HadCRUT systematically selected the meteorological data, giving preference to lower-quality datasets compared to higher-quality ones; to their [the lower-quality datasets’] shorter and less-complete observation sequences; to the data, collected from stations that were moved around and/or located in areas with higher population density. Besides that, it looks like they [HadCRUT] intentionally discarded the data which characterizes the temperature situation in approximately 40% of our country’s area.

In order to test to which extent this [selective-data] approach would affect the final results of the calculations, it is necessary to conduct a comparison of the results based on the [above-mentioned] “narrow selection” to results produced from the analysis of general data.

For analyzing the anomalies of the near-surface air temperature over the territory of Russia compared to the levels of 1961-1990 years (accepted in modern climatology as the baseline), we had conducted an analysis of all 152 cells of the five-coordinate-degree “mesh” (476 stations), as well as of those 90 cells from which the Hadley Center selected the data from 121 stations. In both cases, we averaged all available data for the specific cell, calculated the deviations from the baseline for each specific cell, and calculated the average significance of the deviations for all cells for each year.

Figure 8: Deviations of temperature in average across the territory of Russia compared to the levels of 1961-1990., 11-year smoothing.

The results presented in Figure 8, show a significant discrepancy in the evaluations of the two methods. The amount of warming in Russian territory over 130 years – from the 1870’s to the 2000’s, derived from the data used by the Hadley Center (90 cells, 121 stations), comes out close to 2.0 degrees Celsius. However, the calculations based on [all available] datasets (152 cells, 476 stations), show that the amount of warming for the same period was much less – about 1.4 degrees Celsius.

Even if the period of 1955-1995 shows a similar overall pattern of temperature series calculated according to either method – moving farther into the past, as well as into the last decade, there is a quickly growing discrepancy between the two series. Meanwhile, it is obvious that for the time period up to the mid-1950’s, according to the series based on the datasets selected by the Hadley Center, there is a characteristic decrease of the temperature based on the record which includes all 476 stations – but for the time period after 1995, the series based on the selection by Hadley Center, there is a characteristic increase.

In the 2nd half of the 1940’s, the temperature anomalies according to the HadCRUT selection were 0.14 degrees Celsius lower than the results according to the entire [476-station] measurements across the entire Russian territory; in the 1910’s those anomalies were already 0.26 degrees lower, and in the 1870’s – [a deviation of up to] 0.56 degrees Celsius – see Figure 9.

Taking into consideration the negative divergence of the temperature series up to the mid-1950’s (up to 0.56 deg.C.) and the positive divergence of the temperature series in the mid-1990’s (up to 0.08 deg.C.), the overall artificial increase of the warming, committed by the associated of HadCRUT, for the Russian territory from the 1870’s to the 1990’s can be described as minimum 0.64 degrees Celsius.

At the same time, this consideration is very conservative, since for the calculations of temperatures across Russia, all data contained in the RosHydroMet database was used – without any selection based on the data’s contents, as well as without any necessary corrections, for example, for the effect of cities’ temperatures.

Discrepancies of this scale, for a country as large as Russia (12.5 percent of the global landmass), are significant for creating discrepancies in the scale of global warming, presented by HadCRUT and used in the IPCC reports. In order to calculate the scale of such intentional increases, as well as in order to assure the accuracy of the data on global temperature increase, it is necessary to recalculate the entire massif of global temperature data.

If the procedures of processing the climate data, discovered in the scenario of Russia, were also used in respect to the data from other regions of the world, then the unavoidable correction of the calculation of global temperature and its change in the 20th century may turn out to be quite significant.

I’ve had considerable experience in the use of translated Russian technical papers. Your “Conclusions” translation has a very typical use of language, structure and expression. There is also a very large hangover in Russia from the Soviet bureaucracy, wherein failure to be complete and meticulous permitted you the opportunity to attend such a weather station in Siberia for about 20 years or so 🙂

One consequence of this hangover is that lesser Russian mortals remain extremely careful

You’re welcome – hope this will be useful to folks who are lamenting the mess that Google Translate makes trying to fit a Slavic language into English patterns (yeah, good luck with that, we have 6 noun cases, 3 genders, and a bunch of other “features” that, although making the language far more flexible to work with, make translation into English and Romance languages an utter PITA).

I’ve tried to stay as close as possible to the original sentence structure, hence the need for explanatory [comments] inline with the text.

May I offer an alternate take on the Russian report. It links to the MET office release of records December 08 of a subset the MET (?) or HadCRUT viewed as representative of the entire database. More records to be made available later.

The Russian authors make the case that the stations selected for Russia in this subset are not representative. They do not state that the HadCRUT process as a whole ignores a lot of Russian records. Indeed only an analysis of the CRU station list and their process could do this.

I thought the MET made it pretty clear on their website that they were showing all of the data that they could legally share. Since the Russian federal agency is the only source of historic Russian temperature data, there cannot be any legal constraints forcing the MET to not publish other Russian data. (Maybe for border regions, but certainly not for the inner Siberian areas that were intentionally excluded from the analysis).

Additionally, the MET choosing to ignore data covering 40% of Russia’s land mass is only one part of the demonstration of selection bias. The other parts are independent.

I’m no expert but harry_read_me.txt has some interesting comments about Russian data and gaps as well as lots tables of data — perhaps these may mean something to those who understand the numbers. Search for “RUSSIA” and scroll around. For ex:

It’s the same story for many other Russian stations, unfortunately – meaning that (probably)
there was a full Russian update that did no data integrity checking at all. I just hope it’s
restricted to Russia!!

DO YOU SEE? THERE’S THAT OH-SO FAMILIAR BLOCK OF MISSING CODES IN THE LATE 80S,
THEN THE DATA PICKS UP AGAIN. BUT LOOK AT THE CORRELATIONS ON THE RIGHT, ALL
GOOD AFTER THE BREAK, DECIDEDLY DODGY BEFORE IT. THESE ARE TWO DIFFERENT
STATIONS, AREN’T THEY? AAAARRRGGGHHHHHHH!!!!!

1. Russia is not mentioned in the RealClimate post.
2. RealClimate provided a link to CRU data… and the link is not working today.
2a. The link is stated to be to temperature data, no metadata is mentioned.
3. We still don’t know why CRU chose those Russian stations.

In the two sets, one has 223 (daily) and one 476 (monthly). The 223 one is for the USSR and is truncated outside of Russia when the split happened. There are 149 stations in Russia there (well, if my regex didn’t lie!). Just thinking it wouldn’t be impossible to get a 25% number from 476 if a few of those were discarded too. I will try to check against the Met data.

Just looking at the description of Aisori and data practices, the Russian climate people are MUCH more adept at programming. My biggest surprise of all the email stuff was how even the basics of programming were out the window. I was expecting something very professional, like you would expect from NASA or something, or like it seems to be done at least in Russia.http://meteo.ru/english/tech/

(and note, there is an English version of that site, I’ve seen a few google translate links: http://meteo.ru/english)

I am going to try and take this data and use a cherry picking algorithm to try and make warming as dramatic as possible, and then cooling as dramatic as possible (using 25% of the data). Then check the CRU stuff to see where it lies inside that area. If it is close to maximum warming I think it would say something, if it is in the middle, then maybe not so much. (I’m picking this task as I have experience in both programming and picking cherries in the Okanagan, long ago).

Posted Dec 16, 2009 at 5:27 “As I said the peer review process is broken, at least in the field of climate science research. Either it is repaired immediately or it will soon become the joke of the scientific community.”

Well said Peter but I should add that climate science has already become a metaphor for performing a dubious self serving analysis.

In my field of electronics manufacturing we are routinely required to submit corrective action reports to our customers whenever a problem occurs in the field with one of our products. The team charged with completion of this CAR (Corrective Action Report) must perform a detailed analysis of the customer complaint and then determine a root cause. Finally a description of the appropriate corrective action we promise to undertake is documented in the final report to the customer in order to prevent a recurrence.

The process may take days or weeks to complete and requires skilled detective work and hard nosed interpretation of the analysis data. The temptation is always there to minimize our culpability or otherwise mitigate the seriousness of a particular defect but as professionals this be not be permitted. Lately, whenever anyone makes a suggestion about the data tending toward subterfuge, I’ll blurt out an admonishment like,

“Hey man, don’t climate science the data!”

It’s always good for a laugh and it really reddens the face the perpetrator. But the effect on the group is palpable. There is a moment of silence after the laughter and then we somberly re set the analysis compass back to true north.

Could these maps be showing anything related to this story? They’re vs. GISS, because the HADCRUT maps are mostly blank in Russia. If GISS is in relatively close accord with CRU on a regional scale, then the Russian area seems like it might interesting in this new light.

Alternate translation of the Russian headlines at http://www.iea.ru.
Note: the language is somewhat long-winded in the original, so please bear with what looks like run-on sentences – they do make sense in Russian grammar.

—–

There is no scientific consensus regarding the climate question.
Novaya Gazeta (The New Paper), 16 December 2009.

– Instead of appointing and prosecuting false targets, the political leaders gathered in Copenhagen should concentrate on other issues – developing policies that promote more effective human adaptation to climate change, speeding up economic growth, development of free trade, protection of property rights, and strengthening democracy.

How warming is created. The Russia Scenario.
15 December 2009.

If the procedures used to process the climate data that were found in the Russian scenario, have similarly been used on the data gathered from other regions, then the correction in calculating the global temperature and its changes in the 20th century can turn out to be quite significant.

Theses on climatic changes.
Gazeta.ru, 7 December 2009.

The methods suggested by the proponents of climate alarmism to “fight the global warming” by means of reducing carbon-dioxide emissions are not only scientifically unsound – due to the lack of extraordinary characteristics of the current climatic changes, but also unbelievably expensive in economic terms. If actually implemented, such measures will become, for nations with low-to-average economic development, a path that will actually cut off their ability to reduce the disparity between themselves and the more highly developed nations of the world.

About modernization of Russia.
Radio Svoboda (Radio Freedom), 12 November 2009

Legal and political revolutions precede turbulent economic development. However, the absence of such changes, the absence of transformation, leads to significant delays in development. The political and legal institutions in our country are not being modernized in such a way as to keep up with the events of the surrounding world.

O/T I know but I have just watched a recording of last night’s BBC Newsnight program in which their “ethical man” Justin Rowlatt demonstrated the “science of global warming” in his kitchen with an audience of members of the public.

The reason for my mentioning this programme is because Sir David King, who used to be the UK chief scientist, was there and when challenged by one of the observers of the experiment about the behaviour of the scientists at the UEA, he made the claim that the “hack” was highly sophisticated and that it took place over many years and included “interception of mobile phone messages”. I think he was trying to imply that it was the russians or some state power that had done it.

This is the first time I have heard such an outrageous claim and the mobile phone issue is completely new to me. Can anyone shed any light on this.

Here is the link. It only works for UK residents. Would be good if someone could upload it to youtube. I can’t as I am in France.

I agree that this analysis of the “hack” as an inside FOI process that was leaked somehow makes most sense. However the claims by King of “mobile phone conversations” being intercepted was so so ludicrous that I wondered if there was new information. After all, Sir David King is supposed to be an honest person even though he is a warmist.

Maybe he needs to be challenged on it by the news media. I will send an email to the editor of Newsnight to ask him if they know more.

Some of the figures in the russian document are easy to understand.
Grafik 1 on page 8 shows data from stations North of 70 degrees, not included in CRU – no warming.
Grafik 2 on page 11 shows the same lack of warming for non-CRU stations in a block in the Karelia area, NW russia.
Grafik 3 on p 12 shows warming in 3 stations that ARE included in CRU.
This looks quite damning but we have to be careful about possible cherry-picking.
It’s clear from their website that the IEA is not unbiased.

first time poster, and I have a question that’s been bugging me for two weeks.

Do you have any figure regarding the percentage of the CRU data not released under FOI due to confidentiality agreements, and what their weight is on the CRU climate models.

I have just taken the issue with a few commenters at the Real Climate blog (not sure if they’ll publish my latest post) because they keep saying that 95%-98% of the CRU data was already available to public, and that your persistence with the FOI was merely about a very small percentage which would not have changed the predictions of the climate models produced by the CRU anyway.

Secondly, has the CRU ever offered you whatever raw data not covered by the confidentiality agreements AND the names of the national Met services where you could purchase the rest?

I am only focusing on the data here and not the code for the computer modelling because of the RC commenters’ focus and the same.

Thanks, mate.

Steve: They stonewalled everything. Refused even to identify what countries they believed to be covered under “confidentiality”.

I am angry with myself a little now for not having read in a timely fashion the Willis Eshcenbach article that James Lane just linked to. Perhaps I was discouraged by its length and apparent complexity at the time. That piece not only answered all the questions that had been bugging me for weeks (especially the ones about the specific figures of 95% – 98%) but it also provided the definitive context to the FOI leg of the CRU emails.

Incidentally, my latest comment on the Real Climate blog (mentioned above) seems to have been considered a ‘noise’ detracting from their ‘signal’. Well, here is the comment that didn’t make it thorugh their quality check:

For the love of all things green and cool, could someone who is in-the-know tell me what is the percentage of the data the CRU used for its climate modeling that is NOT covered by the confidentiality agreements? This question will be moot in six or seven months’ time since the UK Met service announced that they’ll be seeking permissions from other Met services in order to release all of its raw data, but it is extremely annoying to see a few evidently non-climate scientists seeking mileage out of the debate by bandying the figures of 95 and 98 percent.

The information I’ve read is that the GHCN database includes all the available data that’s not restricted by distribution agreements. This is what GISTEMP uses, and represents about 95% of the data used by CRU, the other 5% being the data subject to such agreement and which McIntyre et al have been screaming about being “hidden” for all of these years.

and at #186:

Oh, Jonesy, and as far as that other 5% that’s restricted and used when creating HadCRUT (but not NASA GISTEMP), the data can be had from the individual countries, though unfortunately often for a fee.

See, McIntyre et al have been asking for restricted access data for free that in some cases CRU had paid for, rather than going to the source and paying for it themselves. Understandable, who wants to pay for data? And not all of the restricted data has been paid for, but still – you do get the point, I hope?

I skip the blindingly obvious questions of “where did you read the figure of 95%?” and “do you want me to believe that Steve McIntyre would not spare a few lousy dollars to buy the missing data from the gazillions that he was supposed to be receiving from the fossil fuel industry?” I skip these questions because this time I want to hear from climate scientists, not from cheap propagandists.

What is the percantage of data used by the CRU that is NOT covered by the confidentiality agreements? What is the weighing of the ‘missing bits’ on the CRU’s climate models? Has Phil Jones ever made an attempt to release the data not covered by confidentiality agreements AND provide the names of those Met services that sold the rest?

I am not interested in the code for the climate model, or Jones’s right to scientific vainglory, or whether there are other sources proving the AGW. My questions are only about the CRU data because specific figures of 95% and 98% are being spread around, without any citation, as the percentage of the freely available CRU data, and attacks are made on others on this basis. The answers to these questions are important to me not because I intend to replicate the CRU’s climate science in my bedroom on my Pentium 4 PC but because I want to assess the credibility of various claimants like Phil Jones, Steve McIntyre, Marco and dhogaza.

Oh, Jonesy, and as far as that other 5% that’s restricted and used when creating HadCRUT (but not NASA GISTEMP), the data can be had from the individual countries, though unfortunately often for a fee.

See, McIntyre et al have been asking for restricted access data for free that in some cases CRU had paid for, rather than going to the source and paying for it themselves. Understandable, who wants to pay for data? And not all of the restricted data has been paid for, but still – you do get the point, I hope?http://www.realclimate.org/?comments_popup=2351#comment-149618

My comments followed that quote. Apologies for creating some confusion.
Steve: This is OT. Why must people persist in breaking blog rules.

“I have just taken the issue with a few commenters at the Real Climate blog (not sure if they’ll publish my latest post) because they keep saying that 95%-98% of the CRU data was already available to public…”

Then ask why, if Dr. Jones was finding the MM team so irritating, did he not just point them to the data that was already available. I guess it’s because knowing where the raw data is doesn’t tell you which raw data Dr. Jones used. The same applies to the lost data, if it was lost why didn’t he simply say that in response to the first e-mail. Very perplexing.

sHx: They were being cutely coy about the question. “The data are publicly available” is not the same thing as saying which data they used. Sure, the 95%-98% of temperature records are on public servers, but then you have the questions of 1) Which did they use? If you look closely at Harry_Read_Me, about a project to *attempt* to reconstruct their data, you see that their processes typically reject more than half the data fed into them. That leads to a *very* pertinent “which half?” question. Maybe all the temperature data comes from UHI affected weather stations? How do we know without a station list? 2) If you look at the Hockey Stick analysis by Mc and Mc, you see that a small set of data, 20 out of 112 records, created the whole controversy (http://www.uoguelph.ca/~rmckitri/research/McKitrick-hockeystick.pdf). What “Sheep Mountain Curiosities” lie in that 5%? Again, an especially pertinent question, since we know that the processes that lead to IPCC reports yield things like the most unrepresentative weather station in Antarctica being the only one used for the entire continent!

Then again, based on Harry_Read_Me, CRU itself doesn’t have a clue which stations it used!

Reproducibility? Falsifiability? Knowledge transfer? Values associated with science are gone right out the window with this CRU.

So, is this more confirmation bias? Apparently there’s a method for automatically selecting “good” stations (more than likely similar to Mann’s method for automatically selecting “good” proxies, ie. throw it away if it doesn’t look right, after fiddling around with the parameters for a while).

Apparently the method is discussed in the literature. Well, does anyone know what it is?

One of those rejected papers about Siberian temperatures may have been by me. The time is about right. I got it rejected because of nonsense from a reviewer and the editor saw it as an attack on him when I critized the quality of the review. After that, I gave up the idea of ever getting something AGW critical published in a journal.

Sorry to hear that, Lars. It would be fascinating to know just how many papers have been rejected over the last decade or so, not because of their science but because the Team thought they had the “wrong” answer. More fascinating yet to see what they had to say.

And when I think back to all those years of getting arrogant “if it isn’t in the peer-reviewed literature then it doesn’t count” arguments, well it’s just infuriating.

Institute of Economic Analysis (IEA) was founded in 1994 as an independent, non-governmental, non-political and non-commercial organization.

IEA’s main objectives are:

fostering development of economic and social sciences
studying Russian and foreign experience of solving problems of economic policy, market economy and ecomnomic reform
hold consultations concerning economic and social policy with the Russian government bodies and NGOs
and publish economic research materials
IEA focuses on such areas as mutual influence of economic growth, economic and political freedom .

The website is available only in Russian (English version under construction) and it offers more detailed information about Institute’s activities and projects. There are downloadable publications in Russian available at the website.

If you go to the WMO website there is confirmation of their input in the choosing of the weather stations in this press release on 2009 Jan-Oct showing 2009 being in the top 10 warmist years. Look at the beging of the 3rd paragraph, it also confirms that this same selection is also used by the NOAA & GISS datasets. So all 3 use the same weather station selection, so if this is compromised then all 3 are compromised in the same way

Lucy Skywalker provides us with a valuable means of checking a large area of the Russian Temperature History.Lucy has raw data from Giss thermometer records for Russia’s Yamal and Polar Urals which shows a pattern of temperature rise to about 1940, falling temperatures to 1970, a temperature rise to 2000 but not higher than 1940, then levelling off after 2000. Here’s a good sized jigsaw piece of the data that CRU lost and a means of comparison with the Russian record.

Perhaps this was the “smoking gun” that Dr Phil Jones wished to keep from SteveM. It also may explain the continuously very high positive temperature anomalies over Siberia. I wonder how much CRU’s Siberian data accounts for the total amount of warming seen in recent years?

Of course, this bit of new info could be nothing but politics from the Russians. Will the Russians post thier findings for all to see?

“Part of the problem with temperatures has to do with the way the Russian data stations are reported. I have no HARD evidence yet, but I think there has been some “manipulation” to leave out Russian data which would ‘even things out’, IF you accept the FICTION of ‘average temperature’…”

I have a tangential question that I have been wondering for a few days now: When CRU selected these Russian stations I thought back to the old CA and WUWT run at South American UHI calculations, and the absurd distances they were willing to go to find other stations to adjust against (I think the record was 1500 km?) My question is, did CRU adjust Russian stations against only the limited subset of Russian stations…. and if so, how did this selectivity in Siberia cascade of adjustments in other stations around the world?

I’m thinking mostly at this point about the WUWT blog about Darwin Station adjustments where they jumped 500km for an adjustment. It wouldn’t take too many degrees of separation, at 500km a pop, to “teleconnect” screw ups in Siberia to screw ups in Australia.

We are particularly interested in the linear striations which appear in Fig. 1. I have previously documented these striations in satellite measurements, and argued that their slope corresponds to the strength of feedback in the climate system. Unfortunately, two research papers containing evidence of this were both rejected for publication based upon poor reviews from a single reviewer – a rather unusual basis for total rejection by any science journal.
STeve: OT

since I have atm a trainee secretary who happens to be Russian, I thought I might be able to help with the translation … but after having her translate the “introduction”, I think I have to give up here, because the translation from Russian to Dutch, and then to English might take away some nuances and also, it might be too technical for her … (and me directing the translation in a certain way or direction doesn’t seem scientific … let’s not adapt to the Team’s methods in any way) … so, sorry … 🙂

I have read a few comments to the tune of “this not being an official Russian Met Office document” or “why has this not been reported before?”

Frankly, I don’t see any reason why “official Russia” would put out any doubt at global warming. They are benefitting from the hype in many ways. E.g. I don’t see them being invited to pick up a part of the tab. They can literally sell off a lot of fried air for hard currency. And their competitors in the Global Game are wasting lots of time and money on a mirage. Where is the advantage for Mr. Putin to stop the Conference ?

It would be nice to see a Russian temperature trend reconstruction based on the full set of data cited. Can the figure 8 and figure 9 of original IEA paper be (re)posted here? The graphs are not showing up in those translation links and they seem to be the ‘punchline’ – a 0.56 + 0.08C discrepancy in warming trends. 0.64C on 12% of world’s land mass = 0.07C delta in land warming due to this bodge?

Chris Horner showed a curve of declining numbers of Russian reporting met stations after the fall of the USSR, by several thousand (Horner’s Politically Incorrect book). Steve actually did the analysis that Horner published; the graph shows the overplot of increasing global temperature in the 1990s versus the declining number of Russian reporting met stations. According to the IEA the ones remaining were urban heat islands.

GISS and NAOO are also ‘told’ which stations to use by the WMO, so all 3 datasets use the same station selection. Any bias introduced by the method of station selection is common to all three datasets. Makes the ‘independent datasets’ statements seem open to challange.

As far as I can tell, this Russian IEA document is only looking at the ~1500 stations (out of ~5000) for which the MetOffice recently released data. These are the ~1500 stations that are not protected by non-disclosure agreements (NDAs). If that’s true, then whatever conclusions they reach about the selection of stations is not a conclusion about the HadCRUT temperature series. It is only a conclusion about the distribution of stations are covered by NDAs.

The Russian IEA document states:

Over 40% of Russian territory was not included in global-temperature calculations for some other reasons, rather than the lack of meteorological stations and observations.</blockquote

I believe the "other reason" is non-disclosure agreements. Hopefully the complete list of Russian stations will be made available soon.

(Apologies if this has already been posted — I couldn't find it searching on obvious terms)

As far as I can tell, this Russian IEA document is only looking at the ~1500 stations (out of ~5000) for which the MetOffice recently released data. These are the ~1500 stations that are not protected by non-disclosure agreements (NDAs).

Have they actually said that — that these are just the stations not protected by NDA’s?

I took a look at what stations were included for my home state here in the US — Arkansas. Just two: Fort Smith, and Little Rock (the airport). The latter only has data back to 1937. Not one of the fifteen USHCN stations in Arkansas are included. And the two that were included are not in USHCN, so are arguably not the best stations that could have been used.

When they ask themselves, in their FAQ, why these stations, they say

The choice of network is designated by the World Meteorological Organization for monitoring global, hemispheric and regional climate and variability.

To compile the list of stations we have released we have taken the WMO list of GCOS Surface Network stations and Baseline Climate Reference Network stations, cross-matched it and released the unambiguous matches.

I think they are whistling past the graveyard hear. There is certainly no intimation that they have only included data not covered by NDA’s. I’ve got a bunch of data, from my state alone, which are not covered by NDA’s, and which they have not used, at least in their released subset.

I think this subset was released only after checking that it would result in a long term trend corroborative of the trend they show from their full data set. But there was no reason to limit it to just these because of NDA’s.

As far as I can tell, this Russian IEA document is only looking at the ~1500 stations (out of ~5000) for which the MetOffice recently released data. These are the ~1500 stations that are not protected by non-disclosure agreements (NDAs).

Have they actually said that — that these are just the stations not protected by NDA’s?

Quite the opposite – as usual, it would have been helpful to actually read the IEA doc before starting to speculate. IEA page 4:

Of the approximately 1500 weather stations for which data were published by the Hadley Centre,
121 stations are located on the territory of Russia. They make up about 8,1% of the total number of
stations for which data are published, and 2,4% of the total number of stations used to calculate the
global temperature. As far as the accompanying note says that the data on the remaining
approximately 3500 stations remain unpublished for the reason of pending permissions from
national authorities, one can deduce from that that when these remaining data are released, there
will be no additional data from Russian weather stations among them.

(emphasis mine)

The IEA says – and I have not yet seen anything to the contrary – that all Russian climate data is in the Public Domain and thus freely available to use and to re-publish by CRU or anyone else.

To overcome the language barrier, the Russian researchers could express their findings as a triplet of computer code, their input datasets intact with meta-data, and their computation of the output datasets (which would serve as a proxy for their conclusions).

Verification would be straightforward.

The input datasets would need their meta-data carefully understood.

Variable names could be translated by dictionary if necessary, or renamed in deductive ground-up fashion from the input data-sets and code structure.

I would need some help in translation because the text is 21 pages long. Russian volunteers are needed. 🙂

However, look at this. Page 21 of the Russian report (my translation):

“Having in mind a negative divergence of temperature series up until mid 1950s (up to 0.56 degrees C) as well as a positive divergence of temperature series in the mid 1990s (0.08 degrees C), increase in the rate of warming on the territory of Russia, created by the collaborators of the HadCRUT in the period 1870-1995 can be estimated at minimum 0.64 degrees C!”

“In the same time, this assessment appears to be too conservative, because we used for creation of the temperature history of Russia all available data provided by the Russian Met office, without any qualitative selection, as well as without any necessary corrections, for example, adjustment for the urban heat island effect”.

I would expect that these are the same datasets which CRU uses. (Plus 10 more from somewhere?)
How “good” is older info on Russia? From the 1927 Smithsonian volume mentioned on RealClimate, I looked up the Siberian stations and from 1881-1920 the monthly means were figured using unknown hours of observation; from my WWR 1941-1950 book, the means were figured by 1/4(01+07+13+19) local time; for current means I assume the formula is 1/2(Tmax+Tmin).

Could some of the missing Russian stations have been excluded from CRU’s analysis because of missing records of true daily max and min values?

I think we need to remember that there were many many datasets, most secondary(not what we consider raw), some overlapping, that went into building GHCN & CRU.

Tim Lambert says:The red curve is the temperature trend using the 121 Russian stations that CRU has released data for, while the blue hockey stick is from a larger set of 476 stations. I’ve put them on top of the CRU temperatures for northern extratropics. The red and blue curves agree very well in the period after 1950, thus confirming the CRU temperatures. Well done, IEA!
[…]Steve McIntyre will no doubt be demanding the IEA’s data and code for their study. No doubt.

Steve: I endorse the idea of someone verifying these results. However, I am only one person and have other commitments and priorities. Surely the climate science “community” is not so threadbare that no one other than me is capable of the analysis.

I really and totally agree with you. Thanks for reply…
Sure Tim Lambert can ask for verifing, comment and destroy the IEA’s code, and release his code as open source programm for us. But I can’t find his code nowhere. It’s a metodology of AGW religion: the skeptics (or assumed), must prove and release their data, while for the RC (for example) just sufficient read the paper… No doubt. 😀

Based on his linked graph and the IEA graph p20, accurate enough from 1960 to the present

But his comment is cherry-picked for the time grab:

Both graphs run from 1870 to 2008

Cru is significantly diverged from 1870 to 1960, with the largest anomaly difference in 1880, with an [abs] 0.5C difference in the anomaly points of the two graphs

The result of this is that CRU can state an anomaly rise of [abs] 1.75C over 130 years, but the IEA graph shows only [abs] 1.25C. This is a 40% discrepancy pre-1960, which is precisely what the IEA paper says

It’s a really a funny metodology: “hide the problem”, “focus the others error”, “insult the opponent”. According to Lambert and RC the problem is only post-1960, but pre-1960 I see a large difference in graph (like IEA). I see, because I haven’t ham slices on the eyes! 🙂

During the cold war, I was involved with gathering/relaying Soviet weather observations. Many were gathered by CW (ditty-dum-dum!) radio intercepts. On one occasion (1966-67?) when the Soviet’s collection system was down, Suitland asked us (USAF) for our Soviet data and then relayed it to Moscow. We got word back that we had sent reports from stations that Moscow didn’t even know existed. I cannot verify that the Soviets really said that, but I see no reason that our Weather Bureau would make up such a story. And, if we were intercepting ‘spoof’ stations, why would Moscow draw attention to it?
There was a rumor that (at least some) stations in Siberia would report padded lower temperatures in the winter to obtain a larger heating fuel allocations from Moscow. Was it true? I have no idea, but it sounds entirely plausible.
During the 70’s or 80’s (rust plates in the ol’ brain prevent accurate time citing), for an extended period, there was a decoding ‘problem’ created by the Synoptic Code used, that in essence could ‘confuse’ very low temperatures, i.e. colder than -50C. Temperatures were reported unsigned, and those reported above ‘50’ were assumed to be negative. But how do you report for instance -103C? Hard-copy records should not have that problem
Does any of this have anything to do with CRU culling some stations? I have no idea.

UK MET office has released a news release today referencing the IEA complaint. Some how they say the HadCRUT and IEA temps are similar and both underestimate the real warming which is now even greater than the HadCRUT/IEA.

‘The study, carried out by ECMWF (the European Centre for Medium-Range Weather Forecasts) with input from the Met Office, performs a new calculation of global temperature rise.’

This conclusion is in contrast to a recently released study by the Institute of Economic Analysis (IEA) think tank based in Moscow. The IEA’s output is consistent with HadCRUT as they both confirm the global warming signal in this region since 1950, which we see in many other variables and has been consistently attributed to human activities

Note the statement “IEA’s output is consistent with HadCRUT….since 1950”.

[…] ClimateAudit is all over the story, and dug up an “interesting” Email from the CRU hack archive worth quoting here: On Mar 31, 2004 [Phil] Jones wrote to [Michael] Mann as follows: “Recently rejected two papers (one for JGR and for GRL) from people saying CRU has it wrong over Siberia. Went to town in both reviews, hopefully successfully. If either appears I will be very surprised, but you never know with GRL.” Say no more. […]

[…] it “haphazard” represents comic understatement (google HARRY READ ME). And according to Steve MacIntyre at Climate Audit, an email from Dr. Phil Jones to Dr. Michael Mann in 2004 from the Climategate set highlights […]

[…] are included in climate change predictions , but the colder data has been ignored – see Climateaudit.org . The Copenhagen summit has been a farce and it looks like their will be only a limited agreement . […]

[…] to the human-caused global warming theory this past week, this time from evidence that indicates Russian climate station data was cherry picked and only data that showed increased temperatures was included in the report that the IPCC used to […]

[…] you can see that it looks like the blue curve and not the red one. Steve McIntyre will no doubt be demanding the IEA's data and code for their study. No doubt." Russian analysis confirms 20th century CRU temperatures : […]

[…] IEA: Hadley Center “probably tampered with Russian climate data” « Climate Audit Climategate emails show that Phil Jones of CRU, acting as a reviewer of the CRU data used in the HadCRU gridded temperature, “went to town” to block the publication of criticisms of his handling of Russian data. […]

[…] IEA: Hadley Center “probably tampered with Russian climate data” « Climate Audit Climategate emails show that Phil Jones of CRU, acting as a reviewer of the CRU data used in the HadCRU gridded temperature, “went to town” to block the publication of criticisms of his handling of Russian data. […]