The End of CRUTEM?

The UK Met Office has updated their CRUTEM webpage, providing a list of countries that have thus far responded with release permissions. CRU is now pretty much redundant in CRUTEM, with the Met Office having stepped in to do the things that CRU should have done long ago. This is a transfer of responsibility that has been called for at Climate Audit for a long time.

In my interview with Nature reporter Olive Heffernan last summer, I recommended that CRUTEM be transferred to Hadley Center. She didn’t report that it in her story, but I added the comment at the Nature blog and CA at the time (Aug 12, 2009) as follows:

Ross McKitrick has long observed that important indices like the Consumer Price Index are collected by national statistical services and not by professors in their spare time. Such statistical services have a more formal approach to data management and data audit trails and would be far less likely to simply lose agreements and fail to save data, both of which CRU has done. I identified Hadley Center as an alternative to CRU.

I have often used the analogy of national Consumer Price Indexes to illustrate the ridiculous situation of the “Global Temperature” data. Each country has large professional staffs at their Stat agencies working on the monthly CPI using international protocols, using transparent methods, with independent academics looking over their shoulders weighing the various aggregation methodologies (e.g. Paasche, Laspeyres, Fisher, Tornqvist etc index number formulae), and with historical archiving rules that allow backward revisions periodically if needed. It’s by no means perfect, but it’s a far cry from the f**king gong show we’re seeing here. The reason CPI data, GDP data, etc. are handled professionally is that a range of policies (such as money supply control, pension indexing, intergovernmental transfers) etc depend on the numbers; also some labour contracts include CPI-based escalator clauses. In other words the numbers matter.
.
By contrast the Global Temperature numbers are coming from a bunch of disorganized academics chipping away at it periodically in their spare time. GISS numbers are handled (on Gavin’s admission) by a single half-time staffer, and the CRU says they’re stumped trying to find their original files back into the 70s and 80s, as well as the agreements under which they obtained the data and which to this day they invoke to prevent independent scrutiny.
.
I take Steve’s point that these issues are not sufficient to change one’s views on any particular policy which could be shown on other grounds to be desirable or otherwise. And this is not the thread (or the site) to debate policy. But bear in mind that the G8 Summit recently issued an undertaking to prevent the Global Temperature from going up by 2C compared to pre-industrial times, and the Copenhagen Agreement will probably contain similar language. That ‘2C’ number refers, in all likelihood, to a few digits that Phil Jones publishes in his spare time using data from files he has lost, etc. etc. If that kind of quality control is good enough for the purposes of G8 agreements and international protocols, then let’s not waste so much money running national statistical agencies to produce numbers that are used for such trivialities as central bank policy, national pensions and labour contracts. We’ll just ask some professors to cobble together a monthly number in their spare time, we’ll let them do historical revisions any time they want without notice or external scrutiny, and we’ll promise them that nobody will ever check their work.
.
And if the CRU staff want to plead that we are being too harsh on them because they “did the best they could” with such limited means and minuscule staffs, etc, fine. I’ll be the first to join in such a defence: the minute they put a big red disclaimer on their data products saying “Warning: These numbers are offered on an as-is basis and should not be used for any important public purposes.” But they can’t have it both ways, claiming their data products are good enough for policy purposes, while also claiming they cannot be expected to meet the quality expectations that apply to the other data series we use for policy purposes.

This recommendation was not new last summer. I remember Ross making the suggestion around the time that we first met (in 2003). It’s mentioned from time to time in various CA comments and posts – a few examples follow merely to show continuity.

Maybe responsibility for temperature collection should be taken away from CRU. Collecting and collating temperature data is like making a Consumer Price Index. Shouldn’t this be done by a professional statistical service and not by some guys acting like primadonnas?

Or in an April 2007 post here right after getting Chinese data from CRU from a (rare) successful FOI request. This post discussed what E.M. Smith (Chiefio) aptly called the “Great ‘Dying’ of Thermometers”:

If the Team is too busy to update the Chinese station data, dare one ask what the CRU contract with the U.S. Department of Energy contract actually requires them to do? 95% of the Chinese station data was collected by the U.S. itself ( see the TR055 program) nearly 20 years ago. What has the CRU been doing about Chinese data since then? Were they so convinced by Jones et al 1990 about Chinese UHI that they felt entitled to stop collecting information other than one big Chinese city for most gridcells? The Team never ceases to amaze.

There’s an obvious moral to this story. Ross McKitrick observed a long time ago that the Consumer Price Index is never be calculated by a bunch of academics at a university. It’s calculated by a proper statistical service who don’t mind doing little things like actually updating 95% of the Chinese data more frequently than once every 20 years. So aside from being surly and unresponsive with horrendous documentation of what they did, CRU doesn’t seem to be very efficient either. I wonder what justification the U.S. Department of Energy has for giving this inefficient program another cent.

Or again in a couple of 2008 posts herehere noting (in this case GISS’) inability to locate Canadian station data:

How hard can it be to locate Canadian data? Maybe it’s time to ask the people who do the Consumer Price Index to compile temperature statistics. It’s all just data – maybe professional data people would do a better job than the present people who seem to have trouble getting off their La-Z-Boys.

Or here once again in 2008 after Gavin Schmidt had said that the entire GISS quality control effort took only about 1/4 of a man-year annually:

Ross McKitrick has long observed that the collection of temperature data is a job sort of like making a Consumer Price Index and it should be done by professionals of the same sort. It doesn’t make any sense for people like James Hansen and Phil Jones to be trying to do this on a part-time basis. As long as it’s being done on such a haphazard basis, there’s really no way to prevent incidents like this one (or last year’s “Y2K” problem.)

The new Met Office webpage and approach is less adversarial and more professional. It’s hard to see what’s going to be left for CRU to do. Or that anyone would want them to be involved anymore.

This raises an issue not on the Muir Russell list of issues, but one which now needs to be thought about. On the assumption that the Met Office takes over the calculation of the land temperature index (“CRUTEM”), what exactly is the role and mission of CRU? They aren’t players in the GCM business or in the physics of climate. They do some tree ring studies (Briffa, Osborn, Melvin), but I doubt that their tree ring stuff is sufficient to motivate a separate institution.

On the other hand, CRU, their alumni (Wigley, Santer) and associates (Mann, Schmidt) have been disproportionate contributors to the most poisoned part of the climate debate, with Jones contributing to and encouraging venal conduct.

Maybe the most appropriate way to recognize Jones’ contribution to climate studies and to send a firm message to the climate community – that the issues are far too important to indulge such conduct – would be to disband CRU, acknowledging the loss of the CRUTEM franchise and dispersing whatever staff are left.

then it cannot be relied on. In the last few years the NSO was also being used for spin by the UK Govt but now has been put in a more arms length postion and now regularly releases data which before would have been surpressed.

I have an FOIA into CRU for information relating to the dtermination of the following question: Is the confidential data NECESSARY? the guidlines for contracts for CRU specify that in order to enter into contracts that transfer confidential data to CRU the data must be Neccessary or essential to CRU’s mission.

My first request was denied and CRU argued that no study of this was done and that since papers relied on CRU it was somehow necessary.

There is an appeal pending.

Question: if a temperature index without confidential data shows the same result as a temperature index with confidential data is that confidential data necessary?

O/T: (Steve: there are recent threads on what you’re discussing. It’s extremely impolite not to spend the 15 seconds to place your comment on an appropriate thread,

There is something very odd indeed about the statement by the Information Commission on its investigation into “Climategate”, the leak of emails from East Anglia’s Climatic Research Unit. Gordon Smith, the deputy commissioner, confirms that the university’s refusal to answer legitimate inquiries made in 2007 and 2008 was an offence under S.77 of the Information Act. But he goes on to claim that the Commission is powerless to bring charges, thanks to a loophole in the law – “because the legislation requires action within six months of the offence taking place”.

Careful examination of the Act, however, shows that it says nothing whatever about a time limit. The Commission appears to be trying to confuse this with a provision of the Magistrates Act, that charges for an offence cannot be brought more than six months after it has been drawn to the authorities’ attention – not after it was committed. In this case, the Commission only became aware of the offence two months ago when the emails were leaked – showing that the small group of British and American scientists at the top of the IPCC were discussing with each other and with the university ways to break the law, not least by destroying evidence, an offence in itself.

Also probably OT but referring to Michael Larkin legal analysis: if that reading of the law is correct, it would enable legal action now and during the next four months or so, on the facts originally brought to the attention of the authorities about 2 months ago. New facts emerging afterwards have yet more time (up to 6 months) to press charges.

Weird that for this they rely on an “amateur” dataset, and never bothered to contact the DWD who had it “all along”.
Also GISS never bothered to do a correction on the De Bilt Dataset with data publically downloadable at KNMI.

interesting thought, but not really a viable option. CRU is (apparently) 3 full-time, and two part-time funded posts at the university of east anglia, who bring in the funding for a total of ~30 staff. UEA wouldn’t disband CRU, since it would give them enormous problems with their ~26 staff to be supervised (and associated incoming money for the University), and it fundamentally doesn’t make sense; CRU brings in vast amounts of money (and research prestige) to the University. Bringing in that amount of money (as against university expenditure of ~4 FT staff, ~£300k) makes a grouping very valuable- and influential- in a university.

More tractable is that CRU should be relieved of their ability to make the land-surface record, as a contribution to HADCRUT. This might be quite a problem, insofar as it might be a bit tricky to run it as a hostile takeover. There are the IPR issues in the CRU temperature record- which will belong to CRU, and they will be profoundly reluctant to give up the IPR. The IPR is their key to getting future funding for updates to the CRU temperature record.

Looking at the “Harry” problems, it might be quite difficult to reconstruct the CRU contribution method anyway 🙂
per

The point is well taken that computing a good temperature index has a lot in common with computing a good price index.

The administrative issue of who computes the index is of course very important, but just computationally, the “first difference” method of computing a temperature index has a lot to be said for it. I don’t think that this is actually used for any of the major temperature indices, but some of the NOAA establishment like it, and Craig Loehle has a recent (1/13) paper showing that it has many good properties.

I am submitting a paper in a few days showing that first differencing can be used for creating superior homogeneity adjustments. I can send it to anyone who emails me but it is far too early to post or discuss.

One the Met Office page Steve links to there is a FAQ that includes as its first question and answer:

“1. Is the data that you are providing the “value-added” or the “underlying” data?

The data that we are providing is the database used to produce the global temperature series. Some of these data are the original underlying observations and some are observations adjusted to account for non climatic influences, for example changes in observations methods or site location.

The database consists of the “value added” product that has been quality controlled and adjusted to account for identified non-climatic influences. It is the station subset of this value-added product that we have released. Adjustments were only applied to a subset of the stations so in many cases the data provided are the underlying data minus any obviously erroneous values removed by quality control. The Met Office do not hold information as to adjustments that were applied and so cannot advise as to which stations are underlying data only and which contain adjustments.”

The data isn’t very useful then is it? The hope is that they will clean this up at some stage.

I have a concern going forward with the validity of data from countries that are due to cash in from money transfers. One of the images that really sticks in my mind from Copenhagen is the representatives of the African countries furious about not getting the billions of dollars they were being promised. What is to stop these countries from sticking their finger on the scale with future temperature readings? I have to admit if I was the leader of a very poor country and I had a way to help ensure that the money would start flowing in, I would consider it as well. Any thoughts for keeping the countries honest?

With Vicky Pope and Julia Slingo, and the appalling recent record of seasonal forecasts, I still see a Met Office where facts can be bent to advocacy. But the new web page does look nice. And if the BBC fires the Met Office as is hinted, there will be spare hands available.

Me too Lucy, I recently asked for an explanation as to where the budget came from for the round robin demanding support for CAGW, the response is below:

“Could you please provide me under the FOIA the following:

1. Any documentation related to the approval for the spend of public
funds in the pursuit of signatures to support the Met Office;

2. The budget for the pursuit of the signatures and the actual spend
made by the Met Office in terms of wo/man hours and any other expenses made in this exercise.

If you have received many requests in the same vein, please don’t waste time on mine just point me to where the information is available.

Thank you very much for your attention.”

The response:

“Re: Freedom of Information Request
Your email dated 11 December 2009 has been considered to be a request for information in accordance with the Freedom of Information Act 2000.
You asked the following questions:
1. Any documentation related to the approval for the spend of public funds in the pursuit of signatures to support the Met Office;
No documentation was produced relating to the approval for the spend of public funds in the pursuit of signatures.
It should be noted that the signatures referred to above were collected on behalf of the UK Science Community to show support for the statement below, not in support of the Met Office.
We, members of the UK science community, have the utmost confidence in the observational evidence for global warming and the scientific basis for concluding that it is due primarily to human activities. The evidence and the science are deep and extensive. They come from decades of painstaking and meticulous research, by many thousands of scientists across the world who adhere to the highest levels of professional integrity. That research has been subject to peer review and publication, providing traceability of the evidence and support for the scientific method.
The science of climate change draws on fundamental research from an increasing number of disciplines, many of which are represented here. As professional scientists, from students to senior professors, we uphold the findings of the IPCC Fourth Assessment Report, which concludes that ‘Warming of the climate system is unequivocal’ and that ‘Most of the observed increase in global average temperatures since the mid-20th century is very likely due to the observed increase in anthropogenic greenhouse gas concentrations’.
2. The budget for the pursuit of the signatures and the actual spend made by the Met Office in terms of wo/man hours and any other expenses made in this exercise.
It is estimated that over the four to five days taken to collect signatures and produce the statement that a total of around 30-35 hours of staff time – spread across several staff – were used.
There was no budget set for this activity. There was no overtime paid, the time committed was done so alongside the normal roles of the staff involved. There were no other expenses occurred in this exercise.
I hope this answers your enquiry.”

As you can see I had made no accusations against the “scientific community” nor am I aware of anyone else who has. Some people have assumed that the UEA/CRU were up to no good, but I wasn’t one of them. The Met Office needs to be cut back to weather forecasting and privatised, it is simply a government funded environmental advocacy agency.

It is important to distinguish between the maintenance of raw data and the analysis of the data. One independent or trusted entity could perform both tasks but it wouldn’t be necessary. In the interest of transparency it could be more practical to have one entity collect, maintain, and distribute (or make available) the data. Computer/database, etc. experts, not professors working part-time.

‘Official’ data analysis performed along the lines of the CPI, not part-timers, but anyone could supply ‘their own’ experts to analyze the data supplied. Both ‘official’ entities or ‘sides of the house’ would have to collaborate in so far as verifying data inputs, examining current data storage processes, and vetting the current, overlapping, data collections.

Specifically with repect to CRU:
1. Hindsight says it is a defunct, ‘wizards of oz’ (Wigley,Jones) castle. Briffa, Osborn, Melvin can dissolve into UEA departments taking some staff with them. UEA is a medium sized U and can absorb some admin and computer staff. Units and departments get folded up all the time. Take off the ruby slippers, time to go home. It would be UEA’s institutional decision.

2. A conceivable possibility for UEA would be to expand CRU’s mission while still relinquishing CRUTEM responsibilities. Bring in GCM experts, phyics experts, for example. Who would want to go there is beyond me but the Unit could be kept alive by house cleaning followed by a new mission statement.

GISS:

GISS seems to be in the same boat as CRU wrt Temperature data and analysis. It’s time to outsource to professionals in that area. If NASA and LLNL want to keep Schmidt, et al, and Santer as modelers then, of course, it’s their decision. I surmise Schmidt as a modeler could be more productive than “Schmidt the Blogger”.

Oh dear, we are being devious again. The permission of Australia for the use of data is stated as “Based on the original temperature observations sourced from records held by the Australian Bureau of Meteorology”. This does not concede that permission was given.

The copyright provisions that accompany the release of data from the BOM include inter alia

“Permission for downloading, repackage (sic) any registered Bureau user services requires separate formal negotiations as set out in the Access Regime.” Last time I asked, there was no agreement with UEA, CRU or any Parties like Hadley further down the line.

“. not to modify information found in Bureau material without prior written permission of the Director of Meteorology; ”

There are versions of the Copyright notice and I cannot know if this clause applies to Hadley. If it does, the term used above “Based on the original temperature records …” implies that information has been modified. Hard to check when I have an email from Prof P D Jones saying thay lost some relevant Australian data.

There appear to be a number of differences between the Hadcrut3 data presented at http://www.metoffice.gov.uk/climatechange/science/monitoring/data-graphic.GIF and the Hadcrut3 data presented in the middle figure of Figure 10 of “Uncertainty estimates in regional and global observed temperature changes: a new dataset from 1850 P. Brohan, J. J. Kennedy, I. Harris, S. F. B. Tett & P. D. Jones 2005”

I share john’s concern about the Met Office being run by and ex-WWF head. The new Statistics Authority might be a better place. In the US, the GISS and NCDC data should be maintained by the National Bureau of Standards or one of the similar data-storing agencies within the commerce department. It might be a turf problem for NASA to give up their responsibility for GISS, but NOAA is in the Commerce Department, so that should be less of a problem.

There are a lot of good ideas exchanged on these blogs, but to me it seems we are all preaching to the choir. Are you out there sharing these thoughts with your elected representatives? If their volume of mail was proportional to the internet traffic on these climategate issues, I think they would take notice.

I agree with the suggestion to separate the data collection and analysis functions. If all the raw data were collected and stored as such, anybody doing analysis would be forced to document all modifications to the raw historical data that they used in their analysis (e.g., removal of outliers, corrections for UHI and site changes, etc.).

Such a division would ensure that those reviewing the analysis could review all steps of the analysis, rather than having to start with ‘quality controlled’ or ‘value added’ data that is of unknown provenance.

consumer price index – something to keep in mind is that there is an alternate measure of inflation (the GDP price deflator) calculated by a completely separate agency (in the US at least) from a different data universe. It helps keep the cpi honest.

While I am fully supportive of the idea that such an important statistics series should be maintained by professional statisticians in an appropriate Government agency, such an arrangement will not mean that all problems cease.

In New Zealand NIWA, a Government agency dedicated to research, collects and maintains the weather readings, yet, as you no doubt remember, our temperatures are showing just the same “adjustments” towards warming as the rest of the world. The only advantage is at least all our data (raw and “corrected”) are available for download online.

I saw an interesting event on the BBC News this week which made me think more about this issue. There was a fair bit of press interest this week in the UK’s Office of National Statistics publishing the latest economic output data. When the figures came out they were rather disappointing. However what caught my attention was a question from one of the journalists to the official providing the latest data. She asked if he had any forecast figures for the future.

The official seemed somewhat pained by this question and explained that they only dealt with what had happened and left modelling/forecasting to others.

It seems to me, at present, all of the teams that produce global instrumental records are also in the business of running climate models. And, in fact, probably looking after the temperature data gets a tiny percentage of the resources allocated to computer models.

One way, to improve the integrity of the entire process, would be to require a separation. Groups may collate the data or they may do climate modelling – but not both under the same roof.

Modellers can always ‘fit’ the data. They just need to include sufficient arbitrary adjustable parameters into their models. Though must non-technical people might struggle to understand that often, the better the model fits, the worse it’s predictive capability.

But there is a danger of the data being ‘fitted’ to the model. And that is a much more insidious problem.

I could wrong but those types of announcements are usually for announcing that the latest data is out. Others will then use it model-wise or forecast-wise. Of course, they already knew but it lets everyone know.

It is interesting that the Met was able to get so many countries to respond quickly.

I wonder how many have responded to the requests for release that UEA said they sent out last August.

What I truly wonder about is whether UEA actually sent out requests, or were just stalling. Oh well, that’s what FOI requests are good for — to find out if what a bureaucrat says in public matches what he is actually doing.

I’ve been wondering whether CRU actually sent out any requests as well. Given that there was a negligible inventory of consents when the Met Office took over the file and that there is a decent sized inventory now, it’s not an unreasonable surmise that CRU didn’t do anything.

The round robin signed by the “UK Science Community” looks impressive at first glance. Until one looks at just how many scientists did not sign it. I looked through the list to see which of my friends and acquaintances, be they lecturers, professors or post-docs, had signed and could not see any of their names. Then I noticed that several universities were missing. City, Herriott Watt and Buckingham are more highly respected than a number of the former polytechnics whose acadmics and students signed the round robin.
The arrogance of those who purported to represent the UK Science Community and who must have known that the vast majority of that community were not represented is typical of the proponents of “global warming” or should we now call it “climate change”.

This essay by McIntyre on the value of more professional collection and transparent custody and sharing of temperature data makes a great deal of sense. (Ironically, even such thoughtful, socially constructive views might not be acceptable for publishing in those scientific journals which have a vested interest in protecting shoddy management of temperature data.)

I think the public here does not know what it is like to be an university scienist
Universities are chronically understaffed and proffessors become shizophrenic in dividing there time between teaching, administration and research.

At the time the CRU dataset was collected a lot of the issues here were probably a minor concern. Note also that the CRU dataset is only one of the datasets showing a global temperature increase. Its importance is not so big. I could think that, due to the fact that it tries to be spatially explicit, is not even a very good dataset to describe global temperature increase.

Anyway it is one of the demonstrations of global change and there are many others…

Canadian temperature data – it appears that one can now download ALL the Canadian temperature and precipitation data in a single file:

“Canadian Daily Climate Data (CDCD): For those with a high speed Internet connection, the 2006/7 CDCD containing daily temperature, precipitation and snow-on-the-ground data is available for download. Data is available for the complete period of record for each location up to 2007. The file contains software that provides access to the data.

Please note that the CDCD download file is 212 MB and will require 880 MB of disk space when uncompressed. Download times will vary according to each user’s system and internet traffic.”

about 1/2 way down is the link to all the data for all stations in Canada from their inception up to 2006/7.

I have downloaded a copy and it appears indeed to be complete.

Note that the program runs in the DOS environment and that you must run it from a root directory – I run it in a DOS box on my MacBook Pro running Windows 2000 under VMWare Fusion and found that the program name and switch are in the format:

X: cdcd – dX where ‘X’ is the drive root that you are in. (C:, D:, E:, etc.).