No, history was not rewritten. What the folks there don't seem to want to acknowledge is that GHCN circulates two files, described here. The file everyone there wants to focus on is the adjusted file (QCA). This, as explained, has been homogenized. This is a preparatory step for its use in compiling a global index. It tries to put all stations on the same basis, and also adjust them, if necessary, to be representative of the region. It is not an attempt to modify the historical record.

That record is contained on the other data file distributed - the unadjusted QCU file. This contains records as they were reported initially. It is generally free of any climatological adjustments. For the last 15 or so years, Met stations have submitted monthly CLIMAT forms. You can inspect these online. Data goes straight from these to the QCU file, and will not change unless the Met organisation submits an amended CLIMAT file. This is the history, and no-one is tampering with it.

The adjusted file does change, as the name suggests it may. Recently, it has been modified to use an improved pairwise comparison homogenization algorithm due to Menne and Williams. It is now (as of Dec 15 2011) used by GISS instead of their own homogenization algorithm, which makes the QCA file much more significant.

Update. I have a new post which looks at the GHCN adjustments more generally, with visualization.

The Iceland Met Office record

So Paul raised this with the Icelandic Met Office, with the following loaded questions:a) Were the Iceland Met Office aware that these adjustments are being made?
No we were not aware of this.b) Has the Met Office been advised of the reasons for them?
No, but we are asking for the reasonsc) Does the Met Office accept that their own temperature data is in error, and that the corrections applied by
GHCN are both valid and of the correct value? If so, why?
The GHCN “corrections” are grossly in error in the case of Reykjavik but not quite as bad for the other stations. But we will have a better look. We do not accept these “corrections”.d) Does the Met Office intend to modify their own temperature records in line with GHCN?
No.

And the Met sent their own version of the Reykjavik data. But what is missing from this dialog is that GHCN was never altering their QCU record, and is not suggesting that the Iceland records should be changed. I'll show the QCU data for the period most complained of, post 1939. Units are .01°C. Unfortunately, the Dec numbers got lost in formatting.

Year

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

1937

80

-160

-130

500

690

960

1160

1040

860

400

360

1938

10

180

200

470

600

920

1130

1060

940

490

170

1939

-150

130

360

500

870

1080

1300

1230

1180

740

150

1940

160

170

-20

300

760

970

1120

1010

710

630

130

1941

-30

-150

210

540

880

1150

1230

1150

1150

710

480

1942

160

190

270

360

770

970

1140

1130

790

260

410

1943

40

-90

150

290

500

990

1140

990

810

450

200

1944

-170

60

140

410

660

990

1300

1180

800

460

10

1945

-260

10

410

440

760

980

1190

1200

960

720

650

1946

240

-20

280

320

840

870

1060

1060

800

760

140

1947

320

-200

-280

170

830

990

1090

1100

770

560

30

1948

70

200

320

120

460

940

1060

1100

620

310

250

1949

-270

0

10

0

360

980

1050

1010

900

450

250

1950

280

-70

130

220

710

930

1240

1220

720

450

70

1951

-150

-90

-300

0

690

980

1060

1140

970

570

90

1952

-290

-30

100

340

620

830

1010

1010

760

570

260

1953

60

180

270

10

680

1020

1190

1150

970

420

200

1954

200

10

160

450

760

1020

1060

1100

610

370

270

1955

-200

-270

140

560

570

990

1050

1010

810

390

430

1956

-320

270

360

360

660

840

1070

990

930

470

500

1957

110

-150

40

500

700

990

1200

1130

710

470

300

Despite all the invocation of the Iceland data, there is no electronically readable form supplied. You can compare the link above, though, and you will see that the GHCN QCU data is almost identical. There are changes of a fraction of a degree - the Iceland Met do say that they have also made some adjustments.

Actually there are isolated differences. These seem to be sign differences, where GHCN has a negative sign. The sign may have disappeared in scanning the Iceland numbers.

The story of the Reykjavik adjustments was taken up by Ole Humlun, with again the same disdain for the record contained in the GHCN QCU file and the purpose of the homogenization adjustment. He refers to the Iceland data, and to Rimfrost, but does not mention the almost identical QCU.

The V3.1 adjustments

It is true that the adjustments have changed recently. V3.1 was released 4 November 2011. I have a QCA file from 14 July 2011, and this is, for Reykjavik, almost identical to the QCU file. I have a v2.mean file from Dec 2009, which is the V2 unadjusted file, and it is essentially identical to the current QCU file. In v2 style, it had duplicates. there were four for Reykjavik, but they had little overlap and where they did, were consistent. The unadjusted file has not changed, but the QCA file has.

I also have an adjusted v2 file from Dec 2009. The adjustments are substantial, but less than the current ones.

So here is the plot of the current QCU file vs the adjusted QCA file:

And here is the plot of differences

As you see, the adjustments are substantial. I have done a repeat of the GHCN analysis that we did for Darwin, with a histogram of the effect on trends. I will blog about this shortly. Current estimate is that this adjustment is in the top 10%.

17 comments:

Nick, the reason people focus on the adjusted GHCN data, as you know, is that it's the adjusted data that is used by GISS and HADCRUT.

On the GHCN ftp site you can get v3.0 and v3.1 from November, and see that v3.0-adjusted is, as you say, almost the same as the unadjusted.

Can you make the v2 file you have available? Or at least plot the v2 adjusted for Reykjavik?

The interesting question is which of the GHCN 'quality control' procedures (ha ha) are kicking in to generate these weird adjustments. Is the algorithm finding 'breakpoints' at 1940 and 1965 and 'correcting' them? Can you shed any light on this?

Paul,I don't think Hadcrut has used the GHCN adjusted data in the past. Maybe they will.

I have uploaded the zipped v2 Dec 2009 adjusted file here. I didn't include the unadjusted file, or v2 inventory, because they don't change. But I can if you need them.

Homogenization doesn't interact with quality control, which is mainly about finding clerical errors. As you say, the importance of the file is that it is used by GISS. But the test to apply for that is statistical - do the adjustments create a bias? And if so, is that justified?

But I agree that here it is probably interpreting the sudden changes in 1940 and 1965 as extraneous. As I said at WUWT, it would be a big mistake to step in and overrule the algorithm piecemeal. The only option is to make a general rule modification and apply it everywhere. For example, make it reluctant to spot a break if it is shared by neighbors. I suspect they do that already.

I don't think the GHCN adjustment code is available for download. The GISS code is, and has been updated recently; I suspect it will show whether GISS has scrapped its own process in favor of GHCN. I'm checking.

I think an aspect that has bothered Ole Humlun, and maybe others, is the GISS interface for station data. It used to have unadjusted as first option, then the "suspicious values removed" as the saecond and default option, and then the adjusted. Now the first option is adjusted (they don't deal with the unadjusted any more), and the other menu options don't make much sense (they would be the same). I think that just hasn't caught up, but probably GISS should scrap the facility, and leave it to NOAA. They do it well now, and are the custodians of the data.

The GHCN 3.0.0 code (Menne & Williams 2009) was available from the NCDC FTP site; that's where Dan Rothenberg got it to do the ccf-homogenization work last summer. The 3.1.0 code is not available yet. My understanding is that researchers need to get some sort of permission to post code on the internet. Bureaucracies can be like that - possibly a hangover from NCDC's military past? Who knows. Anyway, I believe that the code will soon appear.

Indeed, I put that blog post up myself. Note that the code which Dan started with was apparently not exactly the same as the code which was being used at the time (last summer) for GHCN 3.0. Matt & Claude had already made some improvements, which then combined with fixes for bugs identified by Dan, for the 3.1.0 code (the process is described in their tech note). See here.

I find this wailing and gnashing of teeth somwewhat counter-productive. We need to start from a realization that none of these measures were made in a metrologically traceable manner to SI standards, that change has been ubiquitous even at the best managed sites, and so adjustments are required. In the absence of complete metadata as to changes at each station there will be inevitable uncertainty in how to do the adjustments. For climate trends this is the elephant in the room - because (lack of) adjustments yield red-noise changes which have increasing influence with trend length. Key is to get the adjustments right. There are ways that we can do this better than we have in the past see e.g. this recent effort to benchmark the performance of the algorithm in question over the CONUS sub-domain. But what we really need is more efforts to look at the adjustments problem which is the problem that matters from a long-term change perspective. Then we need a consistent set of benchmarks. More on this at BAMS article.

Peter is right, of course, and I totally support his continuing efforts to develop homogenization testbeds/platforms/systems, which will allow us to improve this whole area of the science. I'm hoping to spend some time thinking about this in February (and revisiting Dan's code, if I have the time).

I'm happy to be reminded of this excellent piece of work, which showed very clearly for GHCN v2 that there is a fairly symmetric (and fairly narrow) spread of trend effects due to homogenization, with very nearly zero net effect on global trends. I'd be glad if you would retread that for GHCN 3.1.

Any homogenization process is bound to have trend effect outliers: stations for which the homogenization has a comparatively large effect on the long-term trend. Because the data is so noisy, it is inevitable that some of the adjustments producing those outliers are 'incorrect': that is, the station record at a change point reflected ground truth and not an inhomogeneity. Those are the eggs we need to break to make the climate omelette. Overall studies of homogenization processes, like the excellent recent Williams/Menne/Thorne paper in JGR-A, show very clearly that they are valuable in recovering climate signal from inhomogeneity noise.

Nick (S), thank you very much for posting up the v2 file from Dec 2009.

Looking at the data for Reykjavik shows that the v2 adjustments cool by about 0.9 in the 60s and 70s, increasing to 1.3 before 1940 and 1.7 before 1920. So the adjustment-fabricated warming in v2 is similar in magnitude to that shown in your graph for v3.1, though more monotonic. It is a similar picture for Stykkisholmur, with downward adjustments of around a degree before 1970.

So to recap, we have fairly monotonic past-cooling adjustments in v2, no adjustment in v3.0, and then very erratic adjustments in v3.1.

Incidentally I also found some sign errors in the v2 adjustments for Stykkisholmur.The unadjusted file for the first few months of 1973 and 1974 are 1973 22 -34 4 211974 -1 -11 26 54and the adjusted file is1973 -22 -34 -4 21 1974 -1 -11 -26 54

Regarding breakpoints, it would, as you say, be sensible if the algorithm checked nearby sites to determine whether an apparent break was genuine or spurious. It's pretty clear that the GHCN algorithm doesn't do this, since the genuine cooling in 1965 is erroneously adjusted away by the GHCN algorithm in 7 of the 8 icelandic stations.

Regarding the GISS adjustments, the current situation is absurd. GISS starts from the GHCN adjusted data, which includes (erroneous) adjustments for breaks and homogeneity adjustments. It then applies its own algorithm to remove 'suspicious records', and then applies its own homogeneity adjustment!It's amusing to see that the GISS homogeneity adjustment puts back some of the 1940s warmth that was deleted by GHCN (0.7 degrees in 1940).

Nick,I'm planning a post which will show the V3.1 histogram, coupled with a Google Maps visualization so you can see which stations have large positive or negative adjustments. It should show up things like this Iceland pattern.

Nick B, perhaps you haven't been following this story. I recommend you go and look through the posts at Paul Homewood's blog. In all 8 of the iceland stations unadjusted data, there is a sharp drop around 1965, and this is well established from a number of sources. In 7 of the 8 cases the GHCN adjustment algorithm incorrectly puts in a break here and gets rid of this sharp drop. Similarly, the raw data consistently shows a warm period around 1930-1940 which is also established in the literature on air temps and SST temps (papers by Hanna et al). In most cases the GHCN algorithm adjusts these downwards. So whatever is flagging the GHCN adjustments, it's not nearby neighbours. If the algorithm looked at neighbours in any sensible way it would know that the raw data was valid.

Here's what the Iceland Met Office says:

' The GHCN "corrections" are grossly in error in the case of Reykjavik but not quite as bad for the other stations. But we will have a better look. We do not accept these "corrections". '

Of course if they would publish the code, the error could be found fairly easily.

As for GISTEMP, until recently it was using unadjusted GHCN data. I gather that in December it was changed to use the adjusted GHCN data instead of doing its own homogeneity adjustments. That's what it says here anyway. The code is here if you want to check.

NickB, look at the dropdown options and download the data for Reykjavik for option 1 and option 3. You can see that GISS is still dong its own homogeneity adjustment after the GHCN one (in fact somewhat correcting the GHCN, as I said above).

Mailman,"when ever adjustments are made the effect is to always cool the past "That's just not true. In the Darwin post that I linked, I showed the histogram of effects of V2 adjustments on trend. It's balanced; the trend is almost as likely to be lowered as raised.

That post was a response to a WUWT post on adjustments to Darwin, where adjustments raised the trend. I showed that adjustments to Coonabarabran, also in Australia, lowered the trend there equally.