NOAA tampers with US temperatures by actually altering the monthly data. This is because the US data set is robust and does not show any warming over the last century.

But with global temperatures they play a different game. Instead of relying on altering the data, they alter the station selection over time. By eliminating rural stations in the data set, NOAA and NASA increase the amount of warming.

You can get any shaped function you want out of the nearly useless “global” temperature data set. NOAA/NASA/CRU use a carefully cherry-picked common set of NOAA stations as their base, and then declare their data sets to be independent. But they keep changing the set of stations being used, which causes the changes over time seen in the animation above.

Transitioning to more heavily UHI infected data, gives them to freedom to create a hockey stick – while actually adjusting current global warming downwards.

28 Responses to More Than One Way To Skin A Data Tampering Cat

The BEST web page says:
“The Berkeley Earth Surface Temperature Study has created a preliminary merged data set by combining 1.6 billion temperature reports from 16 preexisting data archives. Whenever possible, we have used raw data rather than previously homogenized or edited data. After eliminating duplicate records, the current archive contains over 39,000 unique stations. This is roughly five times the 7,280 stations found in the Global Historical Climatology Network Monthly data set (GHCN-M) that has served as the focus of many climate studies. The GHCN-M is limited by strict requirements for record length, completeness, and the need for nearly complete reference intervals used to define baselines. We have developed new algorithms that reduce the need to impose these requirements (see methodology), and as such we have intentionally created a more expansive data set.”http://berkeleyearth.org/about-data-set/

And yet they somehow come up with a very similar temperature trend to the GHCN derived trends. Looks like they are using HADsst for their ocean data and maybe that dominates the trend? They don’t provide any graphs of the land only data that I could find on their web site. They do provide a land only data set for download, but I haven’t looked at it yet.http://berkeleyearth.org/data/

Frank Lanser looked at the BEST temperature set and found they did the same type of cherry picking and adjustments as the other data sets. Also they do not actually use RAW data much of the time. The goal of BEST was to have ‘independent scientists’ confirm the government data sets to squash the rumblings from the Climategate mess and nothing more.
……………….

First the politics.
Muller went out with the BEST data set before publishing and proclaimed he was a skeptic who looked at the data and became convinced of Global Warming. He was LYING!The Truth about Richard Muller

We publicize and illuminate the latest climate research in plain language, making the science more accessible to the public and policy makers.

Examples include our primer on climate change and our feature on extreme weather and its connections to climate change. We’ve also released a report on heat waves and climate change.
← We Assist Journalists
We Support Scientists →

…. I found that Non-coastal temperatures (blue graph) were much more cold trended from around 1930 than the Coastal trends (red).
But Non-coastal stations can be divided further into Ocean Air Affected stations (“OAA”, marked yellow) and then Ocean Air Shelter stations (“OAS”, marked blue).
OAS areas thus have some similarities with valleys in general, but as illustrated above, the OAS areas cover a slightly different area than the valleys.
In general I have aimed to find average OAA temperature trends and average OAS temperature trends for the areas analysed. For each country analysed I have made comparison between national temperature trends as published by the “BEST” project and then the OAA and OAS temperature trends from original data. I want to know if BEST data use both the warm trended OAA data and the more cold trended OAS data. In addition, I have made comparisons of ECA&D data versus original for many countries and also HISTALP data versus original.

3.3 Adjustments: The BEST project
The BEST project collects data from different sources often already related to NOAA and NCDC. BEST often present multiple versions/copies of the same longer datasets already used repeatedly in climate science. BEST have not required [acquired] the large bulk of existing temperature data from the national Meteorological institutes….

For all countries analysed so far, the BEST national data is nearly identical with the coastal trends and the Ocean Air Affected (“OAA”) locations. The data from the Ocean Air Shelter (“OAS”) stations appears to be completely ignored by the BEST project country after country after country. Just as we saw for HISTALP….

Also for Austria BEST closely follow the OAA area station temperature trends; it’s impossible to see that the majority of Austrian stations – the OAS valley stations – have had any impact on the national result from BEST.

Fig 10 Danish temperature stations used in the “Original Temperatures” analysis.
Red arrows: The BEST project only use longer data series from coastal stations.
In fact, DMI (the Danish meteorological institute) will not share any other long temperature sets with even the Danish population, and DMI claimed not to have the older data we asked for on digital format. I had to dig data up myself. (So now i hold tonnes of Danish climate data in digital format that DMI dont have?)

Fig 12
For the Hungarian Valley (one of the largest OAS area in Europe), the BEST team has used an OAS temperature station “Pecs”. Above, the Pecs temperature trend is shown together with other Hungarian stations. These original data do seem rather homogenous?

None the less, the BEST team adds around 0.7 K of warming to the Pecs data. BEST use a so called “Regional Expectation” for all countries i have analysed, and change original data so they approach these expectations. Best also claim that Hungary as a nation has experienced this warming trend.

More examples of how data from OAS stations has been avoided by BEST, see for example from fig 22 and onwards for German OAS stations:

The raw data for Elgin had little trend, but it was adjusted to match the “Regional Expectation”, wherein lies the problem I suspect. The result is a big upward trend just like the region. The regional expectation could easily be UHI biased and adjusting relatively rural sites to match this bias is a glaring mistake.

Recent recovery of truly raw data from Europe demonstrated that BEST is not using raw data. Further, the results demonstrated that BEST has a bias for coastal stations, just like GISS. Mosher equivocated when asked why OAS stations were dropped from BEST.

… Non-coastal stations can be divided further into Ocean Air Affected stations (“OAA”, marked yellow) and then Ocean Air Shelter stations (“OAS”, marked blue).
OAS areas thus have some similarities with valleys in general, but as illustrated above, the OAS areas cover a slightly different area than the valleys.

In general I have aimed to find average OAA temperature trends and average OAS temperature trends for the areas analysed. For each country analysed I have made comparison between national temperature trends as published by the “BEST” project and then the OAA and OAS temperature trends from original data. I want to know if BEST data use both the warm trended OAA data and the more cold trended OAS data. In addition, I have made comparisons of ECA&D data versus original for many countries and also HISTALP data versus original.

For all areas analysed (almost 20 countries by now) we see a large group of stations with warm temperatures trends after 1930 (“OAA” stations) but also a large group of stations with very little or no warm trend after around 1930 (“OAS” stations).
The classification of OAA versus OAS simply depends on geographical surroundings.
In the writing “RUTI Coastal stations” (based on GHCN V2 raw) I found that Non-coastal temperatures (blue graph) were much more cold trended from around 1930 than the Coastal trends (red).

<b.3.3 Adjustments: The BEST project
The BEST project collects data from different sources often already related to NOAA and NCDC. BEST often present multiple versions/copies of the same longer datasets already used repeatedly in climate science. BEST have not required [acquired] the large bulk of existing temperature data from the national Meteorological institutes.

Frank’s best illustration that “Regional Expectation” is THE BIG LIE, is BEST / HUNGARY.

Fig 12
For the Hungarian Valley (one of the largest OAS area in Europe), the BEST team has used an OAS temperature station “Pecs”. Above, the Pecs temperature trend is shown together with other Hungarian stations. These original data do seem rather homogenous?

Fig 13
None the less, the BEST team adds around 0.7 K of warming to the Pecs data. BEST use a so called “Regional Expectation” for all countries i have analysed, and change original data so they approach these expectations. Best also claim that Hungary as a nation has experienced this warming trend.

Zeke Hausefeather did a hatchet job on Steve in his article @ Judith Curry’s Understanding Adjustments to Temperature Data. The Mosh pup is there defending BEST. Zeke and Mosher are defending the TOBS adjustments that Steve (and I) say are bogus.

Zeke Hausfeather states:

……Nearly every single station in the network in the network has been moved at least once over the last century, with many having 3 or more distinct moves. Most of the stations have changed from using liquid in glass thermometers (LiG) in Stevenson screens to electronic Minimum Maximum Temperature Systems (MMTS) or Automated Surface Observing Systems (ASOS). Observation times have shifted from afternoon to morning at most stations since 1960, as part of an effort by the National Weather Service to improve precipitation measurements.

All of these changes introduce (non-random) systemic biases into the network. For example, MMTS sensors tend to read maximum daily temperatures about 0.5 C colder than LiG thermometers at the same location. There is a very obvious cooling bias in the record associated with the conversion of most co-op stations from LiG to MMTS in the 1980s, and even folks deeply skeptical of the temperature network like Anthony Watts and his coauthors add an explicit correction for this in their paper…..

I will take those three points separately.

SCALPEL and KRIGING
Elsewhere Zeke explains they use a computer program to ‘detect’ station moves. They are too darn LAZY to actually go to the first hand accounts and SEE if there are changes even when looking at current data. It is all automated.

An excellent example is my nearby station.
Original readings: July 30 2014 Min 59°F — Max 68°F
The Max Temperature was 68 °F because a cold front with rainy weather came through.That now is a Max Temperature 81 °F

For the maximum thermometer they state:“…When a maximum thermometer is not read for several hours after the highest temperature has occurred and the air in the meantime has cooled down 15° or 20°, the highest temperature indicated by the top of the detached thread of mercury may be too low by half a degree from the contraction of the thread….”

That would indicate the max thermometer should be read just after the heat of the day and any adjustment for reading at the wrong time of day should RAISE the maximum temperature not lower it!

He also states there are 180 to 200 ‘regular weather stations ordinarily in the larger cities, 3600 to 4000 coop stations and 300 to 500 special stations.

The observations of temperature taken at a regular station are the real air temperature at 8am and 8pm, the highest and lowest temperatures of the preceding 12 hours, and a continuous thermograph record…. (Richard Freres thermograph) ….these instruments are located in a thermometer shelter which is ordinarily placed 6 to 10 feet above the roof of some high building in the city. At a Cooperative station the highest and lowest temperatures during a day are determined, and also the reading of the maximum thermometer just after it has been set. The purpose of taking this observation is to make sure that the maximum thermometer has been set and also to give the real air temperature at the time of observation.

If a good continuous thermograph record for at least twenty years is available, the normal hourly temperatures for the various days of the year can be computed….

“the average temperature for a day is found by averaging the 24 values of hourly temperature observed during that day”

If the normals are based on twenty years of observations, it will be found that there is not an even transition from day to day, but jumps of even two or three degrees occur….

I thought it quite interesting that Willis Isbister Milham was talking about 20 years of hourly data in 1918. Also remembernone of the ClimAstrologists go and actually look at the raw data. Instead they interperted a ‘jump in the data’ as a station move. Yet Milham says these ‘jumps of even two or three degrees occur’ naturally.

On page 68 Milham says a thermometer in a Stevenson screen is correct to within a half degree. It is most in error on still days, hot or cold. “In both cases the indications of the sheltered thermometers are two conservative.”
>>>>>>>>>>>>>>>>>>>>>

TRANSITION from traditional glass thermometer measurement stations to the new electronic measurement system
In addition to the TOBS ‘adjustment’ BEST does another 0.5 C colder adjustment to the older liquid in glass thermometers. However that adjustment is just as bogus, actually more so than the TOBS adjustment.

The last couple of days I posted on an 8.5 year side-by-side test conducted by German veteran meteorologist Klaus Hager, see here and here. The test compared traditional glass mercury thermometer measurement stations to the new electronic measurement system, whose implementation began at Germany’s approximately 2000 surface stations in 1985 and concluded around 2000.

Hager’s test results showed that on average the new electronic measurement system produced warmer temperature readings: a whopping mean of 0.93°C warmer. The question is: Is this detectable in Germany’s temperature dataset? Do we see a temperature jump during the time the new “warmer” system was put into operation (1985 – 2000)? The answer is: absolutely!http://notrickszone.com/#sthash.Es2IbMZo.sAqMRsUB.dpbs

So that ‘adjustment’ just like the TOBS adjustment is also in the WRONG direction. This mean up to a 1.5°C cooling adjustment to the older data in the WRONG DIRECTION!
>>>>>>>>>>

Now add in the station drop out leaving just ocean affected or airport stations and you turn a cooling trend very neatly into a warming trend.

BEST has problems they do not acknowledge. Starting with data ingestion. Example Rutherglen Australia. A famous, pristine long record agricultural research station most of which data is simply missing in BEST. Menn stitching (splicing), example Zhang et al. theor. appl. climatol. 115: 365-373 (2014). Regional expectations QC, example Best station 166900 (Amundsen Scott). Best excludes 26 months of record cold based on disagreement with McMurdo, which is 1300 km away on the coast 2700 meters lower. Ridiculous

Or maybe the similarity of NH trends for BEST to NOAA/NASA/HadCRUT has more to do with the choice of sites to represent grid areas. By favoring sites with strong UHI influence the trend over time is artificially inflated.

Hey oz4caster! Site choice makes a huge difference, just as you point out. If you have not read E.M. Smith’s look into NASA’s software and site selection, I suggest you take a look. It is years old now, but still very much relevant. https://chiefio.wordpress.com/gistemp/

Or just call it something else.Steven Mosher | June 28, 2014 at 12:16 pm | [ Reply to the ” ” prior post & spelling in the original]
“One example of one of the problems can be seen on the BEST site at station 166900–not somempoorly sited USCHN starion, rather the Amundsen research base at the south pole, where 26 lows were ‘corrected up to regional climatology’ ( which could only mean the coastal Antarctic research stations or a model) creating a slight warming trend at the south pole when the actual data shows none-as computed by BEST and posted as part of the station record.” The lows are not Corrected UP to the regional climatology.

DD, today I was trying to take a deeper look into the BEST data because they seem to be more open and transparent about what they are doing with the data than NOAA/NASA/HadCRUT. I was trying to figure why their results match so closely with NOAA/NASA/HadCRUT even though they are using a much larger data set and supposedly use different methods. At first I thought that their use of HadSST might be the main reason for the similarity since oceans dominate the earth. However, after finding their Land Only graphs, that obviously is not the case. So, I started looking into their station data and intend to spend more time in the near future looking at this data. They provide “raw” data through a “data table” link below the “raw” data anomaly graph. Below is link for data from Elgin, Texas which is an example I posted in an earlier comment:http://berkeleyearth.lbl.gov/stations/26751

It shows the raw temperature anomalies and how that raw data was adjusted to match the “Regional Expectation”. I have yet to learn more about how they derive the “Regional Expectation” but suspect that may be the main problem with their approach. To their credit, they appear to be very transparent in what they do, as in the example of Elgin, but also appear to be very wrong in the adjustment they made to raw data from this relatively rural site that had little trend but after adjustment matches the upward regional trend that I suspect is UHI dominated.

Because of problems that others here have mentioned with the BEST “raw” data, I am hoping I can spot check some of the “raw” data, at least as compared to some Austin area temperature data I have collected over the years. Their tool for displaying the station data and downloading the raw data looks very helpful, provided the raw data are truly “raw”. I suspect verifying older raw data may not be easy since it was all on paper originally and at some point had to be digitized. To thoroughly verify it would involve tracking down digital images of the paper originals if they exist or tracking down the actual paper originals if they still exist. I’m not sure that I will go to that much trouble.

For older data, I favor the approach of using a few good sites with longer records than trying to “correct” lots of data from lots of poor sites. So I will be looking through the BEST data to find some of the better sites to see what they show.

For more recent years, I prefer the shotgun approach of using all the available data that go into the global weather models four times per day as in the CFSR and ERAI data sets. If it’s good enough for weather forecast models, it should be good enough for climate studies and should provide much better spatial coverage than GHCN data sets. However, I do have reservations that even the CFSR and ERAI data sets could end up being corrupted if they don’t fit the party line, since they involve a “reanalysis” of the original weather model input data. So far, I’m not seeing any obvious evidence of corruption in the CFSR data, although the ERAI data seems to very suspiciously match the HadCRUT data as if the reanalysis was designed to provide a better match.

(2) http://www.ncdc.noaa.gov/sotc/global/199813
Global Analysis – Annual 1998 – Does not give any “Annual Temperature” but the 2015 report does state – The annual temperature anomalies for 1997 and 1998 were 0.51°C (0.92°F) and 0.63°C (1.13°F), respectively, above the 20th century average, So 1998 was 0.63°C – 0.51°C = 0.12°C warmer than 1997

(3) For 2010, the combined global land and ocean surface temperature tied with 2005 as the warmest such period on record, at 0.62°C (1.12°F) above the 20th century average of 13.9°C (57.0°F).
0.62°C + 13.9°C = 14.52°C http://www.ncdc.noaa.gov/sotc/global/201013

(4) 2013 ties with 2003 as the fourth warmest year globally since records began in 1880. The annual global combined land and ocean surface temperature was 0.62°C (1.12°F) above the 20th century average of 13.9°C (57.0°F). Only one year during the 20th century—1998—was warmer than 2013.
0.62°C + 13.9°C = 14.52°C http://www.ncdc.noaa.gov/sotc/global/201313

(6) average global temperature across land and ocean surface areas for 2015 was 0.90°C (1.62°F) above the 20th century average of 13.9°C (57.0°F) = 0.90°C + 13.9°C = 14.80 °C
The annual temperature anomalies for 1997 and 1998 were 0.51°C (0.92°F) and 0.63°C (1.13°F) [16.92 + (0.63-0.51)= 0.12 >> 17.04 ] for 1998

So the results are 16.92 or 17.04 >> 14.52 or 14.52 or 14.59 or 14.80

And per the written sections –
2010, the combined global land and ocean surface temperature tied with 2005 as the warmest such period on record – @ 14.52°C
2013 ties with 2003 as the fourth warmest year @ 14.52°C but was at the same temp as 2010 & 2005 which were records.

Since 1997 was not even the peak year (per 2015 write-up 1998 was 0.12°C warmer), which number do you think NCDC/NOAA thinks is the record high. Failure at 3rd grade math or failure to scrub all the past. (See the ‘Ministry of Truth’ 1984).

Taking the statement “The annual temperature anomalies for 1997 and 1998 were 0.51°C (0.92°F) and 0.63°C (1.13°F), respectively, above the 20th century average”
Then, using 1997, their calculated ’20th century average’ was 16.92°C minus 0.51°C = 16.41 °C.
Then they switched to “the 20th century average of 13.9°C (57.0°F)”

WTF?!? do I have that correct?

Time for a screen shots Tony!

(This is not the first time I have seen that the baseline average has changed but I can’t find the link.)

“The best way to control the opposition is to lead it ourselves.” – Vladimir Lenin

Clinton’s mentor Georgetown University Professor Carroll Quigley in his book Tragedy and Hope, (1966) said the same.

The argument that the two parties should represent opposed ideals and policies, one, perhaps, of the Right and the other of the Left, is a foolish idea acceptable only to doctrinaire and academic thinkers. Instead, the two parties should be almost identical, so that the American people can ‘throw the rascals out’ at any election without leading to any profound or extensive shifts in policy.

It would not surprise me in the least if these guys have algorithms remove or add stations depending on trend of the data at those stations, especially for data from places like Russia where they “lost” a whole heap of stations, and now by adding some back in, show a large positive anomaly.

Thing is , that while it was a bit warmer in Russia, it was no warmer than 33 years ago. And of course, this is winter, so I’m guessing people up there really don’t think its a bad thing !!

The previous post looked at the potential effect of missing months on station annual mean tempertures and anomaly values where one month of data was missing and showed in the last figure just what a high proportion of stations have at least one month missing per year in recent years. Many stations, however, have much more than one month missing. Figure 1 shows the missing months in GHCN data from 1800. There may have been very few stations reporting in the 19th Century, but up to 1875, those that are present in the record had more than 90% reporting rate. The record is 85-95% complete up to ~1960 then drops rapidly: currently only ~70% of station data is complete in each year. The rest? Oh dear.

Figure 2. Distribution of missing months by month and year (analysis and graph: Andrew Chantrill).

Fig. 2 shows how missing months are distributed across the each year. Is is possible there is greater loss of Winter months, particularly in more recent years? Loss of cooler months in Northern latitudes – surely not? We showed previously that missing Winter months in stations with large temperature variations affect anomaly values more than missing summer months – lower accuracy of the data – again.

Now do I have to say it? If we are heading for cooler times, and colder winters, with lower or negative anomalies, how convenient that more Winter months are dropped.