Quality Control – Jones and Hansen Style

Anthony Watts has an excellent post showing the calibre of the quality control carried out by Phil Jones and Jim Hansen and the quality of Phil Jones’ “proof” that the “overall urban bias …is greater than 0.05 deg in the 20th century”. Marysville CA (425745000030), GISS population 12,000, is in the USHCN network and is used in the GISS, CRU and NOAA calculations of global temperature. Here’s one of Anthony’s pictures; go to his site for more sickening pictures.

Here is a plot of the temperature history at Marysville as downloaded by Anthony from GISS

Anthony contrasted this with results from nearby Orland CA (425725910040), GISS rural area, where the temperature history is shown below:

One small tweak on Anthony’s interpretation. His GISS graphic shows the “raw” not the “adjusted” GISS version. But as far as I can tell, the NOAA calculation uses the “raw” version according to their the readme which says:

This data set contains gridded temperature anomalies calculated by the “anomaly method”. Gridpoint temperatures are calculated by averaging the unweighted raw data from all stations within the grid box, and then the anomaly is the difference from 1961-90 mean.

So this is worth bearing in mind when NOAA tells you that March 2007 was the warmest in a milllllll-yun years or was it a billlll-yun years. The “adjusted” Marysville data from GISS is shown below:

The UHI warming here isn’t as bad, but it’s still arbitrary and biased relative to Orland. (And they reduced “warm” 19th century Orland temperatures no doubt for a “good” reason.) They also reduced Orland temperatures in the 1930s; the adjusted Orland temperatures are warmer now than in the 1930s; the opposite is the case with unadjusted. I wonder how they adjusted the Marysville temperatures. But what to do with the adjustments to the Dawson, Yukon data which Rob Wilson rejected. How does one go about sometimes using their adjustments and dometimes not.

But let’s say that they got all their adjustments exactly right. What does that say about the quantum of UHI? Here’s a town of 12,000 which qualifies as “rural” in all the UHI studies. The non-climatic effect here is at least 3 deg C. In this case, we can estimate it because there’s a less bad station nearby. What does one do in China or Indonesia where nearly all the stations seem to be in large cities (even if they started out small, they’re large now)?

[UPDATE: Eli Rabett has claimed that I “blew” it when I said that this “town of 12,000 qualifies as ‘rural’ in all the UHI studies”, because it is not used as a GISS unlit site. The phrase “UHI studies” is not especially clear, but is intended to cover studies like Jones et al 1990, Karl et al 1998 purporting to show that there is negligible UHI effect in global composites; the GISS composite is not a “UHI study”. Jones et al 1990 used Chinese cities with populations under 100,000 as “rural” comparanda. Karl et al 1988 said:

Due to the large number of stations located in sparsely populated areas [over 85% (70% of all stations had a 1980 population of less than 25 000 (10 000)], the impact of urbanization is not large in relation to decadal changes of temperature in the United States.

Marysville falls squarely into Karl’s definition of a “sparsely populated” area. The salient point is that the GISS adjsutment estimated a substantial UHI effect at Marysville requiring adjustment; if this adjustment is justified, then GHCN and CRU (and IPCC relying on CRU) should have used it. Eli can’t have it both ways. [End UPDATE Jul 30, 2007]

And what can one say about quality control as practiced by Phil Jones and Jim Hansen if they let this sort of stuff into their network? Maybe they should spend a little less time going to conferences and a little more time doing quality control on the temperature data that they’re being paid to report on.

Here are plots of the GISS adjustments for Marysville and Orland. I haven’t tried to wade through GISS adjustments yet. At a first impression, the adjustments look to implemented in 5-year steps. This is obviously something different than Time-of-Observation adjustments or step adjustments. Any thoughts on what they’re doing would be welcome.

Update 2:
Here is the difference between the GHCN raw data and the GISS raw data:

Here is the difference between the GHCN raw monthly data and the USHCN raw (“Areal”) data.

In each case, the adjustments have the effect of increasing the trend.

A GISS Rural Cooling Adjustment?
Here’s another odd plot, which I’ll have to triplecheck. The first diagram below shows 3 calculation stages: GHCN Raw- GISS Raw (which should deal with all the time of observation and station history stuff); GISS Raw – GISS Adjusted (the GISS urban heat adjustment); the combined GHCN Raw – GISS Adjusted. I’ve done this first for Marysville GISS population 12,000, periurban lighting; and Orland GISS rural; unlit. We’ve seen a strong UHI effect at Marysville and GISS makes a substantial adjustment in the early portion of this series – up to 2 deg C, notwithstanding Jones view that UHI does not exceed 0.05 deg per century.

Marysville CA Adjustment Stages

Now the same things with Orland CA. This time the GISS adjustment goes the other way – I presume that the rationale for this adjustment is the well-known “rural cooling” effect, but I’ll ocntinue investigating.

113 Comments

How in hell does Jones figure that only 0.05 deg per century adjustment is accurate? I’ve stood in the middle of an asphalt covered parking lot in summer and it’s damn hot. If I move over to a field with natural ground cover the temperature is a hell of a lot less.

From that very revealing photo they might as well have placed the Marysville weather monitoring station in the middle of the asphalt. LOL

On second thought I have a better idea. Separate the urban temperatures from the non-urban temps and show both results. It should be readily apparent that UHI is definitely a factor in global warming. It doesn’t make sense to “average” the temperatures because the people in the cities don’t live in “average” temperature environment.

On second thought I have a better idea. Separate the urban temperatures from the non-urban temps and show both results. It should be readily apparent that UHI is definitely a factor in global warming. It doesn’t make sense to “average” the temperatures because the people in the cities don’t live in “average” temperature environment.

Can I suggest that just as you have rendered the word average as “average”, so we should render the words global warming as “global warming”.

How do we find out where our local temp gauges are? How about we all do digital photos of our local stations and post them at some common site on a common thread? There’s a lot of us now. Perhaps we could start something useful. In the beginning of course it would be completely annecdotal, but if we collected appropriate info for each site, perhaps at some point it would become more than just annecdotal. But maybe we should first send Bender some Dramamine or something:)

I wrote a letter to New Scientist about their recent article. I pointed out that their graph seemed exaggerated (it showed a recent rise of about 0.7 degrees). Of course they didn’t print the letter, but to their credit they are willing to enter into some discussion. The editor informed me that the graph was based on the GISS data, which does indeed show a very steep rise in recent years. By choosing GISS and northern hemisphere data New Scientist had chosen the most extreme combination which almost certainly exaggerates the true state of global warming.

I was already familiar with the Hadley graphs. Clearly there is a big difference between Hadley and GISS. Hadley shows that global warming has effectively come to a halt, even if temporarily. The question obviously arises: which data set is more reliable and why? The example above is appalling. Is this typical of GISS or is it an exception? Overall, how reliable are Hadley and GISS?

Thanks for the comments, you may also enjoy this post I made on a spot test of latex paint compared to whitewash as was originally specified for Stevenson Screens, but appears to have been discontinued around 1979.

I have a running experiment right now in addition to this spot test with a 4 channel datalogger with NIST calibrated temp probes measuring the temperature of boreholes in the painted, whitewashed, and bare wood (for control) I can tell you that preliminary results show whitewash consistently 2-3 degrees cooler through the day once solar heating commences. I’m also logging aspirated air temperature within 2 feet of the experiment and I’m making wind and solar insolation logs also.

I’m running this wood borehole experiment as a precursor to getting three new Stevenson Screens delivered. They had to be special ordered, without paint, so it takes time.

I will make all the logger data available publicly and will provide experiment details so that anyone whom wishes to reproduce it may do so.

And I have an even bigger project that I will announce soon. As they say in television, stay tuned.

In the meantime, it appears that measuring temperatures in parking lots here in California is becoming a trend. Here is a picture of the Yolo County Cooperative Extension Service Building which also houses the County office of weights and measures. This is a Cooperative Weather Observer station with an MMTS.

Although at the USHCN Version one site here, http://www.ncdc.noaa.gov/oa/climate/research/ushcn/ushcn.html, it does state that adjustments are made and that “The cumulative effect of all adjustments is approximately a one-half degree Fahrenheit warming in the annual time series over a 50-year period from the 1940’s until the last decade of the century.”

Hasn’t anybody noticed the automatic adjustment provided by the National Weather Service for the asphalt’s heat at Marysland ? Just look at the green plants impeccably trimmed (& copiously watered ?) just under the station.

#15. Jones has purported to put error bars on his reconstruction – see Brohan et al for the most recent effort. Whether the error bars mean anything is a different question. They don’t analyze errors like we’re discussing here.

Instead of digital photos of the stations, I’ve found it interesting to use the station coordinates and find them on GoogleEarth.
Gives a good idea of the surroundings, albeit with the occasional jaw-dropping moment.

Steve,
Are you sure CRU use Marysville ?
I can not see it in my Jones 1994 station list.
I am not aware of any published list after that.
Just wondering if you have a list or map that specifies CRU use Marysville.

I am impressed with Anthony Watts work documenting these poor quality sites and have put up a post pointing to some pages on Californian trends.http://www.warwickhughes.com/blog/?p=112
I have also made a series of graphics from Climate Explorer for the LA and San Francisco grid cells showing the often disparate trends for the various global databases.
Off topic, the “Errors in IPCC climate science” Blog has been upgraded after we realised the effect of the tidal wave of spam. Commenting should be easier now.

Good idea! Along with the digital photo, the station ID no., the station coordinates, and a specific focus on foreign stations that are at airports post 1991. When Hans wakes up, we can send him to the airport with a camera:)

Well Anthony I’ll post a comment over on your site, when I get a chance. Orland is
definately more rural than Marysville, but neither are trurely Rural and both areas
have undergone significant land use change ( agri culture and development)

Now, I found a source that you might use to double check or compare to the USHCN data.

It is state of california agriculture data. Daily records back to 1951, min, max, precip. Solar
Intensity in some cases. The cool thing is that Geographiclly you get coverage for areas that are
more rural. So, you could check Marysville data againts cities that are closer than Orland,
like willows or williams or Oroville, I think beale AFB is close by as well. So with Williams
a stones throw to the west ( and very rural) and baeal AFB a few miles to the east of marysville
It might be enlightening.

I am working with Anthony Watts in collecting Sierra USCHN site data, taking digital pictures and conducting site surveys. The results of two surveys are here and here.

I am still working on additional stations. After automation, the Spaulding station data was not added to the GISS data base, but I could not find an explaination. Colfax is still a manual station.

Pacific Gas and Electric is the major hydro electric provider in California and has a strong interest in climate change. I found this in a Pacific Gas and Electric Report on the impact climate change might have on hydro operations:

8. CLIMATE STATIONS
In years prior to the changeover of high elevation, manually read climate stations that were utilized to gather precipitation and air temperature data to remote automated, non-visited stations, station data was cooperatively shared with the National Weather Service (NWS). An NWS cooperator visited the stations daily and standards for data collection in terms of both equipment and data collection quality were for the most part assured with regularly scheduled visits by the NWS station network specialists. Today that situation is changed with the removal of most lake tenders and powerhouse personnel from many of PG&E’s mountain climate station sites. Automation and the ease of satellite telemetry have changed how climate data is collected at many mountain station sites. This change in methodology has contributed to additional uncertainty as to what is believable in terms of having significance for identifying and tracking climate change.

The report concludes that: At this time PG&E’s water management team has not observed any significant change in hydroelectric production that can be directly attributed to global warming or climate change.

On some of the other sites I am surveying, after automation large chunks of data are missing, in some case whole years. More on this as I survey more stations and check the data tables. These preliminary findings seem to justify PG&E concerns about the automated stations.

Douglas Hoyt has found a very complete site that has official photographs of weather stations. I looked up KBOS in Boston, which turns out to be located in a rectangle formed by four runways at Boston’s Logan International Airport. You can see a jet taking off in one photo. It should give excellent unbiased temperature readings.

Re 22, I recall in the early 1990’s there was a policy in Australia to de-person lighthouses. Some mad Govt Dept trying to reduce their wages bill with no thought for say, the pluses of having remote area rangers, coastwatch, keeping world heritage buildings intact and in our case, world heritage weather / climate time series. This lead to installation of AWS (automatic weather stations, I think) at long term rural sites, resulting in inhomogeneities, data gaps as these new devices broke down at the far away localities and repairs took time to arrive. I am sure it is not only lighthouses and the intro of AWS was staggered over many years, I bet that the effect runs like a jagged (timewise) fault line thru Australian data. I recall I could pick the start of AWS data by a change in style of daily readings.
Re 23, yes most of us were aware of the Pielkes good efforts for many years but 15 to 20 years on into the AGW panic and in one of the wealthiest and most advanced societies on Earth, just up the road from Silicon Valley, it is still a shock to see such new, probably expensive installations that are next to totally useless for climate purposes. Without being disparaging one has to ask the question, what could be found in the Third World.
Anyway, it has got me interested to take my camera to a few places around here.

Well, the network WAS designed primarily for local weather forecasting, wasn’t it? So go figure. Who cares about inhomogeneities if you’re trying to forecast 3 days for a single point, as opposed to 300 years over the entire globe?

Steve, I did look at Williams, CA but I don’t know what to make of it because my site survey there uncovered nothing, meaning I can’t find the weather station. The police dept doesn’t know anything about it, as they were listed as the observer. A visit to the poliuce station at the coordinates listed revealed nothing, no sensors of any kind. I have been using the UC Davis IPM reference site to find other stations.

I visited Colusa yesterday, right after Marysville. Its at the sewage treatment plant, 2 miles SW of town, and here are photos looking north and south, resepctively. It truly is in the middle of nowhere, and despite some questionable mounting, seems to read well enough. Lots of wind there helps exposure and mixing.

There are two UHI effects right? One comes into play when you are averaging stations across a geography that contains both cities and rural areas. The second occurs for each station as its immediate environment changes through population growth and development. The other issue is the siting of the station. Here there are also two conditions: Smart and not so smart!

But, arguably, even this site will get reflected heat from those buildings all day long. (One building wall is facing East and the other West, according to your description.) They may be fairly recent, being of good condition except for some thinning paint.

Of course you are right Bender. These station locations should not be a major problem for recording and predicting, local day to day weather. But to use them to conclude that over the last 100 years there has been global warming of .6C, that it is now the hottest in the last 1000/1,000,000 years and to further to conclude what portion may be anthropogenic, well, to say the least, one must be careful. I guess I’m not saying anything that isn’t already well understood.

1) Page 11 notes that calibration isn’t possible on the MMTS, and gives a procedure for checking accuracy. If MMTS and thermometer agree within a degree, it’s considered OK
2) Towards the bottom of page 11, note that observers are only to record temps to the nearest degree.

On a correctly functioning unit, the actual temp could be 71.5, with a recorded temp of 70 and that would be considered within acceptable limits (measures one degree low, plus half degree rounding). Or actual temp could be 71.5, and recorded temp could be 73 and that would be within acceptable limits.

These are thermistor sensors, so in practice the error would be a constant offset from reading to reading, closing that 3 degree window down to +/- 1 degree.

But, if a unit is ever replaced, it’s possible the replacement unit could have a +1 bias while the unit it replace had a -1 degree bias. Both would be within limits.

Your link works. I happened to check the entry for Dubai. Interestingly, it says, “The location information for this site was gathered from 7 data sources. The possible locations for this site are up to 2.2 miles away. This makes it possible that the location chosen is incorrect. The source which most closely matches the consensus is chosen. Many of these sources only report the location to the nearest minute (or hundredth of degree). This can lead to the marker being placed noticeably incorrectly. Please see Discussion of Location Errors for more information.” They indicate they don’t have a picture of the station, but would like one.

The observatory in Prague Klementinum is in the very center of the city – there is a placemark at Google Earth (coordinates:W 14.415801,N 50.086528). Therefore it was a subject of some studies also including UHI effect:

Article about heat waves in 20th century – in 40-50’s it was hot (also observed in Basel, Switzerland), but small interannual differences, whereas in 90’s, there were big differences)Article about differences in UHI under various weather “The increase in the heat island intensity is steeper under anticyclonic than cyclonic conditions in all seasons except for spring.”

Heat waves and circulation – where the heat waves in 1990’s are ascribed to “an enhanced persistence of the atmospheric circulation over Europe in the summer season because all groups of weather types have considerably increased residence times in 1988’€”97 compared with long-term means.”

This site is as close as you can get to the Marysville site. Seems to be about two miles east, if both sets of coordinates are correct. Not as long a series, unfortunately.

The NCDC database I have gives the mean for all readings for the day. Also, tmin and tmax for which I’ve calculated the average. It has 24 readings per day about 28% of the time.

I’ve done maybe a dozen plots this way.(mostly airports) This is the first time I’ve seen the average drop below the mean for any extended time. Usually the mean stays about 3-6 tenths lower than min max average. At this site they swap and there is up to a full degree difference. Very curious.

Dear Anthony Watts, I look forward to the results of your expirement with the threeS tevenson Screens boxes.
I am just wondering if you are going to make air pressure recordings at the same time as you are measuring temperature? I wonder about the total energy in the system, which will depend on both the temperature, water vapor content and the air pressure. It is possible that there may exist air pressure gradients as well as temperature gradients in the different boxes.

With regard to your previous photographic series, I wander what a childrens sized paddling pool at the base of one of the boxes will do to the recorded temperature.

I have a complete weather station on the premises, which will record and log pressure, wind, solar radiation, air temp, and DP simultaneously with the running Stevenson Screen experiment. When I’m ready, I will have real-time data posted to the net in graphical form so everybody can watch.

I also have been in contact with the chief chemist for the National Lime Corporation and the National Lime Institute to acquire the correct hydrated lime formula that meets historical formulas for whitewash. They were gracious enough to send me the supplies I need, gratis.

BTW an interesting side-note to all of this is that by preliminary experiments with wood slats and thermistors indicate that whitewash appears to be a superior coating when it comes to energy efficiency for a/c of buildings. With all the rage being sustainability these days, whitewash will keep a building 2-3 degrees cooler it appears. Old is new again.

There’s a bit of a smoking gun in the document for CWO’s you reference. It says on page 13 “The CRS should kept painted with a semi-gloss exterior white paint at all times, not only for looks but also to reflect the suns rays.”

and then again on page 14 under maintenance:

“Maintenance is minimal for the most part. The CRS should be kept clean with a good quality semigloss white painted surface.”

This contrasts the earlier USWB documents which specify whitewash. It appears that the NWS simply caved to availability, and stopped requiring whitewash somewhere around 1979, which is when I first start noticing the “semi-gloss” paint reference. Note they don’t say what kind, so the variability may now include latex, enamel, oil based, or expoxy based “semi-gloss”.

The U.S. Historical Climatology Network (USHCN) is a high-quality, moderate-sized data set of monthly averaged maximum, minimum, and mean temperature and total monthly precipitation. The USHCN was developed to assist in the detection of regional climate change; it is comprised of 1221 high-quality stations from the U.S. Cooperative Observing Network within the 48 contiguous United States.

I’ve double-checked that Marysville is one of the “high-quality” stations in the station list.

Ya know what, it just hit me, I need to do a parallel test with the MMTS screen at the same time I do the Stevenson Screen tests, since it is the newest shelter. Since I’m comparing shelters, it seems only fitting. Does anybody know where I can buy the standard MMTS stacked plate shelter?

I did a more extensive check on Colusa and did a Annual average. for 1951 to 2006.
Eyeballing it, maybe a slight upward trend, nothing like Marysville.
As mentioned above Yuba is closer, albeit a more urban area than Colusa.

Load the file in Excell for a quick look and subtract Urban monthly Mean from
the unadjusted data.

Interesting, Most months the unadjsuted values exceed the urban adjusted.
So back in 1870 they are adjusting the monthly mean in Marysville ( down .12 F)
for Urban effects. Gold fever I guess.

Then I spot some outliers. Jan 1914. The adjusted file is 6F cooler.
also in the file some of the adjustment go Negative. So, in June 1895
the (unadjusted-urban adjust) = -2.3

Strange, maybe they are making adjustments at long time scales ( like steve
notes) and then propgating those changes down to monthly averages. Not
sure how this would result in a one month 6 degree F adjustment down
or a 2.3F adjustment up?

The decadal cycle is the solar cycle, is it not? Then the lower-frequency fluctuation, the one modulating the amplitude of the decadal cycle, with cooling in the 1960s-70s, is that the presumed effects of aerosols? Then these data are not “raw”, correct?

1) CO2 effect on global temperature is global. It should not be greater in urban areas.
2) Urban and Suburban areas make up a very small part of the actual global surface and data should be weighted by the percentage of earth surface area they represent. Common f***ing sense analysis.
3) Urban areas are much more likely to show solar effects (such decreased cloud cover) due to very low albedo of the surface and should show the exact same temperature rise due to GHG effects as similar rural areas.
4) Urban areas contribution should be further reduces by identifying heat sources and realizing that they ultimated will dissipate throughout the volume of the atmosphere.

I suspected that there would have to be adjustments for station changes etc.
Yet the issue ( to second what steve says in #50) is the adjustment in the GHCN
data appears to have been done on a continuous basis. Essentially, subtracting
a small amount ( .12 F to .17)from the monthly mean. But then every once in a while
you have individual months ( like jan 1914) that have a 6F degree adjust. and periodicly
through the data you have negative adjustments. On an annula basis these odd months
don’t have a big effect, which led me to believe that the adjustemnt may have be made
on a annual scale or multi annual scale and then massaged into the monthly data,
automatically, without anyone making a check on the monthly results.
I dont thnk an individual went over this site and in the middle of 1895 decided that
june needed a -2.3F adjustment. If that were the case, one could expect to find this kind
of adjustment decision documented. Same with Jan 1914. But, I could be wrong.

2. Descriptions and Formats of USHCN Data Files and Supporting Files
The section describes the various USHCN temperature and precipitation data files and several supporting files that contain information about the USHCN stations. Temperature and precipitation data files will be described first, followed by descriptions of all other files.
The data for each station in the USHCN are subjected to several quality control tests, homogeneity tests, and adjustment procedures. These steps are applied so as to sequentially produce six different types of data records, with each successive record type using the preceding record type as input. Each data record type is listed below with a description of the tests and/or adjustments that go into making each of them.

Areal Edited (Raw)
A quality control procedure is performed that uses trimmed means and standard deviations in comparison with surrounding stations to identify suspects (> 3.5 standard deviations away from the mean) and outliers (> 5.0 standard deviations). Until recently these suspects and outliers were hand-verified with the original records. However, with the development of more sophisticated QC procedures at NCDC, this has been found to be unnecessary.

TOBS (Time of Observation)
The temperature data are adjusted for the time-of-observation bias (Karl, et al. 1986), which occurs when observing times are changed from midnight to some time earlier in the day. The ending time of the 24-h climatological day varies from station to station and/or over a period of years at a given station. The time of observation (TOB) introduces a non-climatic bias into the monthly means. The TOB software is an empirical model used to estimate the TOB biases associated with different observation schedules and the routine computes the TOB with respect to daily readings taken at midnight.

MMTS (Maximum/Minimum Temperature System)
Temperature data at stations that have the Maximum/Minimum Temperature System (MMTS) are adjusted for the bias introduced when the liquid-in-glass thermometers were replaced with the MMTS (Quayle et al. 1991). The MMTS program debiases the data obtained from stations with MMTS sensors. The NWS has replaced a majority of the liquid-in-glass thermometers in wooden Cotton-Region shelters with thermistor based maximum-minimum temperature systems (MMTS) housed in smaller plastic shelters. This adjustment removes the MMTS bias for stations so equipped with this type of sensor. The adjustment factors are most appropriate for use when time series of states or larger areas are required.

SHAP (Station History Adjustment Program)
The homogeneity adjustment scheme described in Karl and Williams (1987) is performed using the station history metadata file to account for time series discontinuities due to random station moves and other station changes. The debiased data from the MMTS adjustment are then entered into the Station History Adjustment Program or SHAP. The SHAP allows a climatological time series of temperature and precipitation adjustment for station inhomogeneities using station history information. The adjusted data retain their original scale and are not anomaly series. The methodology uses the concepts of relative homogeneity and standard parametric (temperature) and non parametric (precipitation) statistics to adjust the data. In addition, this technique provides an estimate of the confidence interval associated with each adjustment. The SHAP program debiases the data with respect to changes other than the MMTS conversion to produced the “adjusted data”. Specific details on the procedures used are given by Karl and Williams (1987).

FILNET (Fill Missing Original Data in the Network)
Estimates for missing data are provided using a procedure similar to that used in SHAP. This adjustment uses the debiased data from the SHAP and fills in missing original data when needed (i.e. calculates estimated data) based on a “network” of the best correlated nearby stations. The FILNET program also completed the data adjustment process for stations that moved too often for SHAP to estimate the adjustments needed to debias the data.

Urban (Urban Warming Adjustment)
The final adjustment is for an urban warming bias which uses the regression approach outlined in Karl et al. (1988). The result of this adjustment is the “final” version of the data.

The Areal Edited, TOBS, FILNET, and Urban data files are available in this version of the USHCN. The MMTS and SHAP iterations of the data are also available (email D. Kaiser at CDIAC), but are used less often in favor of the FILNET data files, which incorporate both the MMTS and SHAP adjustments.

Currently all data adjustments in the USHCN are based on the use of metadata. However, station histories are often incomplete, or changes that can cause a time series discontinuity (such as replacing a broken thermometer with one that is calibrated differently) are not routinely entered into station history files. Because of this, NCDC is currently developing another step in the processing that will apply a time series discontinuity adjustment scheme described in Peterson and Easterling (1994) and Easterling and Peterson (1995). This methodology does not use station histories and identifies discontinuities in a station’s time series using a homogeneous reference series developed from surrounding stations.

When I read through the entire link I was a lot more well informed.
It’s a good piece of work. Still, I’m puzzling on these one time “oddball”
adjustments. Since Anthony mentioned chico I looked at that. from 1870
on the adjustments are all in the same range lets say .18 F to .22 F
Then around 1949, the adjustments start to smoothly ramp up ( So, One assumes some kind of station
change or shelter change or whatever) and the adjustment ramps up to around
.8F to the year 2000. All well and good.But then here and there are these “oddball”
months. adjustment of 1.54F in August of 1980. Is this an artifact of the adjustment
routines or an adjustment made for a specific reason that month. done by hand?

“Another factor users should be aware of pertains to thermometers used at the HCN/D stations. In 1984, the NWS introduced a new Maximum/Minimum Temperature System (MMTS) at cooperative network observing stations. Through 1994, 645 out of 1062 (about 60%) of the HCN/D stations had installed an MMTS (the station history file identifies these stations). Concerns have arisen about the calibration of this system as compared to that of the earlier thermometric system. The new system is thermistor-based with a “beehive like” instrument shelter, whereas the older systems consisted of liquid-in-glass thermometers, mounted inside a Cotton Region Shelter (Stevenson Screen). Quayle et al. (1991) looked into performance differences of the two systems and found that the new system produces maximum temperatures about 0.3 deg C lower and minimum temperatures about 0.4 deg C higher than the old system. Unfortunately, because large samples of side-by-side overlapping measurements are not available, site-specific corrections cannot yet be derived and only large-scale temperature changes can be corrected. Furthermore, daily biases, which are likely to be dependent on synoptic conditions, are unlikely to be the same from day to day. Thus, to date there has been no attempt to adjust the daily temperature data from the HCN/D for these instrument-induced biases”

Jerry, here are differences for Marysville USHCN Area minus Filnet. The adjustments seem to be determined by season. While recent adjustments are not large, older adjustments can be significant as shown below and cannot be attributed to rounding.

Jerry, as you pointed out to me, the USHCN code “A” doesn’t mean Area, but Filnet. However, if one looks at the USHCN adjustment stages, the Area-TOBS delta is different for each month, but the TOBS-Filnet adjustmnet (which is what I actually have above) is different by season for some reason. GISS raw is close to USHCN Filnet, but not exactly.

USHCN “raw” should be within rounding of GHCN “raw”.
GHCN adjusted should be within rounding of USHCN filnet.
GISS “raw GHCN data+USHCN corrections” should be close to
GHCN adjusted minus filnet adjustments for missing data.

I may have missed the discussion, but my experience with thermistors (metal juctions) is that they fail open, not shorted, which means that cumulatively they must progress to higher impedences. Might that mean higher temperatures as they age?

Here’s another odd plot, which I’ll have to triplecheck. The first diagram below shows 3 calculation stages: GHCN Raw- GISS Raw (which should deal with all the time of observation and station history stuff); GISS Raw – GISS Adjusted (the GISS urban heat adjustment); the combined GHCN Raw – GISS Adjusted. I’ve done this first for Marysville GISS population 12,000, periurban lighting; and Orland GISS rural; unlit. We’ve seen a strong UHI effect at Marysville and GISS makes a substantial adjustment in the early portion of this series – up to 2 deg C, notwithstanding Jones view that UHI does not exceed 0.05 deg per century.

Marysville CA Adjustment Stages

Now the same things with Orland CA. This time the GISS adjustment goes the other way – I presume that the rationale for this adjustment is the well-known “rural cooling” effect, but I’ll ocntinue investigating.

With apologies: I forgot to warn you that the USHCN version
currently at ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/
is not used either in GHCN, or in GISS, and it has different
adjustments than the previous version that was posted there.

A previous USHCN version, which was not posted at the USHCN ftp
site, is what is being used both by GHCN and GISS. I do not
have a copy of that version, but I do have copies of earlier
versions. Let me check the filnet vs GHCN adjusted from
an earlier version and get back to you.

I decided I’d drop some more fun with entropy your way. Here is the USHCN station in Redding, CA # 425725920010 now operated by the US Forest Service at their HQ located at the Redding Airport it used to be operated by the NWS, but that WSFO closed in the mid 90’s.

Like Marysville, the site is surrounded by asphalt, and the surface is unnatural – its wood chips over weedmat, and I’ll have to say it was hot as hot to walk on.

But the kicker is the “accessories” they’ve
added for convenience of running the hygrometer and for night observations. Yes is another fine high-quality USHCN site. I winder how many times they forgot to turn off the light? Its a 65 watt bulb by the way.

“Approximately 1,000 homogeneity-adjusted stations in the United States were
computed using a different technique. These are high quality rural stations
taken directly from the U.S. Historical Climatology Network (U.S. HCN,
Easterling et al. 1996a) a sister project to GHCN. These data were adjusted
using a metadata approach as part of the creation of the U.S. HCN and their
adjusted time series were directly incorporated into GHCN.”

I do not clearly recall what led me to conclude that those adjustments
were up to, and including, the USHCN filnet adjustments. In any case, I
must withdraw my earlier comment that they are, and go back and figure out
just what they do include, a task which will not get done tonight.

GISS #1 and #2 track the USHCN very well. throughout the whole series.

Homegenity, as you note, diverges substantially from USHCN. For grins I compared two records
from Colusa : one with records from 1951 on, the other with records from 1983 on. ( by the
way both series were remarkable similair in shape to each other and to the marysville record.)
Colusa rural sites were 1-1.5C lower than Marysville over the period.

I suppose Hansen does not publish the slope of his adjustment curve for Marysville?
He just describes it:

The nonrandom inhomogeneity of most concern is anthropogenic influence on the air sampled
by the thermometers. Urban heat can produce a large local bias toward warming [Mitchell, 1953;
Landsberg, 1981] as cities are built up and energy use increases. Anthropogenic effects can also
cause a nonclimatic cooling, for example, as a result of irrigation and planting of vegetation, but these
effects are usually outweighed by urban warming.
We take advantage of the metadata accompanying the GHCN records, which includes
classification of each station as rural (population less than 10,000), small town (10,000 to 50,000),
and urban (more than 50,000), to calculate a bilinear adjustment for urban stations. The adjustment
is based on the assumption that human effects are smaller in rural locations. We retain the
unadjusted record and make available results for both adjusted and unadjusted time series (section
10). The homogeneity adjustment for a given city is defined to change linearly with time between
1950 and the final year of data and to change linearly with a possibly different slope between 1950
and the beginning of the record. The slopes of the two straight line segments are chosen to minimize
the weighted-mean root-mean-square difference of the urban station time series with the time series
of nearby rural stations. An adjusted urban record is defined only if there are at least three rural
neighbors for at least two thirds of the period being adjusted. All rural stations within 1000 km are
used to calculate the adjustment, with a weight that decreases linearly to zero at distance 1000 km.
The function of the urban adjustment is to allow the local urban measurements to define short-term
variations of the adjusted temperature while rural neighbors define the long-term change. The break
in the adjustment line at 1950 allows some time dependence in the rate of growth of the urban
influence.”

This reminds me of a water resources study we were doing in S. Iran in 1969. At the town of Jask, at the entrance to the gulf, the mayor was also the local weatherman. We stayed at his place overnight, and I noticed the rain gage and thermometer were indoors on his mantle; when I asked why they were there, he said he had to bring them in at night so no one would steel them. So the night half of the record is blank.

I would think this or similar bias might be a fairly common occurrence, especially in the earlier 3rd world records. However the colonial records – pre WWII – may be of somewhat better quality. Not sure to what extent things may have improved; probably very spotty. War and revolutions have left their mark in terms of long periods of no data in many countries between 30N and 30S.

Brohan et al gives a good overview how this kind of adjustments are taken into account when computing uncertainties. I’m not sure if I understood it correctly, but it seems to be a humoristic approach

1. Make a histogram of known adjustments. You’ll lose time information, but stochastic processes are not invented in climate science yet, so there are no other options.

2. This histogram is bimodal, but assume that true required adjustment histogram is Gaussian -shaped. So you’ll need to fit a Gaussian distribution to that bimodal histogram.

3.

The homogenisation uncertainty is then given by this missing component (dotted line in figure 4), which has a standard deviation of 0.4 C.

4. Tell everyone that you understand that adjustments are not symmetric, but add a sentence

Although a real effect, this asymmetry is small compared with the typical adjustment, and is difficult to quantify; so the homogenisation adjustment uncertainties are treated as being symmetric about zero.

5.

So the homogenisation adjustment uncertainty for any station is a random value taken from a normal distribution with a standard deviation of 0.4 C.

6. Finally, tell everyone that it is important to make adjustments

Figure 4 also demonstrates the value of making homogenisation adjustments. The dashed line is an estimate of the uncertainties in the unadjusted data, and the dotted line an estimate of the uncertainties remaining after adjustment. The adjustments made have reduced the uncertainties considerably.

In this paper Jones, like Hansen in the previous papaer, is not differentiating between “urbanization effect” and microsite effects due to changing circumstances around the sensor. The UHI effect may or may not be as marginal as he claims. But the microsite effects, which are ignored, may be significantly larger.

Of those files, the file hcn_shap_ac_mean.Z has numbers that include
TOB, MMTS, and SHAP adjustments, but not FILNET adjustments. Based on
comarisons using Marysville data, it appears to me that the GHCN adjusted
data for USHCN stations uses that level of USHCN adjustments: SHAP, not
FILNET.

Whoops, posted a little too quickly there. In the next paragraph they write:

Bias correction uncertainties are estimated following [Folland et al., 2001] who considered two biases in the land data: urbanisation effects [Jones et al., 1990] and thermometer exposure changes [Parker, 1994].

So they are well aware of the two scales of effect.

But later still:

Thermometer exposure changes. Over the period since 1850 there have been changes in the design and siting of thermometer enclosures; many early shelters can differ substantially from the modern Stevenson-type screen. It is sometimes possible to determine the time of change by the homogeneity assessments discussed in section 2.3.1, but this is only possible if changes at neighbouring stations are implemented at different times. The bias errors in this section, therefore, allow for the possible simultaneous replacement across entire countries with Stevenson-type shelters. The possible effect of such changes was investigated in [Parker, 1994], who concluded that there was a possible difference between 1900 and the present day of about 0.2°C because of such exposure changes.

So they are ASSUMING the modern screen is superior to (a) old screens and (b) non-screens, and they are ASSUMING the exposure effect is dominated by screen design. But screen design, as A. Watts points out, is only one factor. Aside from all the other things influencing sensor microenvironemnt (e.g. whitewash vs. semigloss latex), the dominant factors may be meso-scale, in between that of the city and that of the sensor screen. The environment within 10-100m of the sensor, say. (asphalt, steel towers, bbqs, ac vents, auto rads, auto exhaust, etc.)

Bottom line: they offer up 0.2°C as the net increase. Seems low to me. Anyone: do I hear 0.3°C? 0.4? Place your guesstimates now. I will collate them along with the A in AGW guesses.

Re #86 Yes, understood. But I think maybe I am more shocked that you are by these revelations. For the first time I’m having a “stop the presses” moment … whereas you’ve been there from the start. It’s one thing for a denialist/skeptic to go around arm-waving about Jones underestimated UHI effects (along with a bunch of other effects), and quite another to have a disinterested party precisely identify and then quantify the nature of the bias. I’ve not been on the UHI bandwagon, but I’ve also not criticized anyone who suggested a better quantification of UHI. Now I include microsite in there as well. So, yes, arms are waving.

Bottom line: they offer up 0.2°C as the net increase. Seems low to me. Anyone: do I hear 0.3°C? 0.4? Place your guesstimates now. I will collate them along with the A in AGW guesses.

How about zero? Folland 2001:

Our estimated 2 extratropical hemispheric uncertainties are 0.2°C before 1900, declining to zero by 1930 because universal standard exposures are assumed and random errors in standard exposures are already included in the optimal averaging.

Using EOFs to take account of spatial inhomogeneity, Shen et al. (1994) developed such an optimal method for the average on the entire globe. They concluded that about 60 well-distributed stations can yield a global average annual mean surface air temperature with an error less than 10% compared with the natural variability.

The systematic errors are assumed to have been removed from the raw data, and hence the remaining random error is uncorrelated with the anomaly field and with errors at other locations:

Note that gavin just mentioned 60 stations. IMO the problem here is that Shen 94 used U.K. dataset to get that number, does the same number apply to global dataset? I don’t know, I find Shen’s papers interesting. On the other hand, Folland 2001 and Brohan 2006 papers are clearly deeply flawed articles, and their publication has damaged the reputation of this promising new branch of science.

How can anyone state, with any degree of confidence, that they know what the world’s temperature is?
The majority of the planet is virtually unmeasured.
Of the fraction that is measured, much is measured too sparsely to have any meaning.
Of the tiny fraction that is measured with enough stations to have any meaning, the stations themselves are of poor quality.

Anyone who declares that with this data, we can tell what the earth’s temperature is, within 2 degrees, much less a tenth of a degree, is delusioanl.

Marysville is now a de facto suburb of Sacramento. The route 65 corridor is one of the fastest growing areas in the state if not the US. It’s right up there with the famous area between I-80 and US-50 to the NE of Sacramento, which was one of David Brooks’ archetypal new exurbs in his book a couple years ago “On Paradise Drive.” In the case of both the 65 corridor as well as Brooks’ archetypal (aptly named) Placer County, a widely sprawling suburbarn landscape with nodes of urban concentration has been superimposed on the rural infrastructure. It’s pretty amazing.

Lincoln, CA – for decades since its Victorian Era founding date, a wide spot along the road from Sacramento to Yuba City – Marysville. Now, it’s the land of Del Webb and Starbucks – Montessori schools and endless Volvos can’t be too far behind:

92: Like you I can’t see how we can rely on these stations to provide any meaningful data. The scary thing is that the US stations are probably amoung the best in the world compared to many others. How this stuff can be used with any confidence is beyond me. It would be like finding out that Lonny T regularly urinates down his boreholes but still claims hundredth of a degree accuracy.

Anthony Watts is doing more CSI, and asked me to post these pictures for him.

Anthony writes:

First site is the Lodi Municipal Service Center, 38.11619N 121.29003W.

Its like “worlds collide” where solar power to offset GHG’s meets the reason for global warming in the first place.

The second site is at the Lodi Fire Station #2, 38.10586N 121.28815W.

I’m not sure which one is USHCN yet as will have to call NWS Sac, but they are both equally bad. To the right of the picture on north side of fire station is the fireman’s BBQ pit wit 3 grills…and like many others, this weather station comes with parking and trash service…

I will have complete set of pictures later

The two stations are about 1/2 mile apart, so lat/long is useless to tell which is USHCN.

Based on a short discussion in another thread, here is what I think could be a very interesting study based on this new work.

Once a large number of temperature stations are located and documented, I think a “test” should be drawn up. I saw someone posted guidelines for setting up a proper temperature station on one of these threads. I would create a score sheet. Assign a score to each requirement for a properly located and set-up temperature station. The scores add up to some number like 100 or 1000. For each documented station, determine whether it meets or fails the criteria, and sum the scores for those which it meets. This gives you an objective overall score for each station, which can be entered into a database. Hopefully these scores would not vary much depending on who is doing the scoring if the criteria are well enough defined, and thus can easily be reproduced and agreed upon.

Similarly, come up with criteria for what defines a rural station – number and size of nearby structures, nearby population density, area of land use changes, etc. Score each station on this test and enter the score into the database also.

Thirdly, I think there should be a homogeneity/data quality test. It could include scores for such things as the number of times a station has moved, how far it moved, how many years contain missing data, etc. Once temperature data and meta-data is available, score each station on data quality.

Ideally this would all be done without actually looking at any of the temperature series. Since these would ideally be objective and deterministic measurements, once the tests are created, anybody can apply them to a station.

Finally, download temperature series for all examined stations. Discard all of those with a station quality score below a certain threshold – ideally 100%. Discard any of these remaining series with a data quality score below a certain threshold. Sort them into a few bins based on their rural/urban score. Then average the series in each bin. Plot them, and compare the trends.

I think the results could be fascinating, and importantly, totally replicable. I’d especially like to see the trend of the average of all rural stations with high scores on station quality and data quality, and compare that to the urban trend.

There may need to be a third team. I don’t know how you could do metadata adjustments without looking at the data from both the station in question, and nearby stations.

If you are going to exclude those stations that need metadata adjustments, I don’t think you will have many stations left.
Even if the station hasn’t been moved, it’s been changed once or twice over the years.

RE: #65 – Every one of those stations is in an area which has experienced massive growth since WW2. In some of them, it has literally gone from rural to heavy suburban during that time period. Even in the obviously urban ones like SF, Oakland and Berkeley, the density has increased and in SF, the City went from mostly low rise buildings everywhere, to a significant high rise financial district in the down town area with much mid rise infill in other areas, much of this since the 1960s. Another significant thing to note affecting the South and East Bay suburbs has been the advent of massive light industrial/R&D/business parks, with 2 – 3 story glass buildings containing data centers, PCs in every office and in some cases, semi conductor processing lines (you want to talk about electrical current!).

Thomas Jefferson and Special Awards
‘€¢ Holm Awards Year Length of Service Awards
‘€¢ 75-100 Year Institutional Length of Service Awards
‘€¢ 50 Year Length of Service Awards
‘€¢ 45 Year Length of Service Awards
‘€¢ 40 Year Length of Service Awards
‘€¢ 35 Year Length of Service Awards
‘€¢ 30 Year Length of Service Awards
‘€¢ 25 Year Length of Service Awards
‘€¢ 20 Year Length of Service Awards
‘€¢ 15 Year Length of Service Awards
‘€¢ 5-10 Year Length of Service Awards.

The above awards are linked to the WWW and are part of the public domain.
Senator Graham, and Jean Carter Johnson, FOIA officer of NOAA, please, note that NOAA has refused to answer FOIA requests to several FOIA submitters (legitimate requests as far as I can determine) based on the fact that NOAA sites and volunteer observers are claimed to be private even though their names and accomplishments have been made public (see above, details are available on climateaudit.org). Senator Graham, the emails to and from Jean.Carter.Johnson@noaa.gov, and other NOAA officials are easily obtainable by you or your staff. Up until approximately a week ago, these names, locations could be accessed by the internet. Positions and names were directly obtainable, accessible, and in the public domain, and available. Now there is the claim that they are private and this information has been removed from the web. There is no such thing as” a little bit pregnant”. Senator Graham as far as I can determine US law and NOAA FOIA guidelines have been violated. Could you and your office help me? Senator Graham, for your information, on a blog, I will ask that all that have been refused cooperation as required by law to contact you. Please, I hope you and your staff don’t mind a citizen of SC for all of his life of almost 54 years asks for this to be resolved according to US law and precedence.

But let’s say that they got all their adjustments exactly right. What does that say about the quantum of UHI? Here’s a town of 12,000 which qualifies as “rural” in all the UHI studies.

Eli objected that Marysville was not “unlit” in the GISS data set and thus did not contribute to GISS trends.

The full quote was:

But let’s say that they got all their adjustments exactly right. What does that say about the quantum of UHI? Here’s a town of 12,000 which qualifies as “rural” in all the UHI studies. The non-climatic effect here is at least 3 deg C. In this case, we can estimate it because there’s a less bad station nearby. What does one do in China or Indonesia where nearly all the stations seem to be in large cities (even if they started out small, they’re large now)?

The “UHI studies” that I had in mind (and had been discussing) were articles like Jones et al 1990, which defines Chinese “rural” stations as having populations under 100,000. Karl et al (J Clim 1988) said in regard to the US system:

Due to the large number of stations located in sparsely populated areas [over 85% (70% of all stations had a 1980 population of less than 25 000 (10 000)], the impact of urbanization is not large in relation to decadal changes of temperature in the United States.

My point here was that, if the GISS adjustment is right, then the required adjustment for Marysville in the GHCN data set used by NOAA, CRU and thus IPCC was up to a couple of deg C – despite claims that the UHI effect was no greater than 0.1 deg C per century. Or if Jones is right, then GISS is over-correcting. The GISS adjustments are not used in CRU or GHCN or USHCN.

RE: #104 – And as Mr. Watt can rightly attest, since 1980, Marysville (and most other places along route 99) have exploded in population. The explosive growth of that exurb made famous in Brook’s “On Paradise Drive” – Placer County – is mimiced by the other communities just to the north, such as Marysville.

In spite of the limitations just alluded to we feel that the magnitude of the sampling error is probably small compared to the other errors in the budget once the number of well-distributed stations is above about 60 and the optimal weighting is applied.

This was based on UK gridded data (Jones). Note that they had some problems with Hansen data,

We note of course that the correlation to the U.K. data at fine scale is not necessarily the same as with perfect data. In fact, we correlated the coefficients of the zeroth harmonics in T11 truncation derived from the U.K. data with the Hansen-Lebedeff data and found the correlation coefficient to be only 0.93! This is presumably because the two groups used different means of removing biases.

In the course of this thread the question arose “what USHCN adjustments are
included in GHCN the adjusted mean file”? Previously, I had thought that the
USHCN FILNET adjustments were included, but upon looking at Marysville station
data, I concluded that SHAP, but not FILNET, adjustments were included.

Today, pursuing a notion of a temporary substitute for the GISS special USHCN
file with SHAP, but not FILNET, adjustments, I must conclude that the GHCN
adjusted mean file sometimes include SHAP, some times FILNET, sometimes both,
and sometimes something other than either.

I expect to write to Peterson and Vose to inquire what USHCN adjustments are
“supposed” to be included in the GHCN adjusted mean file. I expect to mention
examples of GHCN adjusted numbers which do not match any USHCN numbers to within
0.1 C.