Part 2: How the Data Base was Produced

10.0 Known Imperfections within the NSRDB

The procedures and processes described in the previous sections produced a data
base with known quality characteristics, representing significant improvements
over the SOLMET/ERSATZ data base. Nevertheless, as might be expected in a project
of this magnitude, there are known imperfections. It is likely that there also
are unknown imperfections that the data base users will find during the coming
years. Those problems of which we are aware at this time are noted in this
section.

NCDC's quality control procedures detect and flag data that fail internal
consistency checks or exceed preselected climatological limits. When bad data are
detected, edited values that pass all quality checks are often inserted
immediately after the bad data. Sometimes, however, edited data are not inserted.
During the data base production, the last value in the TD-3280 and TD-3210 files
for a given hour and element were always used. Therefore, whenever an edited
value was available it was used. However, when bad data were not followed by an
edited value, the bad data were used.

Although the same interpolation procedures used to replace missing data could
have been used to replace bad data, they were not. When this oversight was
discovered, it was too late to effect a correction with the time and resources
remaining.

Only two instances have been discovered when this problem affected the estimation
of solar radiation data, but it is likely that others exist. Abnormally large
snow depths, 300 to 900 inches, were found during one month and one year for
Kansas City, Missouri, and San Antonio, Texas. These two problems were corrected,
of course, but they led to the realization that similar problems of lesser
magnitude could go undetected. Although bad meteorological data may be infrequent
and may be insignificant in most instances, exceptions undoubtedly exist.

Special means were employed to maintain serially complete files of solar
radiation data when long segments (more than 47 hours) of missing meteorological
data were found. The majority of these situations occurred at stations that were
not operated during the evening or on weekends, but in some instances a station
would be shut down for several weeks or even longer. When these situations
occurred, the gaps in the data were filled with data from other years, for the
same days of the year. This procedure is described in
Section 5.2.1.

Only those elements required to provide input to the METSTAT model were replaced.
These elements include total and opaque sky cover, dry bulb temperature, relative
humidity, and atmospheric pressure (the last three were used to estimate
precipitable water). Because all of the data replacement processes were done one
element at a time, it is possible that different
elements may have been selected from different years. Therefore, it is possible
that this replacement process produced some inconsistencies among the
meteorological elements. The impact on model estimates of solar radiation,
however, should be minor, because total and opaque sky cover were always checked
for consistency, and errors in precipitable water have a relatively small effect
on solar radiation estimates.

To mitigate this problem, the station notes in
Appendix B contain footnotes that
identify the stations for which this replacement method was invoked. Furthermore,
any extended periods of time (greater than 10 days tin a row) that were replaced
in this way have been identified by the beginning and ending dates of the
replacement period. NREL intends to investigate the impact of this replacement
method in more detail and will notify users of their findings.

For some stations and years, the first field position in the present weather data
was inadvertently set to 0 for all hours. The possible presence of a 9 in this
position, indicating missing data or no observations made, was lost. Therefore,
it is possible that reportable weather events did occur, although the data
indicate otherwise.

This did not affect the model estimates of solar radiation because no attempt was
made to replace missing present weather data. Interpolation cannot be used to
estimate the occurrence of discontinuous events such as rainfall. This problem
should be of concern primarily to those using present weather for other
applications.

Some NWS stations limit their operations to daylight hours and, in some
instances, to only part of the day. When data for hours at the beginning and
ending of the sun-up period were missing for these stations, the meteorological
elements needed for model estimates of solar radiation were filled-in through an
extrapolation process, but only to sunrise and/or sunset. No reasonable method of
filling in nighttime data could be devised. Therefore, users who are concerned
with nighttime conditions will have to avoid these stations or devise their own
replacement scheme.

We should also note that no replacement of missing data was attempted for
discontinuous elements or elements not required for model estimates of solar
radiation. This includes dew-point temperature, wind direction and speed,
horizontal visibility, ceiling height, present weather, snow depth, and number of
days since last snowfall.

As described in Section 5.2.3, aerosol
optical depths were estimated from direct
normal solar radiation measurements. Direct normal data were especially sparse
for three regions: (1) most
of Alaska; (2) all island locations such as Hawaii, Guam, and Puerto Rico; and
(3) the industrial heartland of the United States from Chicago to New York City.

In the states of Illinois, Indiana, Michigan, Ohio, Pennsylvania, and New York,
direct normal data during the period from 1961 to 1990 were available, at NREL
and NCDC, only for Pittsburgh, Pennsylvania and Albany, New York. Furthermore,
the data from Pittsburgh were sparse and of questionable quality.

Therefore, the model estimates of direct normal and diffuse horizontal solar
radiation for these regions must be considered as suspect, perhaps even more
suspect than is indicated by the assigned uncertainty values. Fortunately, errors
in aerosol optical depth have a relatively small effect on global horizontal
values.

From January 1981 through October 1985, the data collected by NOAA-NWS was not
processed or quality controlled. Although NREL attempted to effect a quality
assessment of this data during the data base production, it is known that some
defects are difficult to detect. One problem of particular concern are errors in
time. Large errors in time, resulting in shifts of daytime data to nighttime
hours, were easy to detect. However, we know that smaller shifts in time, of one
or two hours or even partial hours, could go undetected by our quality assessment
software.

NREL's quality assessment software detected measured data with gross errors for
all years. Nevertheless, users are advised to treat measured solar radiation data
from NWS stations for the period from 1981 to 1985 with extra caution.

From an examination of the quality flags for some of the 26 SOLMET stations, the
user will find modeled global horizontal data during periods of time when
measured data may have been expected. This is particularly true for stations in
the western United States from about 1968 to 1975. Although the serial plots of
15-day averages (solar noon values) found in
SOLMET Vol. 2 (1979) indicate that
data were available, the extraction of hourly values from the strip chart records
was sometimes not completed. The station notes in
SOLMET Vol. 1 (1978) identify
the periods of time when the global horizontal data were modeled. The global
horizontal data in the SOLMET/ERSATZ data base for these stations and years were
modeled.

During the production of the NSRDB, the SOLMET modeled data were replaced with
estimates using the METSTAT model. More specifically, whenever temperature
corrected global horizontal data were not found in field 109 of the SOLMET/ERSATZ
data records, the value was estimated with the METSTAT model.

We and other NSRDB users may discover other problems. As we learn of these, we
will have addendum sheets inserted into this manual to alert users to these
problems. Therefore, we encourage NSRDB users to notify NREL or NCDC of any
problems or questionable data that are discovered. This will also give us the
opportunity to correct these shortcomings in future upgrades of the data base.