1 – Abstract – Knowing the accuracy and representativeness of new observing systems is critical for their optimal use in data assimilation and other applications. One such new system is the laser-diode based Water Vapor Sensing System (WVSS-II) currently being deployed and tested on commercial aircraft in the US and, to a more limited degree, in Europe. The paper discusses the results of recent WVSS-II to Rawinsonde intercomparison studies, not only in terms of assessing the accuracy of the humidity data, but also for determining how best to use these data as a supplement to other upper-air moisture measurements and as background for using the data as a source of asynoptic observations to validate/calibrate a variety of satellite moisture products.

2 – Background – In the late 1990s, the North American Observing System (NAOS) study group performed an evaluation of the value of automated aircraft reports relative to and as a supplement for rawinsonde observations. The results of that study clearly identified the need to include moisture data in addition to the temperature and wind reports included in the aircraft. The prime objective for obtaining atmospheric moisture profiles from commercial aircraft is to provide sufficient numbers of high-resolution (both spatial and temporal) moisture data to fill the gaps left between the 400 km and 12 hr spacing of the national rawinsonde network; the only other source of in-situ moisture observations. Other benefits will include improved monitoring and prediction of cirrus clouds by obtaining more frequent observations of high-tropospheric moisture and a low-cost means of validating/calibrating satellite sounding moisture profiles and assimilation systems.

In an early attempt to obtain these types of observations, a set of approximately 30 instruments of the first generation Water Vapor Sensing System (WVSS-I) have been operating aboard United Parcel Service (UPS) aircraft for approximately five years. These humidity data never saw wide use in objective forecasting, due both to limited availability and variable accuracy do to engineering limitations. A second generation WVSS-II has been developed and flown on UPS aircraft and have already undergone several tests beginning in 2005. Validation tests pointed to the need for a number of engineering modifications, most importantly improvements to observations during aircraft descent and system reliability, which were completed in 2008.

The re-engineered version has been subject to climate chamber testing at NOAA/NWS (USA) and DWD as well at the Research Centre Juelich (Germany). Upgrades to existing USA fleet of 25 WVSS-II sensors on United Parcel Service (UPS) Boeing B757 aircraft have been completed and 31 new WVSS-II units are being installed on 31 Southwest Airlines Boeing B737 aircraft. Presently in Europe up to 15 units are planned to be installed on a number of E-AMDAR participating aircraft.

2.1 – Summary of previous Test Chamber Evaluations - Several simulation runs were conducted in the DWD chamber whereby the water vapor Mixing Ratio (MR) was varied from 30 g/kg downwards to 0.001 g/kg while pressure and temperature were adjusted from 1000 hPa and 300 K through 200 hPa and 200 K, typically for real atmospheric conditions between surface and 12 km altitude. Figure 1 shows that the results are near the ideal line. Even at an extremely low MR value of 0.003 g/kg the WVSS-II accurate and precise measurements. Because of some actual limits for very low pressure operation of the chamber these values were obtained in the range of 800 to 1050 hPa. The detection limit seems to be
lower than that of the previous version (having been at about 0.05 g/kg at ground pressure). With the MR values actually being reproducible by the DWD climate chamber the lower sensitivity bound of the WVSS-II unit S/N 0302 could not be reached.

Additional chamber experiments show that WVSS-II tracks humidity structures very well, whereby the performance at low values of Relative Humidity (RH) as well as at almost saturated conditions is virtually the same. A comparison of the WVSS-II versus the reference hygrometers is shown in figure 2. At MR values between 0.05 g/kg (≈ 80 ppmv) and 20 g/kg (≈ 30,000 ppmv) the WVSS-II performs well with a relative uncertainty better than ± (5 – 10) %. However, at Specific Humidity (SH) values below
0.05 g/kg (≈ 80 ppmv) the deviations of the WVSS-II compared to the Lyman (α) are getting larger and reaching the detection limit at about 0.02 g/kg (≈ 30 ppmv) at air pressure of 200 hPa.

The chamber experiments have shown that the WVSS-II performs well with a relative uncertainty of ± (5 – 10) %. For humidity levels between 20 g/kg (≈ 30,000 ppmv) and 0.05 g/kg (≈ 80 ppmv) typical for the lower and middle troposphere the performance is good with relative accuracy of WVSS-II is ± (5 – 10) %. In upper tropospheric conditions where MR values fall well below 0.05 g/kg, the accuracy of WVSS-II is reduced to the detection limit of about 0.02 g/kg.

3 – 2009–2010 Field Evaluation – The Cooperative Institute for Meteorological Satellite Studies (CIMSS) University of Wisconsin–Madison (UW) has performed intercomparison analyses of the co–located rawinsonde and WVSS-II data taken on evenings during fall 2009, spring 2010 and summer 2010 at Rockford IL (RFD). The intent was to assess the general performance of the systems and to provide statistical evidence of the accuracy of the re-engineered WVSS-II system.

As in analyses of previous inter-comparison tests conducted and reported by CIMSS, the rawinsonde observations were taken using Vaisala model RS–92 instruments with new humidity sensors. The rawinsonde observations were made at approximately 3-hourly intervals, immediately before periods when UPS–757 aircraft equipped with the WVSS-II instruments landed, immediately after aircraft departed, and ½ way between these two times. The WVSS-II data were obtained from the NOAA/ESRL MADIS database.

The analysis shown here was done using all available data from all aircraft, excluding those with known engineering failures. This is different from previous inter-comparisons, which had purposefully been limited to analyze only 1) WVSS-II data taken in ascent (to eliminate systems problems evident during descent), 2) WVSS-II Specific Humidity (SH) values greater than 2 g/kg (to eliminate low-level ambient systems moisture contamination) and 3) observed WVSS-II SH values <10 g/kg (to eliminate a data encoding error). It should be noted that all comparison shown here were obtained between the WVSS-II observations and the single closest rawinsonde reports, without time interpolation between the bounding validation data.

3.1 – Results – The discussion here will first assess characteristics of specific sets of observations made during the WVSS-II data collection period, followed by an overall systematic statistical evaluation.

3.1.1 – Direct Data Intercomparison – Subjective comparisons between the aircraft and rawinsonde observations provide a direct means of comparing the data from WVSS-II equipped aircraft to individual rawinsonde reports. As exemplified by figure 3, the majority of both ascent (solid) and descent (dashed) reports showed very similar vertical structures to the rawinsonde data. This was especially true in the lowest 300 hPa of the soundings where all aircraft were flying essentially along the same paths and close to the rawinsonde launch site. The agreement between the successive (and independent) WVSS-II reports provides confidence in the accuracy of and consistency between the individual WVSS-II observations. Aloft, the degree of the "fit" of the WVSS-II profiles to the rawinsonde data showed more variability — a portion of which is probably related to small-scale moisture variability and/or clouds in the area around the rawinsonde path.

In figure 3-top, the WVSS-II moisture data fits to rawinsonde data were within ±0.5 g/kg, including across the moisture inversion. In many ascent cases, however, the temperature profiles showed notable warm Biases from most aircraft. Although ascending reports dominated the co-located data sets, the fit and consistency between descending reports was also very good, as shown in figure 3-bottom, including across inversions in this very moist environment. Unlike the ascending observations, the descent moisture data in general show a slight moist Bias and a smaller warm Bias. In both examples, most of the WVSS-II observations fell well within the bounding rawinsonde reports. It should be noted that the over-estimate of SH in the WVSS-II descent data near the melting level in some of the fall 2009 data was not apparent in other observations and that although the moisture profiles in the bottom-left panel agree very closely, the slight misalignment of observing levels between the WVSS-II observations and the validating rawinsonde data led to a large +/- error couplet across the inversion. Differences like this, especially in high moisture situations, can have large impact on the overall statistical evaluation presented later.

Figure 3 - Top Left - Plots of co-located ascending WVSS-II observations (colored) and rawinsonde data (black) taken around 0827 UTC on 30 April 2010 at Rockford IL. Temperature (C) at left, Specific Humidity (g/kg) at right, solid lines during descent, dashed lines during ascent. Changes between bounding rawinsondes shaded. Top Right - Differences between rawinsonde and ascending aircraft observations using same color convention as top panel. Shading indicates ±50% of hourly changes observed between bounding rawinsonde reports. Bottom – Same as top panels, except for descending WVSS-II observations taken around 0310 UTC on 24 August 2010.

Another way of assessing the quality of the limited number of WVSS-II reports is through scatter plots comparing the full set of co-located WVSS-II to rawinsonde observations. These results again show data for all levels, all observation periods and both ascents and descents. Several characteristics of the WVSS-II data become readily apparent from the ascent and descent matchups shown in figure 4. Overall, the more numerous ascent data show a smaller spread than the descent reports. The fits of the ascent data are extremely good for the middle and upper moisture range (> 6 g/kg), although there are a number of outliers from 2 specific aircraft at larger time differences in the 9–12 g/kg validation range. Even though the high specific humidity reports were obtained near the surface and more temporally separated from the
verifying rawinsonde reports, all aircraft that made these higher value reports agreed very closely with each other and with the validation data set. For values less than 6 g/kg, the spread between the WVSS-II and rawinsonde data increases, even as the time differences decrease. This is particularly apparent in the right panel showing spring-time results only and is partly the result of increasing separation distance between the two observation points. Overall, the ascent data exhibit a positive Bias of 0.19 g/kg (systematic difference), with a Standard Deviation (StDev − random difference) of 0.91 g/kg. In middle/lower moisture ranges (left panel), the Bias increases slightly to .21 g/kg, while the StDev decreases to .56 g/kg, due in part to the elimination of the previously mentioned outliers.

Figure 4 - Scatter plot showing inter-comparison of WVSS-II and co-located rawinsonde Specific Humidity reports for all levels for all observation intercomparison periods (left) and period from 27 April-10 May 2010 at Rockford, IL. . Color coded by aircraft. Marker size indicates time spread between WVSS-II and rawinsonde observations – larger markers indicate matchups that are closer in time. Ascents presented as solid squares, descents by open squares. Statistics included in lower right.

The Specific Humidity data from descending aircraft show a slightly smaller Bias (+0.09 g/kg) and random difference (StDev of 0.85 g/kg), compared with +0.25 g/kg Bias and 0.91 g/kg StDev for ascending reports. The flight that descended into the moistest conditions showed slight moist biases relative to the co–located rawinsonde observation, while the ascending flights showed better fits. The fact that multiple aircraft showed these same behaviors points both to good consistency between the aircraft reports and to the possibility that the rawinsonde data taken at 3 hourly intervals may not have been fully representative of the small–scale moisture structures observed by the WVSS-II aircraft.

The WVSS-II and rawinsonde SH reports displayed only small variations across the full range of rawinsonde RH observations, as shown in figure 5. The WVSS-II data show small SH StDev at lower RH ranges and smaller Biases at higher rawinsonde RH values, especially during descent. In the ascent data (left panel), random differences remain near 0.5 g/kg below 50% RH, then increase to 0.9 g/kg for higher
amounts, and then return to 0.5 g/kg at RH values above 85%. The larger StDev at lower RH (which usually occurred at higher altitudes) in the descending data are due in part to the larger special separation of the WVSS-II and rawinsonde observations. The higher StDev above 85% in the descending data may reflect the potential inability of the rawinsondes to detect saturated conditions.

Figure 6 provides an example of the differences seen between an individual WVSS-II equipped aircraft and rawinsonde data throughout the spring 2010 data collection period. Subjective assessment shows a consistent warm Bias in the temperature reports at all but the lowest levels, although descending data from this aircraft appeared to have smaller warm Biases. The spread of the temperature difference profiles showed random variations at all levels. By contrast, the SH data show almost no Bias and very small random differences in the lowest 200 hPa of the profiles. Larger random differences are seen immediately above 800 hPa in several ascents and above 650 hPa in some of both the ascent and descent reports, perhaps a reflection of the greater distance of higher-altitude WVSS-II reports from the validating rawinsonde observations.

3.1.2 – Statistical Assessment – Summary statistics were calculated over the entire test period comparing all aircraft observations (ascent and descent) to the nearest rawinsonde report. For these calculations, an additional set of criteria were used to limit evaluations to cases with small variability between the rawinsonde data surrounding the WVSS-II observations. This was done by limiting the temporal difference in RH between successive reports to be <7% and vertical differences between adjacent vertical levels to <10%. The effect of this was to eliminate both the effects of scattered clouds that could have been along the rawinsonde trajectory and to eliminate cases where shallow banks of moisture and fronts were moving vertical during the test period.

The best agreement in figure 7 appears in the lowest 250 hPa, where RH Biases are very small and random differences are on the order of 5-7%. Above 875 hPa, the random error increases to near than 10% and a slight positive Bias forms. These differences decrease again to near-surface values above 700 hPa, but with a second spike in both Bias and StDev differences near 600 hPa. As shown on the right panel, the upper level spike occurs in a region where the sample size of co-location matches is decreasing and the mean separation distance between the aircraft and rawinsondes is increasing. Similarly, the mean separation distance between observations more than doubles between the zone of good low-level agreement below 900 hPa and the region of increased differences above 700 hPa. The anomalous increase in both Bias and StDev above 600 may be due to unpresentativeness within all sample size at this level.

As in past tests, temperature comparison statistics obtained for the same cases (figure 8-left) show quite large Biases, even for observations that were separated by less that 10 km. (It should be noted that the aircraft temperature observing system is independent of the WVSS-II hardware.) Although the Bias is small near the surface, it increases rapidly to 0.5°C and greater between 925 and 850 hPa. Above that level, the Bias deceases to about 0.3°C at other levels. These differences are consistent with but slightly larger than the temperature Biases seen with the same aircraft in previous tests. Because the data set was dominated by reports taken during ascent (a situation where the aircraft would be flying from areas of warmer air into areas of colder air), hysteresis could be a factor contributing to these large Biases. By contrast, the random variability (StDev) between the aircraft and rawinsonde observations is greater than 1°C below 950 hPa and decreases to around 0.8°C above that level.

When the effects of the temperature Biases are removed from the RH data by using rawinsonde temperature in both the WVSS-II and rawinsonde calculations (figure 8-right panel), the compensating effects of the temperature and moisture observation Biases on the RH statistics becomes evident. Most importantly, the Random Error (StDev) remains essentially unchanged. The fact that the Random Errors obtained from the WVSS-II-to-rawinsonde matchups during the 1 hour comparison window are near the scaled hourly differences observed in the bounding rawinsonde reports indicates that the variation detected in the WVSS-II data are in a large part accounted for by the large-scale variability observed during the assessment periods. Although the Biases increases from negligible near the surface to exceed ±5% at several higher levels, this should not present a major problem in applications since the effect of these systematic errors can be removed if the WVSS-II aircraft data (both SH and temperature) are monitored regularly.

Figure 8 - Left - Same as left panel of Figure 5, but for Temperature. Right - Same as left panel, except for RH calculated using WVSS-II moisture data and rawinsonde temperature data as a means of determining the exclusive effects of WVSS-II moisture observation on RH differences.

Profile of differences in SH (the variable reported by the WVSS-II instrument) are shown in Fig. 9. These results also show positive Biases at all levels, ranging from 0.1 to 0.3g/kg, except near 600 hPa, where small negative biases are noted. Again, this systematic behavior of the instrument can readily be removed and modified over time if the instrument performance is monitored against a calibrated standard on a regular basis.

Figure 9 - Same as Fig. 5, expect for Specific Humidity.

More importantly, however, the random difference between the WVSS-II and rawinsonde moisture observations ranges between 0.2 and 0.7 g/kg at the majority of levels. Compared with previous evaluations, the larger StDev near the surface is due almost entirely to the observations made in moister environments that were included in these test, situations where the natural variability (not observational error) is larger than in drier conditions. The increases aloft occur where the number of intercomparison and average time/space separation between the 2 observing systems increases. This performance is well with WMO standards for both global and mesoscale applications.

As another measure of the robustness of the WVSS-II observation, intercomparisons were computed between WVSS-II observation pairs made within specific time, height and spatial limits. The results shown in Fig. 10 indicate not only that the variability increases systematically with both time and space separation, but more importantly, that WVSS-II observations taken within 15 minutes, 20 km and 55 m altitude of each other agreed to within better than 0.2 g/kg.

Figure 10 - Variability (RMS) of Specific Humidity observations between nearby WVSS-II observations by time difference and range interval for all levels from the surface to 5km throughout the full intercomparison period.

Even using the longer 1 hour and 60 km limits (periods comparable to those used in the rawinsonde-to-WVSS-II comparisons), the variability is <0.4 g/kg. The fact that this figure is much lower than that obtained in the rawinsonde-to-WVSS-II comparisons, indicates that a substantial portion of the difference detected in the multiple instrument intercomparisons may have been due to errors and unrepresenativeness in the validating rawinsonde reports.

Summary: When the large match-up differences that are probably due to small-scale natural variability are excluded, the re-engineered WVSS-II systems appear to meet WMO observing requirements across all SH and RH ranges and in both ascent and descent.

Overall, the WVSS-II SH observations match the rawinsonde data very closely, with random differences ranging primarily from 0.2 to 0.7 g/kg, well within WMO recommendations. Although the data show a slight moist Bias (ranging from 0.1 to 0.3 g/kg), the Bias should be correctable in ground-processing if the WVSS-II data are monitored regularly. When the WVSS-II SH data are converted to RH and the impact of aircraft temperature biases are removed, the RH statistics show Biases that range from negligible near the surface to exceed ±5% at several higher levels and Standard Deviations generally
between 5–10%. Intercomparison within the WVSS-II data set further validate these results, with WVSS-II observations made within 15 minutes, 60 km distance and 55 m altitude showing variability of less than 0.2 g/kg. This agreement between independent observations of moisture exceeds that of most, if not all, other operation data sets.

The engineering changes made to the WVSS-II since 2008 seem to have alleviated the problems with data taken during descent observed in earlier tests. Likewise, the data encoding problem has been resolved and observations in environments of low SH (<2 g/kg) seem to match much more closely than previous tests showed.

4.1 - Additional Work Underway: - Supplementary analyses are underway to further enhance and clarify the results presented here. Among others, these are planned to include:

Removing the restriction of comparing the WVSS-II data only with the nearest-time rawinsonde report. Instead, the two rawinsonde reports bounding the WVSS-II observations will be temporally and spatially interpolated to the time of the WVSS-II observations. This should remove any Biases that are introduced by the fact that all of the aircraft observations made during descent occurred in early evening after the verifying rawinsonde and all ascending data were collected during the middle of the night before the verifying rawinsonde reports, as well as accounting for large scale, systematic advection processes that could bias the comparisons.

Adjusting distance criteria used in filtering the WVSS-II data to be more consistent with the parcel displacement distance expected from the mean winds observed during the test period.

Adding wind observation variability limits to the statistical analysis process to provide an independent standard for identifying periods of low and high natural atmospheric variability.

Dover, J.M, and Childs, B., 2004: A new low maintenance dew point sensor for the National Weather Service Automated Surface Observing System. 12th Symposium on Meteorological Observations and Instrumentation, paper 14.3, January 2004.