Alaska Ceiling and Visibility Analysis (CVA-AK) Product

Project Tabs - CVA-AK

Overview

Poor weather conditions, particularly restricted visibility and low cloud tops, were the leading cause of fatal general aviation (GA) accidents in Alaska from 2001-2012. Traditional weather observations from Alaska’s widely dispersed airfields inadequately forewarn of weather likely to be encountered along routes between stations or, in particular, through hazardous mountain passes with localized conditions. In 2014, the National Transportation Safety Board (NTSB) included “GENERAL AVIATION: IDENTIFY and COMMUNICATE HAZARDOUS WEATHER” on its Most Wanted List to improve transportation safety.

The NCAR/RAL, the MIT/LL, the Alaska Aviation Weather Unit (AAWU), and the FAA are involved in a three year collaborative effort to produce a rapidly-updated, high resolution, gridded product of ceiling and visibility (C&V) conditions across Alaska. This product, known as the Ceiling and Visibility Analysis – Alaska (CVA-AK) serves as a “first guess” estimate of C&V conditions across Alaska at or near instrumented and non-instrumented airfields and along data-sparse routes between airfields including treacherous and heavily-traveled mountain passes.

Visible satellite image for Alaska on 15 August 2016. METAR station reports for ceiling (cyan), visibility (magenta) and current weather (yellow) are overlaid.

The CVA-AK product combines ceiling and visibility information from the latest NCEP RAP model with METAR observations of C & V using data fusion techniques to produce Flight Category, Ceiling and Visibility gridded fields. These fields are updated every 20 min and hourly analysis products are viewable by AAWU forecasters on the IC4D display system that they use to produce their aviation forecasts.

The latest version of the CVA-AK product includes geostationary observations. Future versions will also include polar orbiter satellite observations and visibility estimates retrieved from the FAA web camera imagery collected in Alaska and output from the Alaska HRRR model, when it becomes available.

An initial version of the CVA-AK was installed at the AAWU in May 2016. The 2 h forecast of ceiling and visibility from the 11 km RAP model forms the base layer. METAR ceiling and visibility observations are ingested and compared to the RAP values at those sites, showing agreement or disagreement with the model. CVA-AK products are interpolated to a 6 km AK National Digital Forecast Database (NDFD) grid. Ceiling, visibility and flight category products are updated every 20 min with most recent METAR reports, but only the output on the hour are currently displayed on the AAWU forecasters’ IC4D display system.

Updated Version 1 and Version 2

Ceiling difference between CVA-AK version 2 (with satellite cloud mask) and version 1.

CVA-AK Version 2.0 was installed at the AAWU in October 2017. This version of the product includes the real-time calibration technique, and the integration of a GOES satellite cloud mask (Jedlovec et al) into the product. The Jedlovec algorithm uses GOES 3.9 and 11 micron channels to identify clear and cloud areas at every pixel and to create the cloud mask. This information is then applied over the RAP/METAR analysis and is used to “clear” the ceiling only. This mask is not applied to the visibility field. A preliminary comparison was done for a 30-day initial period (April 18-May 17, 2017), and used a grid-to-grid match of the ceiling values. These results show how the incorporation of the satellite cloud mask increases ceilings in version 2 over version 1, particularly over the oceans, as can be seen in Fig. X. A few terrestrial locations also show some large differences, such as between METAR stations on the north slope of Alaska. When METAR observations of ceiling are available, these override the other data sources in the product. Thus, the small differences seen within the METAR ‘circles’ occur only when METAR ceiling reports are not available (e.g. missing). The most significant effect of inclusion of the mask into the product is over the maritime regions and off-shore coastal areas where there are no METAR observations. The mask helps to constrain errors in the model over these area where there are no surface observations. Research into incorporating a cloud mask from the polar orbiting satellites (POES) is ongoing.

Version 2 - Satellite products will be included into version 2 of the CVA-AK product using the GOES-West, GOES-R, and Himawari (as available) datasets

The Jedlovec cloud mask will be run on GOES data to identify cloud/no cloud grid points and this information will be included in the blended CVA-AK product. Polar orbiter data from the Suomi NPP VIIRS and the AVHRR are being ingested for Alaska and will be evaluated for inclusion in the CVA-AK. The RAP model will be replaced with the Alaska HRRR model when it becomes available.

Version 3 –FAA web camera observations will be incorporated into the CVA-AK to improve the product

Top Panel, FAA Web camera locations in Alaska. Red shaded polygons highlight camera imagery showing reduced visibility or reduced cloud ceilings on 4 October 2016. Bottom Panel, web camera imagery to the southwest on 4 October 2016 from McKinley North site. Imagery on a clear day and camera viewing angles are also shown.

MIT/LL is in the process of refining an algorithm to retrieve visibility estimates at selected camera locations. Ultimately, this algorithm will be scaled to process visibility retrievals at all of the Alaska web camera sites and this input will be included into version 3 of the CVA-AK.

Model Calibration

Data flow diagram for calibration technique.

Understanding the level of consistency between surface-based observations and RAP forecasts of C&V is critical for developing a useful analysis product. The model provides much greater detail in terms of resolving coastal and terrain driven gradients in the frequency of low ceilings, but notable biases in the model predictions are also evident. These differences become more clear when looking at frequency difference (modeled minus observed frequency) plots. These comparisons can be used to remove first order bias in the model and to better understand the representativeness of the surface observation measurements and how this information should be spread in the gridded C&V analysis product.

Two new applications were developed for producing model calibration files in real-time. The data flow diagram that shows how these two new applications interface with CVmodelCal is shown below. The application called ObsClimoFreq reads in the METAR station data for a configurable number of days and calculates the frequency of occurrence for the requested fields (ceiling, visibility) using a configurable list of thresholds. The output is a comma-separated ascii file of frequencies for each METAR station along with latitude and longitude. The application called FrequencyMatch reads in this ascii file along with the gridded RAP model fields (ceiling and visibility) for a configurable number of days. From the model data, it calculates the frequency of occurrence using the same set of thresholds that were calculated for the METAR stations at each of the METAR station locations. The RAP model and METAR frequencies are then compared to determine if calibration is necessary. If deemed necessary, the threshold used to compute the RAP model frequency is adjusted until the model frequency matches the METAR frequency as closely as possible. Several simple data quality checks are performed along with checks to ensure that an adequate amount of data was available to perform these comparisons. FrequencyMatch then uses a circular filter to place these optimized thresholds onto a RAP grid. For grid points where multiple METAR stations have influence, we perform a distance-weighted average to determine the optimized threshold value.

Example of the ceiling calibration values for a single level (2.5 km) as obtained on 12 October 2017.

An example of the calibration data for a single level (2.5 km) is shown in the figure below. In this example, adjustments are evident as the non-green areas. For example, the RAP model had a consistent over-forecasting bias in the vicinity of Anchorage. The resulting adjustment is to increase the threshold value used to identify ceilings of 2.5 km or less. This is done in order to reduce the frequency of occurrence at this ceiling height threshold. Note that values are converted to the requisite units of kFT for display and user evaluation.

These two applications take the place of a time intensive manual offline process that had previously only been done intermittently. The new procedure runs once per day, thus providing the capability of the system to reliably respond to changes in model bias. As shown in the diagram above, the output of FrequencyMatch feeds into the previously developed application called CVmodelCal which reads in the model data along with the ceiling and visibility calibration files and adjusts the model ceiling and visibility values in areas that are consistently out of line with the observations. The real-time calibration process has been included in the CVA-AK Version 2.0 products.

Model-METAR Errors

RAP model ceiling and visibility estimates are compared to METAR measurements collected every 20 min at Alaska METAR sites. Daily and weekly error plots (offsets between RAP and METAR values) of ceiling and visibility estimates are being generated automatically. The offset values are calculated as Model minus METARS. Negative values show where the model is under estimating the ceiling and/or visibility. Daily plots and weekly plots are produced daily and can be accessed at the following links. Weekly plots contain that day and the 6 days prior to it.

For reference, the figures above show the locations and the 3 and 4 letter identifiers for the Alaska METAR stations. (click images to enlarge)

Clicking on the link for the daily plots will pop up the menu, shown in the top panel to the left. Selecting day 20160927 pops up the middle panel for selection of a specific METAR station. Plotted in the bottom panel are differences (offsets) between Model and METAR values at a particular METAR location.

Threshold values are used to color code the station names. Station names are green if at least 70% of the values are within the thresholds and red if its less than 70%. The current thresholds are 10,000 ft for ceiling height and 5 nautical miles for visibility. These values are also marked on the plots (along with the 0 line) as colored dashed lines. Station names will be black if both the ceiling and visibility data has fewer than 10% of the expected data points or if all the data points are zero. A light gray is used if only one of the two fields has this condition. So the black and gray colors are meant to show plots that have data issues.

Product Evaluation

A thorough evaluation of CVA-AK Version 1.0 was conducted using a cross-validation approach where a set of 25 METAR stations were withheld from the product and used as an independent truth dataset to evaluate the accuracy of the product. This cross-validation was done in post-analysis mode, not real-time. Data from the nine-month evaluation period were analyzed and summarized in a technical report. Information presented included seasonal and regional assessments of performance of ceiling and visibility individually, as well as flight category. An example of performance broken down by regional statistics is shown in the figure, for the CVA-AK flight category product.

Boxplots of CVA-AK Ceiling by METAR observed ceiling category for each region.

The evaluation showed that the CVA-AK has skill at diagnosing aviation hazards. Regional differences in performance are being investigated (e.g., Anchorage; Region 2). Detailed analyses showed that the complex terrain of Alaska affects the C&V performance in the model. Use of a higher resolution model such as the HRRR could help improved performance in this regard. Seasonal differences in performance are pronounced for ceiling. Visibility has consistent performance across the seasons but overall is less accurate than ceiling performance.

Changes were made to address the forecasters’ concerns and to correct other problems identified with the products. These changes included making using of the NDFD-based terrain data, using the RAP model terrain grid for ceiling AGL computations and allowing the METAR station observations to override the RAP model values when METAR observations show worse weather conditions.

Projects

Overview

Overview

Poor weather conditions, particularly restricted visibility and low cloud tops, were the leading cause of fatal general aviation (GA) accidents in Alaska from 2001-2012. Traditional weather observations from Alaska’s widely dispersed airfields inadequately forewarn of weather likely to be encountered along routes between stations or, in particular, through hazardous mountain passes with localized conditions. In 2014, the National Transportation Safety Board (NTSB) included “GENERAL AVIATION: IDENTIFY and COMMUNICATE HAZARDOUS WEATHER” on its Most Wanted List to improve transportation safety.

The NCAR/RAL, the MIT/LL, the Alaska Aviation Weather Unit (AAWU), and the FAA are involved in a three year collaborative effort to produce a rapidly-updated, high resolution, gridded product of ceiling and visibility (C&V) conditions across Alaska. This product, known as the Ceiling and Visibility Analysis – Alaska (CVA-AK) serves as a “first guess” estimate of C&V conditions across Alaska at or near instrumented and non-instrumented airfields and along data-sparse routes between airfields including treacherous and heavily-traveled mountain passes.

Visible satellite image for Alaska on 15 August 2016. METAR station reports for ceiling (cyan), visibility (magenta) and current weather (yellow) are overlaid.

The CVA-AK product combines ceiling and visibility information from the latest NCEP RAP model with METAR observations of C & V using data fusion techniques to produce Flight Category, Ceiling and Visibility gridded fields. These fields are updated every 20 min and hourly analysis products are viewable by AAWU forecasters on the IC4D display system that they use to produce their aviation forecasts.

The latest version of the CVA-AK product includes geostationary observations. Future versions will also include polar orbiter satellite observations and visibility estimates retrieved from the FAA web camera imagery collected in Alaska and output from the Alaska HRRR model, when it becomes available.

An initial version of the CVA-AK was installed at the AAWU in May 2016. The 2 h forecast of ceiling and visibility from the 11 km RAP model forms the base layer. METAR ceiling and visibility observations are ingested and compared to the RAP values at those sites, showing agreement or disagreement with the model. CVA-AK products are interpolated to a 6 km AK National Digital Forecast Database (NDFD) grid. Ceiling, visibility and flight category products are updated every 20 min with most recent METAR reports, but only the output on the hour are currently displayed on the AAWU forecasters’ IC4D display system.

Updated Version 1 and Version 2

Ceiling difference between CVA-AK version 2 (with satellite cloud mask) and version 1.

CVA-AK Version 2.0 was installed at the AAWU in October 2017. This version of the product includes the real-time calibration technique, and the integration of a GOES satellite cloud mask (Jedlovec et al) into the product. The Jedlovec algorithm uses GOES 3.9 and 11 micron channels to identify clear and cloud areas at every pixel and to create the cloud mask. This information is then applied over the RAP/METAR analysis and is used to “clear” the ceiling only. This mask is not applied to the visibility field. A preliminary comparison was done for a 30-day initial period (April 18-May 17, 2017), and used a grid-to-grid match of the ceiling values. These results show how the incorporation of the satellite cloud mask increases ceilings in version 2 over version 1, particularly over the oceans, as can be seen in Fig. X. A few terrestrial locations also show some large differences, such as between METAR stations on the north slope of Alaska. When METAR observations of ceiling are available, these override the other data sources in the product. Thus, the small differences seen within the METAR ‘circles’ occur only when METAR ceiling reports are not available (e.g. missing). The most significant effect of inclusion of the mask into the product is over the maritime regions and off-shore coastal areas where there are no METAR observations. The mask helps to constrain errors in the model over these area where there are no surface observations. Research into incorporating a cloud mask from the polar orbiting satellites (POES) is ongoing.

Version 2 - Satellite products will be included into version 2 of the CVA-AK product using the GOES-West, GOES-R, and Himawari (as available) datasets

The Jedlovec cloud mask will be run on GOES data to identify cloud/no cloud grid points and this information will be included in the blended CVA-AK product. Polar orbiter data from the Suomi NPP VIIRS and the AVHRR are being ingested for Alaska and will be evaluated for inclusion in the CVA-AK. The RAP model will be replaced with the Alaska HRRR model when it becomes available.

Version 3 –FAA web camera observations will be incorporated into the CVA-AK to improve the product

Top Panel, FAA Web camera locations in Alaska. Red shaded polygons highlight camera imagery showing reduced visibility or reduced cloud ceilings on 4 October 2016. Bottom Panel, web camera imagery to the southwest on 4 October 2016 from McKinley North site. Imagery on a clear day and camera viewing angles are also shown.

MIT/LL is in the process of refining an algorithm to retrieve visibility estimates at selected camera locations. Ultimately, this algorithm will be scaled to process visibility retrievals at all of the Alaska web camera sites and this input will be included into version 3 of the CVA-AK.

Technology

Model Calibration

Data flow diagram for calibration technique.

Understanding the level of consistency between surface-based observations and RAP forecasts of C&V is critical for developing a useful analysis product. The model provides much greater detail in terms of resolving coastal and terrain driven gradients in the frequency of low ceilings, but notable biases in the model predictions are also evident. These differences become more clear when looking at frequency difference (modeled minus observed frequency) plots. These comparisons can be used to remove first order bias in the model and to better understand the representativeness of the surface observation measurements and how this information should be spread in the gridded C&V analysis product.

Two new applications were developed for producing model calibration files in real-time. The data flow diagram that shows how these two new applications interface with CVmodelCal is shown below. The application called ObsClimoFreq reads in the METAR station data for a configurable number of days and calculates the frequency of occurrence for the requested fields (ceiling, visibility) using a configurable list of thresholds. The output is a comma-separated ascii file of frequencies for each METAR station along with latitude and longitude. The application called FrequencyMatch reads in this ascii file along with the gridded RAP model fields (ceiling and visibility) for a configurable number of days. From the model data, it calculates the frequency of occurrence using the same set of thresholds that were calculated for the METAR stations at each of the METAR station locations. The RAP model and METAR frequencies are then compared to determine if calibration is necessary. If deemed necessary, the threshold used to compute the RAP model frequency is adjusted until the model frequency matches the METAR frequency as closely as possible. Several simple data quality checks are performed along with checks to ensure that an adequate amount of data was available to perform these comparisons. FrequencyMatch then uses a circular filter to place these optimized thresholds onto a RAP grid. For grid points where multiple METAR stations have influence, we perform a distance-weighted average to determine the optimized threshold value.

Example of the ceiling calibration values for a single level (2.5 km) as obtained on 12 October 2017.

An example of the calibration data for a single level (2.5 km) is shown in the figure below. In this example, adjustments are evident as the non-green areas. For example, the RAP model had a consistent over-forecasting bias in the vicinity of Anchorage. The resulting adjustment is to increase the threshold value used to identify ceilings of 2.5 km or less. This is done in order to reduce the frequency of occurrence at this ceiling height threshold. Note that values are converted to the requisite units of kFT for display and user evaluation.

These two applications take the place of a time intensive manual offline process that had previously only been done intermittently. The new procedure runs once per day, thus providing the capability of the system to reliably respond to changes in model bias. As shown in the diagram above, the output of FrequencyMatch feeds into the previously developed application called CVmodelCal which reads in the model data along with the ceiling and visibility calibration files and adjusts the model ceiling and visibility values in areas that are consistently out of line with the observations. The real-time calibration process has been included in the CVA-AK Version 2.0 products.

Resources

Model-METAR Errors

RAP model ceiling and visibility estimates are compared to METAR measurements collected every 20 min at Alaska METAR sites. Daily and weekly error plots (offsets between RAP and METAR values) of ceiling and visibility estimates are being generated automatically. The offset values are calculated as Model minus METARS. Negative values show where the model is under estimating the ceiling and/or visibility. Daily plots and weekly plots are produced daily and can be accessed at the following links. Weekly plots contain that day and the 6 days prior to it.

For reference, the figures above show the locations and the 3 and 4 letter identifiers for the Alaska METAR stations. (click images to enlarge)

Clicking on the link for the daily plots will pop up the menu, shown in the top panel to the left. Selecting day 20160927 pops up the middle panel for selection of a specific METAR station. Plotted in the bottom panel are differences (offsets) between Model and METAR values at a particular METAR location.

Threshold values are used to color code the station names. Station names are green if at least 70% of the values are within the thresholds and red if its less than 70%. The current thresholds are 10,000 ft for ceiling height and 5 nautical miles for visibility. These values are also marked on the plots (along with the 0 line) as colored dashed lines. Station names will be black if both the ceiling and visibility data has fewer than 10% of the expected data points or if all the data points are zero. A light gray is used if only one of the two fields has this condition. So the black and gray colors are meant to show plots that have data issues.

Participants

Product Evaluation

A thorough evaluation of CVA-AK Version 1.0 was conducted using a cross-validation approach where a set of 25 METAR stations were withheld from the product and used as an independent truth dataset to evaluate the accuracy of the product. This cross-validation was done in post-analysis mode, not real-time. Data from the nine-month evaluation period were analyzed and summarized in a technical report. Information presented included seasonal and regional assessments of performance of ceiling and visibility individually, as well as flight category. An example of performance broken down by regional statistics is shown in the figure, for the CVA-AK flight category product.

Boxplots of CVA-AK Ceiling by METAR observed ceiling category for each region.

The evaluation showed that the CVA-AK has skill at diagnosing aviation hazards. Regional differences in performance are being investigated (e.g., Anchorage; Region 2). Detailed analyses showed that the complex terrain of Alaska affects the C&V performance in the model. Use of a higher resolution model such as the HRRR could help improved performance in this regard. Seasonal differences in performance are pronounced for ceiling. Visibility has consistent performance across the seasons but overall is less accurate than ceiling performance.

Changes were made to address the forecasters’ concerns and to correct other problems identified with the products. These changes included making using of the NDFD-based terrain data, using the RAP model terrain grid for ceiling AGL computations and allowing the METAR station observations to override the RAP model values when METAR observations show worse weather conditions.