REOS Metrics

Defining a common set of metrics to evaluate ocean model simulations depends on the purpose of the ocean model and the scientific questions it is being used to address. The challenge is to try to establish a consistent evaluation process for ocean models. It is hoped that the discussion that this webpage will foster will ultimatly lead to the development of a set of community-wide approved metrics, endorsed by WGOMD, that will provide a quality standard for ocean model performance that the community should endeavour to meet.

Recommendations from CLIVAR basin panels

The CLIVAR basin panels have been asked to provide WGOMD with input on ocean metrics to evaluate model performance in terms of ocean processes occurring in the different ocean basins.

Integrated Mass Transports - Observational Estimates

There are a number of climatologically important straits, throughflows and current systems whose integrated mass transport is measured observationally (though some have wide uncertainties). These mass transports provide a useful means of characterizing an ocean model simulation. The following recommendations are made regarding the diagnosis of model integrated mass transports:

The following table notes the approximate geographical location of the straits and currents. Given considerations for model grid resolution and grid orientation, precise values for the coordinates may differ between models. In general, it is recommended that the simulated transport be computed where the strait is narrowest and shallowest in the model configuration, and where the model grid is closely aligned with the section.

For most ocean model grids, the requested transports can be diagnosed by aligning the section along a model grid axis. In this case, it is straightforward to assign a positive sign to transports going in a pseudo-north or pseudo-east direction, and negative signs for the opposite direction. We use the term pseudo here as it refers to an orientation according to the model grid lines, which in general may not agree with geographical longitude and latitude lines.

Some models may have a strait artificially closed, due to inadequate grid resolution. In this case, a zero transport should be recorded.

References to observational estimates of the mass transports are provided in the table below. These references can be updated as new estimates become available. See References page for full citations and links to the articles where possible.

Smedsrud et al. (2010): 2.0 Sv eastwards (1 Sv=1°—106 m3/s). This net flow is derived from four sources; the inflowing Norwegian Coastal Current (NCC) at the southern coast, the Atlantic inflow in the central area, and two westward flows in the northern areas

Qu et al., 2009: The total transport through the Luzon Strait is essentially westward. Prior studies have arrived at a broad range of this transport estimate, varying from 0.5 to 10 Sv, while recent observations favor a value near the middle of the range.

Smith et al, 2007: The transport entering Windward Passage is highly variable, including reversals to net outflow. Transports measured during the cruises ranged from -0.3 Sv (outflow) to 9.4 Sv (inflow), with an average inflow of 3.8 Sv. Corresponding transports derived from the current meter array range from approximately -5 to 15 Sv, with an average inflow of 3.6 Sv. On average there is net inflow in the surface and thermocline layers (above ~600 m), net outflow in the intermediate layer (~700-1200 m), and a deep inflow just above the bottom.

Bulgakov et al., 2003: A compilation of earlier results

Johns et al., 1999: Measurements made just to the northeast in the passage between Great Inagua Island, the Bahamas and Haiti, assuming that flow through this passage was a good approximation to flow through the Windward Passage. The estimate was made using directly observed currents in the upper 200 m, the mean transport from 9 crossings of the section: 2 Sv, stdev 1.5 Sv (westward)

Other activities related to defining ocean model metrics

Workshop on Ocean Model Metrics (25 February 2006, Hawaii, USA)

This one day workshop was held to discuss the current state of ocean model metrics and how the ocean modelling community, working together with the observations community can improve how ocean models are evaluated. Ocean models are being run at higher resolutions that should be capable of resolving not only the climatology but also high frequency and mesoscale variability. This motivates the development of metrics that help evaluate and understand the variability at the space and timescales resolved by the models, as well as the representation of physical processes.

Since the definition of metrics depends on the purpose for which the model is being used, the workshop divided the ocean modelling community into three groups: climate modelling (see here for the recommendations), operational modelling, and modelling for process orientated studies.

A key recommendation is that the design of process orientated experiments must be designed from the outset with model evaluation and comparison as a goal. Models and observations must both be components for the study to be successful. The workshop participants also recommended that a website be developped for the discussion of ocean model metrics, to facilitate access to observational datasets, and to feature a comparison of metrics derived from ocean models and from observations.

- Timeseries of meridional MOC of the global ocean, Atlantic and Indo-Pacific as a function of latitude and depth and for the global ocean as a function of latitude and potential density.- Timeseries of meridional heat and freshwater transports of the global ocean, Atlantic, and Indo-Pacific as a function of latitude.- Time series of maximum MOC strength and heat transport at 25N, 48N in North Atlantic

- Monthly means of averaged temperature (proxy to heat content) and salinty over the upper 300m/750m and 3000m.- Time series for spatial averages within a list of 30 pre-defined boxes in various parts of the ocean.- Monthly means of sea level, and optionally steric height and/or bottom pressure.- Time series for spatial averages within a list of 30 pre-defined boxes in various parts of the ocean.

- 18-C water volume in the N Atlantic Ocean, volumne-weighted average salinity of the 18C water as a function of month.- Annual Maximum mixed layer depth within the Labrador sea and the T,S properties of that mixed layer.- Warm-water volume in the equatorial Pacific (5S-5N, 120E-80W) AND tropical Pacific (20S-20N, 120E-80W),- Depth of 20 degree isotherm in Pacific Ocean as a function of longitude, latitude, and month.

MERSEA Integrated Project Metrics

A European system for operational monitoring and forecasting of the ocean physics, biogeochemistry, and ecosystems, on global and regional scales is being developed by the Marine Environment and Security for the European Area (MERSEA 2004-2008) Integrated Project. The system will be a key component of the Ocean and Marine services element of GMES (Global Monitoring for Environment and Security).

2D and 3D daily averaged fields interpolated onto a common set of GODAE horizontal and vertical grids. Used as 'instantaneous' esitmates of the ocean mesoscale circulation for direct comparison with observed quantities.

Metrics to measure the performance of the forecasting system, limited to 'observational space' rather than 'model space'. Observations, preferably not assimilated by the forecasting system, are compared to the forecast and hindcast for a given day.

Diagnostics TableDownload here a table compiled from the MERSEA report that summarises the list of diagnostics planned for an intercomparison between GODAE partners to verify 1) the consistency of the ocean dynamics and variables using the "best estimate" or "hindcast products, and 2) the quality, or precision, of the operational daily estimates of the ocean and sea ice circulation and dynamics.