Model Accuracy Guides

Setup & Instruction Manual

Step 1.

Click "New Race" to create a new race file.

This creates a folder (named after the race) inside the Model
Accuracy "Races" folder, in which prepared data, saved analyses,
and analysis settings are saved, to allow you to quickly load your
preferences in future analysis runs.

Select “True” or “Magnetic” mode. This is to determine the
format of the wind direction stored in your log file. It is either
True or Magnetic. If your log files capture TWD in degrees True,
then select True. If your log files capture TWD in degree
Magnetic, then select Magnetic. Consult Expedition settings if
necessary.

Set mast height. This must be the distance in meters from the
water line to the anemometer.

Set boat data smoothing window. This smooths out the boat log
data, removing large changes in short time period that are caused
by, e.g., the swinging of the mast, short gusts of wind, etc.

Step 2.

Tell Model Accuracy where to find the boat's log files that you
want to evaluate.

The folder that stores the log files must contain only log
files.

Tip: Click the folder icon to open up the file explorer.

Tip: Model Accuracy has a quick reference for the folder where
that Expedition Sailing software package always stores the log
files.

Model Accuracy will analyze the boat's log files wherever you
specify their folder. It does not have to be where Expedition
stores the files, although that folder is referenced for easy
operation.

Step 3.

Tell Model Accuracy where to find the GRIB files that you want
to evaluate.

The folder that stores the GRIB files must contain only GRIB
files, or sub-folders that contain only GRIB files.

Tip: click the folder icon to open up the file explorer.

Tip: Model Accuracy has a quick reference for the folder where
that Expedition Sailing software package always stores the GRIB
files.

The GRIB source name is set by:

If the GRIB files are in a subdirectory, use the subdirectory
name.

If the GRIB files are in the specified raw GRIB directory,
Model Accuracy uses the GRIB file name to work out the source
name.

BIG_GRIB (taken from the file name)PWG_ALL1 (taken from the directory name)GFS (GFS is well known)

Step 4.

ADP (Advanced Data Preparation) - This feature prepares the boat
log and GRIB data. It determines the earliest and recommended
analysis start times, and the latest and recommended analysis end
times.

Model Accuracy recommends using the ADP capability before every
analysis!

Step 5.

Select your preferred Analysis Mode.

“All GRIB runs” creates an analysis on the freshest available
data within the GRIB folder, reviewing all the GRIB files
available.

“Single GRIB run” will allow you to analyze only one GRIB file
to compare against the boat log files. Model Accuracy will choose
the GRIB file with a start time closest to the start time of the
analysis window you select.

“Stepped” will allow you to select time steps of 6, 12, 24, 48
hours within each GRIB file source to review its accuracy. This is
helpful in determining the trends of each GRIB file for
comparison.

Step 6.

Set your analysis start and end times. Here you tell Model
Accuracy exactly from when to when you want the boat log files
compared to the GRIB files. The times must be within the range of
the boat log files' data and the GRIB files' data.

By clicking the ADP (Advanced Data Preparation) button earlier
in the set up process Model Accuracy is capable of determining the
earliest start and latest times possible for analysis, and will
provide you a “no earlier than” and a “no later than”
recommendation. These time recommendations are the earliest and
latest time where both GRIBs and boat logs overlap, which is
critical to the analysis process.

The “Use” buttons to the right of the recommended times allow
the user to immediately accept the recommendation of the start/end
time and use at as the analysis start/end time. Consult the Model
Accuracy Manual – Warnings – Notes – Definitions for more
information regarding the meaning of the recommended times.

Set analysis granularity. Model Accuracy can produce results
accurate to the minute, though the computation time is longer.

Step 7.

Select “Plots” if you would like to see the graphical plots in
addition to the statistical analysis provided.

Click "Analyze!"

Optional Step "Save"

Clicking this option automatically saves your results and data
plots for future reference, in the SavedAnalyses
subfolder of the race folder created in Step 1.

Select “View” to see the saved analysis and plots of your data
stored neatly in a folder for the race you created.

How to Understand the Analysis

True Wind Speed (TWS) Graphical Plot: Allows the user to
visually track the forecasted accuracy of each GRIB vs. the logged TWS
captured by the instruments onboard. Each GRIB is color coded and the black
line is the boat's TWS plot.

Red Squares: represent GRIB delivery times
(when you got a new GRIB). This is most commonly every 6hrs or 12hrs.

Correlation: Tells you if the GRIB prediction followed the boat
log data in parallel. Example: If the TWS was predicted to increase and
the boat data did in fact increase the correlation will be high. HIGH is
good.RMS Error: LOW is good. “Above’s” and “Below’s” do not cancel each
other out! This is the most accurate error measure, taking into account
how much time the prediction spent not aligning with the boat data. Average Error: LOW is good. This is the simple comparison if you
averaged all the differences between the GRIB data and the boat log data.
In theory here you can have a GRIB prediction way above and then way below
but then averages to the exact average number of the boat data and have a
zero Average error.” “Above’s” and “Below’s” do cancel each other out in
this analysis! % Below % Above: Counts the number of times the GRIB prediction is
above/below the boat log data, and forms a % of how often it is
above/below the boat data line and how inaccurate it is when above and
below the boat data line.

RMS Error is the most important number in determining the
accuracy of a GRIB file.

Analysis Mode Definitions

All GRIB runs Analysis mode: Compares the freshest GRIB
data to the boat log data. For example, if you get a new GRIB every 6 hours
then the oldest GRIB data that will be analyzed in this mode is 6 hours old.
This is a broad and general analysis to determine which GRIB is doing well
at predicting weather. (most common analysis)

Single GRIB Analysis mode: Compares a single chosen
GRIB file (one source, one delivery time) to the boat log data.

Stepped Analysis mode: Compares a chosen GRIB delivery
time of all the GRIBs in the folder, by examining in small blocks of time
within the analysis time frame set by the user. The user can choose
the size of the blocks of time to compare within the overall analysis (6
hr, 12 hr, 24 hr, 48 hr). This mode is helpful to see the trend more
clearly of when each GRIB's forecast gets inaccurate. The stepped mode
outputs the results in blocks of times analyzed. Example:

Warnings and Notes

Your computer must be set to Zulu time. This allows the boat log file
to be time-stamped in the same way that the GRIB files are created
(i.e., in Zulu time).

Model Accuracy applies the appropriate variation as applicable to the
GRIB files. Make sure you understand how your TWD is being stored in
your GRIB files as either True or Magnetic. Consult your settings in
Expedition to make sure you select the proper setting in Model Accuracy.

The height of your anemometer (wind instruments on your mast) must be
entered in meters. This measurement is from the waterline to the
anemometer (birdie and cups).

Model Accuracy will read only .grb and .grb2 type
GRIB files.

The time selected for the analysis must be covered by both
the boat log files and GRIB files, or else the analysis will not work.
Use the Advanced Data Preparation button/feature to help learn what are
the earliest and latest times where both the GRIB and log files overlap
in your directory.

Analysis Start Times: The EARLIEST start time is constrained by the
start of the boat log data & the single earliest GRIB run. If you
start the analysis at this time only that one GRIB source gets analyzed
(because the others start later). The RECOMMENDED earliest start time is
constrained by the start of the boat log data and the latest GRIB run.
If you start the analysis at this time all GRIB sources get analyzed.

Analysis End Times: The LATEST end time is constrained by the end of
the boat log data and the single latest GRIB step time. If you end the
analysis at this time only that GRIB has data (in the form of a GRIB
step) at the end time. The others will have older data (because their
last GRIB steps are earlier). The recommended latest end time is
constrained by the end of the boat log data and the earliest last GRIB
step time. If you end the analysis at this time all the GRIBs have data
(in the form of a GRIB step) at the end time.

It is recommended that you keep all of your GRIB files organized by
Zulu run time in separate folders, as you download them. When you run an
analysis, create a new folder of all the GRIBs you want to analyze and
specify that folder as outlined in Step 3 of the instruction manual.

It is recommended that you rename each GRIB file that you download.
This is can be tedious, however it helps you quickly identify each
individual GRIB file for future analysis, and ensures as a navigator you
always know which GRIB you are using. For example:

Here the navigator has renamed a GFS GRIB file so that it is clear that
this GRIB is the 0000 Zulu run from June 4th 2016 and has a 96hr
forecast. This provides quick reference not only for the Model Accuracy
analysis, but also for navigating.