We have written a fair amount at Ars recently about the superiority of the European forecast model, suggesting to readers that they focus on the ensemble runs of this system to get a good handle on track forecasts for Hurricane Irma. Then we checked out some of the preliminary data on model performance during this major hurricane, and it was truly eye-opening.

Brian Tang, an atmospheric scientist at the University of Albany, tabulates data on "mean absolute error" for the location of a storm's center at a given time and where it was forecast to be at that time. Hurricane Irma has been a thing for about a week now, so we have started to get a decent sample size—at least 10 model runs—to assess performance.

The model data

The chart below is extremely busy, but when you understand how to read it, the data is striking. It shows the average position error (in kilometers) at forecast lead times of 12, 24, 48, 72, 96, and 120 hours (so, out to five days). It compares several different classes of models, including global models that forecast conditions around the planet, nested models focused on hurricanes, and consensus forecasts. Specifically, the models are referenced as follows:

Forecast models typically show their skill with three-, four-, and five-day forecasts. For simplicity's sake, we will focus on 120-hour forecasts. At this lead time, the average error of the European model with respect to Irma has been about 175km in its position forecast. The next best forecast is from the hurricane center, which is slightly more than 300km. An automated model, then, has so far beaten human forecasters at the National Hurricane Center (looking at all of this model data) by a wide margin. That's pretty astounding.

What is particularly embarrassing for NOAA, however, is the comparison between the European model and the various US forecast modeling efforts. The average 120-hour error of the GFS model is about 475km. The operational, hurricane-specific model, HWRF, does better, with an average error of 325km. But the experimental HMON model does terribly, at nearly 550km of error. A similar disparity in quality goes all the way down to 24-hour forecasts.

Another method of determining track accuracy is by looking at trend maps, which show a time series of tracks. In the gallery below, you can gauge the consistency of forecast models and their accuracy based upon the actual track of the storm.

These trend maps in recent track forecasts show which models have made consistent forecasts, and which have varied significantly. You can also track accuracy by looking at the actual position (X) taken by the storm. This is the European model.

Brian Tang/U Albany

Trend map for the US GFS model.

Brian Tang/U Albany

Trend map for the Canadian global model.

Brian Tang/U Albany

Trend map for the UK Met global model.

Trend map for the US Navy global model.

Trend map for the HWRF hurricane model.

Brian Tang/U Albany

Trend map for the HMON experimental hurricane model.

Brian Tang/U Albany

Trend map for the TVCN consensus model forecast.

Brian Tang/U Albany

Trend map for the official forecast from the National Hurricane Center.

Brian Tang/U Albany

Why the US lags

So what's the deal here? The overall performance of the National Weather Service's GFS model has lagged for years behind the European forecast system, which is backed up by superior resources and computing power. Finally, this year, the GFS was upgraded. However, even before those upgrades went into effect, hurricane forecasters were raising concerns about the new GFS.

Shortly before the beginning of the 2017 Atlantic hurricane season, in fact, forecasters at the National Hurricane Center in Miami pushed back against the upgrade. They had noted degraded performance during internal tests of the GFS model on Atlantic tropical cyclones. The track forecasts were about 10 percent worse with the newer version of the model than the older one.

Further Reading

In a presentation posted on the National Weather Service website, first reported by Mashable, the hurricane center officials said, "The loss of short- to medium-range [tropical cyclone] track and intensity forecast skill for the Atlantic basin in the proposed 2017 GFS is unacceptable to the National Hurricane Center." Ultimately, the upgrade was initiated anyway.

An independent expert on global forecast models, Ryan Maue, said the NOAA office responsible for developing US computer models, the National Centers For Environmental Prediction, is understaffed and has less funding than the European forecasting center, which is based in the United Kingdom. America, he said, is getting what it pays for.

"NOAA and the National Weather Service are stretched a mile and an inch deep in some places for all of the responsibilities that they have," said Maue, a research meteorologist at the Cato Institute. "If we want to focus on having the best weather forecast in the world, we should focus on having the best weather forecast."

This is unacceptable. We should be leading the world on this, not because we are the USA, but because we have the resources and these storms cost us a lot more than they should. The GOP, and their crony capitalist puppet masters, refuse to fund the NOAA like they should because they worry about the resources being used to undermine their short-term profits through climate change warnings. Rome, indeed.

Interesting - the Canadian model seems to be the only one showing Irma curving through FL but actually pulling back off the East coast, rather than heading straight up the center like the other models.

I wish we could tangle out the details of the model - figure out what element made the Euro model move westward well in advance of the other models (that were mostly showing tracks heading up the east coast without actual landfall at that point).

OFCL is performing well and it is the official forecast NOAA Hurricane Centre forecast - is this a consensus forecast of the various models and not an actual statistical/dynamical model?

It should be the best, bar none. Forecasters at the NHC review all of the model data, current movement, satellite observations, and use their own insights and experience to create the "official forecast."

Man the people making these wind intensity maps need to get better at their color coding of values... and their dammed obsession with rainbows...

it's nearly impossible in that lead chart to tell if something colored greyish is 15, 76, or 150.. ditto for anything redish.. you HAVE to depend on the colors around it, and other contextual clues, and even then (such as near the eyewall) it's hard to tell.

and gods forbid if you are color blind (just try installing the utility at colororacle.org and looking at it with that turned on)

It's not like this is not well researched and a solved problem, in terms of better color scales that are easier to read..

It would be interesting to see if you could correlate accuracy of forecasting with total economic storm damage costs

My guess is the EU model would win again because to my knowledge we don't have these kind of storms. Sure some floods here and there after winter, ice melting and that stuff in spring, but nothing in the category of "OMG WE ARE ALL GONNA DIE!"

From a EU point of view those models are a "Ha! Cool, it works." kind of intelectual exercise.

It would be interesting to see if you could correlate accuracy of forecasting with total economic storm damage costs

My guess is the EU model would win again because to my knowledge we don't have these kind of storms. Sure some floods here and there after winter, ice melting and that stuff in spring, but nothing in the category of "OMG WE ARE ALL GONNA DIE!"

From a EU point of view those models are a "Ha! Cool, it works." kind of intelectual exercise.

Also an insurance against shifting weather patterns in the future... Just saying.

| the European forecast system, which is backed up by superior resources| and computing power

Politics, fucking or otherwise, is the way we make public policy. There's billions for rebuilding -- after the fact -- but scrimping and budget reductions for more accurate forecasting and risk reduction. That's politics.

By the way, IMHO, I'm utterly amazed at the accuracy of even the least accurate forecasting. To say these storms have many moving parts somewhat understates the enormity of the challenge.

What's interesting about the earlier Irma forecasts was the drastic and sudden change of direction they predicted when it was south of Florida. I thought that was a bit odd that a hurricane this large would pivot on a dime and end up east of Florida. These newer forecasts seem much more realistic in terms of change of direction. I'm willing to bet this hurricane ends up moving even more west than the recent forecasts are predicting.

I wonder if the movement west in the forecasts has more to do with wind and pressure conditions being different/not as drastic or with the storm's momentum being more powerful than anticipated and therefore harder for winds/pressure to change its direction.

I think I heard that Katia (hurricane in the Gulf by Mexico) is affecting some of those conditions. So now you have to figure out what one hurricane is doing in order to tell what the other one is doing...yeah this is not an easy job.

It would be interesting to see if you could correlate accuracy of forecasting with total economic storm damage costs

My guess is the EU model would win again because to my knowledge we don't have these kind of storms. Sure some floods here and there after winter, ice melting and that stuff in spring, but nothing in the category of "OMG WE ARE ALL GONNA DIE!"

From a EU point of view those models are a "Ha! Cool, it works." kind of intelectual exercise.

No they are not you seem to forget that there are EU territories and lives in this region that are impacted by those storms as well as important assets like the space center in Guyana . Far from a simple "intellectual exercise "

My dad is currently in Cape Coral. I just got off the phone with him 20 minutes ago, and he's telling me they're going to get a 3-4 ft storm surge, and have 75 mph winds.

I just saw the above pic and it looks like it's making landfall less than 100 miles away from him. He said he's listening to the local stations and the governor (I'm assuming his staffs updates?), but I don't know wtf they're using for their forecasting.

This is going to be his first hurricane, and I don't think he really respects the possibilities that can happen during these storms.

I guess two neighbors came back yesterday because they were on the road for a full day and couldn't make it to Tampa in 12 hours, so I don't know if they could get out now even if I pleaded with him.

So I'll just be sitting here in Michigan, alternatively thinking what an idiot for not leaving on Tuesday like I said when I called him then, and hoping he's alright Sunday night.

More accurate forecasts would mean less unnecessary costs (financial and psychological) preparing for an emergency in areas (like SC & NC for this storm) where it now looks like there won't be hurricane force winds.

Are better forecasts really necessary? I always thought the main reason we need these forecasts is whether to issue evacuation orders. And they seem to be already good enough for that. Pinpointing the exact point of landfall doesn't seem all that important to me, since the damage will be assessed after the fact anyways.

I mean, would there be any real practical difference for Irma if we were able to predict its exact path over Florida 5 days ago?

Read one post up. You have people making life and death decisions based on different forecast models.

Are better forecasts really necessary? I always thought the main reason we need these forecasts is whether to issue evacuation orders. And they seem to be already good enough for that. Pinpointing the exact point of landfall doesn't seem all that important to me, since the damage will be assessed after the fact anyways.

I mean, would there be any real practical difference for Irma if we were able to predict its exact path over Florida 5 days ago?

There is certainly too much focus put on where the eye will make landfall. Look at Harvey, eye never came close to Houston but that is where a lot of the damage is. I grew up in hurricane alley, and we were constantly reminded to not focus on the eye. If it's going to be anywhere near you, you need to prepare. The NHC even says that "nobody should focus on the exact track of the center".

That being said, if Irma keeps moving west and actually ends up in the gulf, then you have a very dangerous situation for gulf coast communities who aren't expecting much from this hurricane at all.

It would be interesting to see if you could correlate accuracy of forecasting with total economic storm damage costs

My guess is the EU model would win again because to my knowledge we don't have these kind of storms. Sure some floods here and there after winter, ice melting and that stuff in spring, but nothing in the category of "OMG WE ARE ALL GONNA DIE!"

From a EU point of view those models are a "Ha! Cool, it works." kind of intelectual exercise.

Also an insurance against shifting weather patterns in the future... Just saying.

Hmm true but the EU has one thing going if my geology teacher didn't lie... if the weather changes then the first thing to go is the Golf stream (no heat difference= no exchange)that currently heats the EU. So the EU cools and would turn into a nice place to ski or snowboard but at the same time the earth gets warmer so... we might have it a bit swampy but kind of ok.

Unless of course someone goes nuclear then we have the Shadowrun nuclear swamps of middle EU and can only hope for dragons and magic to return.

Are better forecasts really necessary? I always thought the main reason we need these forecasts is whether to issue evacuation orders. And they seem to be already good enough for that. Pinpointing the exact point of landfall doesn't seem all that important to me, since the damage will be assessed after the fact anyways.

I mean, would there be any real practical difference for Irma if we were able to predict its exact path over Florida 5 days ago?

Read one post up. You have people making life and death decisions based on difference forecast models.

Absolutely. An accurate model can allow for much better evacuation planning and can allow for you to pick the best locations for shelters. ECMWF was literally the only acceptably accurate model during Harvey 3 days out - and NHC eventually just started essentially releasing the forecasts from that model as their official ones. They've taken to prioritizing ECMWF with Irma now too as Eric has said.

This is unacceptable. We should be leading the world on this, not because we are the USA, but because we have the resources and these storms cost us a lot more than they should. The GOP, and their crony capitalist puppet masters, refuse to fund the NOAA like they should because they worry about the resources being used to 7ndermine their short-term profits. Rome, indeed.

Congratulations. You politicized rain.

No. Rain was politicized by being an inevitable sequel to more CO2 in the atmosphere, making it any ally of, cough, global warming, cough. Blame Clausius-Clapeyron for pre-politicizing the rain.

Are better forecasts really necessary? I always thought the main reason we need these forecasts is whether to issue evacuation orders. And they seem to be already good enough for that. Pinpointing the exact point of landfall doesn't seem all that important to me, since the damage will be assessed after the fact anyways.

I mean, would there be any real practical difference for Irma if we were able to predict its exact path over Florida 5 days ago?

Knowing if the one needed to evacuate Miami 5 days ahead of time is better than 24 hours, I would think. But beyond forecasts, better models -- absolutely. The better we can model the various systems, the more we learn. And better modeling of weather assists with better modeling in other areas, and that improves quality of life as well.

It is ridiculous that the U.S. does not have the best models. Europe is beating us with these storm models, and China is beating us with ocean models. We should have the resources to put towards these endeavors, but we don't. The simple fact is we are under-investing in modeling and simulation and the requisite computational hardware, and it will have ramifications in other areas.

Are better forecasts really necessary? I always thought the main reason we need these forecasts is whether to issue evacuation orders. And they seem to be already good enough for that. Pinpointing the exact point of landfall doesn't seem all that important to me, since the damage will be assessed after the fact anyways.

I mean, would there be any real practical difference for Irma if we were able to predict its exact path over Florida 5 days ago?

There is certainly too much focus put on where the eye will make landfall. Look at Harvey, eye never came close to Houston but that is where a lot of the damage is.

That being said, if Irma keeps moving west and actually ends up in the gulf, then you have a very dangerous situation for gulf coast communities who aren't expecting much from this hurricane at all.

Harvey was different as it was mostly a stationary conveyor belt for water from the Gulf. Winds are going to be more significant for Irma.

Irma is not going into the Gulf, it's not going to just bust straight through that trough. A gut feeling isn't a substitute for the models.

Are better forecasts really necessary? I always thought the main reason we need these forecasts is whether to issue evacuation orders. And they seem to be already good enough for that. Pinpointing the exact point of landfall doesn't seem all that important to me, since the damage will be assessed after the fact anyways.

I mean, would there be any real practical difference for Irma if we were able to predict its exact path over Florida 5 days ago?

There is certainly too much focus put on where the eye will make landfall. Look at Harvey, eye never came close to Houston but that is where a lot of the damage is.

That being said, if Irma keeps moving west and actually ends up in the gulf, then you have a very dangerous situation for gulf coast communities who aren't expecting much from this hurricane at all.

If you based your shelter planning in Houston on the ECMWF forecasts then you could have planned for over 35 inches of rain for 3 or 4 days. Far too much planning was done based on the incorrect models predicting 20-25 inches of rain though. And sure enough some shelters flooded.

Actually most of the traditional hurricane damage from Harvey is exactly where you'd expect it - near the point of eyewall impact at Port Arkansas and Rockport. The damages in Houston are due to flash and river flooding and it was a very abnormal event. For most storms you're really going to be most interested in the location the eye makes landfall.

It would be interesting to see if you could correlate accuracy of forecasting with total economic storm damage costs

My guess is the EU model would win again because to my knowledge we don't have these kind of storms. Sure some floods here and there after winter, ice melting and that stuff in spring, but nothing in the category of "OMG WE ARE ALL GONNA DIE!"

From a EU point of view those models are a "Ha! Cool, it works." kind of intelectual exercise.

No they are not you seem to forget that there are EU territories and lives in this region that are impacted by those storms as well as important assets like the space center in Guyana . Far from a simple "intellectual exercise "

I wouldn't call it EU territory because Guyana went independent in the 60s?70s? something like that. And it is on the south side of the golf so given that most events go to the north, not really a danger there.

French Guyana is not independent as are the close to a million people living in Martinique and Guadeloupe two other French territories in the West Indies (think Porto Rico but with actual votes in the parliament).