How IBM’s Deep Thunder delivers “hyper-local” forecasts 3-1/2 days out

What's the weather going to be on your city block two days from now?

Deep Thunder's forecast for Hurricane Irene, based on its model for New York's weather, predicted two days ahead that it would be reduced to tropical storm status.

Photograph by IBM

Predicting the weather accurately is a hard enough computing problem. Predicting the weather for a specific location down to a square kilometer—and how it will affect the people and infrastructure there—is a problem of a much different sort. And it's that sort of "hyper-local" forecasting that IBM’s Deep Thunder aims to provide. Unlike the long-term strategic weather forecasts that many companies rely on to plan business, Deep Thunder is focused on much more short-term forecasts, predicting everything from where flooding and downed power lines will likely occur to where winds will be too high for parade balloons up to 84 hours into the future.

IBM executives hope Deep Thunder, which has been in development since 1996, will become a must-have tool for local governments, utility companies, and other organizations with weather-sensitive needs. By changing how weather data is displayed from the traditional meteorological map to a more manager-friendly approach tailored to the needs of a government or utility company, IBM researchers are hoping to get customers to buy into weather modeling as part of their operations management.

So far, major utilities and other commercial customers aren't biting. But the company has signed on at least two public-sector customers so far who are piloting their own Deep Thunder systems. One of them is the City of Rio de Janeiro, where officials are looking to beef up their emergency services preparedness in advance of the 2014 FIFA World Cup and 2016 Olympics.

In a recent demonstration for government officials using a test deployment of the system in the New York City metropolitan area, IBM showed how data from Deep Thunder could be deployed through a mobile application to give precise predictions about weather at a specific location, as well as alerts to specific risks based on the needs of the user. The "apps" IBM showed members of Congress and press last week, aren’t anything you’ll be able to download from Apple’s or Google’s online stores anytime soon, but the visualization technology used in them is being integrated into IBM's Intelligent Operations Center software and other existing operational environments.

While traditional weather forecasting can predict general weather conditions with some accuracy, it doesn’t always give government agencies and utilities the kind of information they can take action on. For example, the record snowfalls that fell in New York in January of 2011 shut down the city’s transit systems and airports, but had nearly no impact on utilities—because there were relatively little winds, and the snow didn’t stick to power lines. Still, Deep Thunder’s forecasting can sort through weather conditions and alert users to specific conditions that affect their operations.

Lloyd Treinish, the leader of the environmental science team at IBM’s Thomas J. Watson Research Lab, told Ars in an interview that the mobile app prototype—which runs on Apple iOS, Google’s Android OS, and in Web browsers—was indented as an exploration of how to disseminate the complex data that comes out of Deep Thunder’s forecasting. "Right now it’s set up to use our New York metro area forecast," he said. "If you use the default settings, based on your current location, it will give you the forecast at that location for 84 hours, updated every 12 hours. Alternatively, you can enter an address and get a forecast for where you’re going to be."

That data can be used in the field to make decisions, for example, about whether winds will be too high to do "overhead work" without having to rely on a dispatcher to make the call. Using the time and location data of forecast, maintenance crews could plan their work based on conditions at different sites, avoiding stoppages because of localized weather conditions. “You’d see the differences for JFK and La Guardia (airports) and from Central Park to lower Manhattan,” Treinish explained.

A readout from Deep Thunder's minute-by-minute windspeed forecasts for lower Manhattan on March 13 from the iOS visualization interface.

IBM

The iPad and Android interfaces, which IBM showed members of Congress and press last week, aren’t anything you’ll be able to buy from an app store. But the technology that creates the data behind them is already deployed as a result of two partnerships formed by IBM with government customers. The Rio de Janeiro effort is one of the partnerships that IBM launched last year, in the wake of a major weather-related disaster. In April 5, 2010 coastal storms in the Brazilian city caused massive mudslides, killing 200 people and leaving 15,000 homeless. And IBM's other partnership is with the University of Brunei Darussalam, which is using the technology at a national level for flood forecasting, and as part of a program to predict the impact of climate change on the country’s rainforests.

Models upon models

Predicting weather in such a highly localized fashion, whether it be in New York or Rio, is a much more complex problem than most weather forecasting models can handle. “Weather isn’t just driven by the atmosphere,” Treinish explained. “It’s also driven by how the atmosphere interacts with surface features. In urban areas, you have to look at how the urban environment impacts weather, and the sorts of feedback the atmosphere gets from highly urban environments, such as urban heat islands.”

Accurately modeling that sort of interaction requires much greater precision than weather models are capable of. While Deep Thunder’s weather models are relatively reliable at a square kilometer scale, “when you go to flooding forecasting, some of the simulation issues go to even finer scale,” Treinish said—using geographical and hydrological data that gets to resolutions of one square meter per data point. And for some predictions, such as high winds, the weather requires greater “vertical” resolution as well, to take the affect of features such as hills and buildings into account.

But collecting data to drive models of cubic-meter precision for weather is impossible, because there’s no way to either collect or process the sensor data required to feed such a beast. While it might be conceivable to cover a city with weather sensors, few cities have the IT infrastructure to handle that sort of an effort—and even those sensors wouldn’t be able to provide information about the conditions in outlying areas that will drive local weather. Even if the data was magically available for a one-meter resolution forecasting model, each step up in the “resolution” of forecasts in time and space can require an exponential increase in the computing power required to create the three-dimensional models. “If you want to double resolution, that requires eight times the compute power,” Treinish said. “If you need to increase the vertical resolution, it could be 10 or 20 times the compute power.”

To overcome that problem, IBM is using an approach for Deep Thunder that relies on “coupled models”—where one model’s data is used as an input for another’s—and historic statistical data and other data sources that are used to validate the results.

“For Rio, we’re running the weather models at four different resolutions at the same time, using telescoping grids” said Treinish. “We start off with a global model from NOAA, and from that zoom in on Rio—the horizontal resolution goes from 27 kilometers to 9, then to 3, then to 1. That's a way of balancing geographic area with the physics required to solve the business problem.” The precipitation data extracted from the 1-kilometer model is in turn fed into a 1-meter-resolution geographic and hydrologic model to forecast flooding, he explained.

By taking this layered model approach, the IBM Research team has been able to take what qualifies as a fairly big high-performance computing problem and shrink it down to a relatively small IT footprint of a few parallel processing systems—“a rackful of equipment,” Treinish said, for the level of targeted weather forecasting currently being done in Rio and other places. “The architecture is similar to the large systems that IBM sells to NOAA and Air Force’s weather agency, but just much smaller.”

The Rio configuration includes a few POWER7 systems interconnected with an Infiniband switch. For larger applications of the technology, the compute power can scale up quickly, however—a nationwide meteorological monitoring system based on Deep Thunder being run at the University of Brunei Darussalam runs on an IBM Blue Gene supercomputer.

Building and tweaking Deep Thunder’s mass of models requires a lot of data sources. “We use whatever data we can get our hands on,” Treinish said. That includes public sources such as satellite and other sensor data from NOAA, NASA, the US Geological Survey, and the European Space Agency. “In the work we’ve done at Rio and Brunei, we’re utilizing remote sensing of surface temperatures.

IBM is also using private data sources to feed Deep Thunder—or at least validate and tune its forecasts, depending on the type of data available. In addition to connecting to any weather sensors a municipality or company may have in place, IBM has also connected to partners for validation data; in New York, for example, the company has teamed with Earth Networks to pull data from its WeatherBug network of sensors. “Their sensors are clustered in the US around urban and suburban areas,” Treinish said, “but so are a lot of our applications. We get the data every five minutes—it’s very valuable to validating and improving the model.”

Is the Deep Thunder validation being published anywhere? I read the article over a few times and then visited their site but could not find any validation information. When you are talking about a numerical weather model being run at 1km grid spacing, one of the biggest issues has always been the lack of observational data at that resolution to initialize with. And without a proper initialization, your usually end up with a sub-par forecast. I see the IBM site has some animations but they appear to be only from the Deep Thunder output.

I'm also curious about how they are doing the coupled modeling. Are they strictly using the initialization from NOAA's global model to initialize their model or are they using forecasted data. The latter is being used with off hour NCEP model runs (06z and 18z) which result in forecasts that perform rather poorly compared to those initialized with observational data.

Finally, and completely off topic: does anyone think the 3D products like the one at the top of the article useful? While it is pretty to look at, I find using products like that to forecast with difficult at best.

*snorts* I'll believe it when I see it. Minnesota weather is schizophrenic at best. They predict snow in the metro 12 hours out, and we get rain. They predict 6-8 inches of snow, we get .25" They predict heavy rain...it goes 50-75 miles south of us. I've started getting my weather from my local fortune teller. Its 50% more reliable.

What would have been neat with that graphic is a forecast display for a period past overlaid with the actuals measured for the period forecast. That said, even a large disparity between the two may not be particularly significant in assessing its legitimate value.

This is not news, but many are acting as if its something new worth stealing our precious time and attention, mostly because IBM has created an iPad UI for it. Zzzzzzz ... The businesses such as utilities are wisely not biting because they know its flaws and limitations, the dependency issues, and so on (besides, why pay $ to IBM for this?). If its so great, why doesn't NOAA use it aka "dog fooding"? Truth be told, what Deep Thunder really is is a marketing tool for IBM to showcase their systems (like Watson). Meanwhile, people like Tom Skilling at WGN-TV can still beat Deep Thunder (did Deep Thunder forecast today's record breaking heat in Chicago? Nope, but Tom called for it, so Skilling trumps the machine)!

Strange that they do not use an ensemble approach, which would give uncertainty ranges that are inevitable given the chaotic nature of the weather. A forecast for "scattered showers" may be more accurate than a random realisation even if the latter looks more precise.

How accurate is it? That's all I want to know. It says it can start forecasting 84 hours in the future, and updates every 12 hours. Obviously the accuracy would improve as it gets closer, but how much? It sounds amazing, I'd just like some quantifiable stats to back it up.

This reminds me of drug companies coming out with 8 different kinds of drugs for erectile dysfunction while spending little to no money researching new antibiotics. Two weeks ago, tornadoes killed 38 people in the Midwest and South. Find something that can predict *that.*

This reminds me of drug companies coming out with 8 different kinds of drugs for erectile dysfunction while spending little to no money researching new antibiotics. Two weeks ago, tornadoes killed 38 people in the Midwest and South. Find something that can predict *that.*

Tornado Alley is huuuuuge and most of it is sparsely populated. The amount of processing power and input resources currently needed for these sorts of models is only really practical for small, dense areas right now.

I did live through a sudden and completely unexpected tornado once. My neighborhood was fine, but a few miles away was completely destroyed. It happened very quickly out of perfectly routine warm weather. Predicting this with any sort of specificity is the goal to work towards, not the starting point.

@BRock97> Any IBM verification information that is public is likely either available from their IBM Journals (Systems or Journal of Research and Development) or from AMS conference proceedings. I don't know of any, but that's not saying much.

I take it their "coupled modelling" is combining it with non-meteorological / non-climatological data such as very high resolution (guessing 1 to 10m) topography, bathymetric, and 3-D building modeling (a la Google's SketchUp). Modelling wind through buildings (urban canyons) becomes a computational fluid dynamics (CFD) modelling rather than a numeric weather prediction (NWP) model. I don't know anything about coupling CFD with NWP.

@gjvanoldenborgh> Ensemble needs multiple similar ("final output") models (target region, ideally at similar resolution), and by the sounds of it, IBM's project only has "one" output model, customized per customer.

@Chaster Mief> One side effect of higher geographic resolution, is that computations must be done at finer (smaller) time steps, so it becomes feasible to produce 3 hour if not 1 hour time steps. I suspect many commercial / municipal customers like this, so they can have a better idea of when the model says a storm will reach their airport / city or amusement park.

@abadidea> NOAA does run mesoscale (finer resolution than their national, "regional" resolution) modelling for hurricanes, I don't know if they have anything equivalent for tornadoes. I believe rapid scan satellite is the primary source of tracking, as tornadoes have such little warning in most cases.

@Postulator> Sure, they/you can buy it from IBM, but it trades off forecast area (city rather than a country) for resolution (1 to 10 meters rather than 2.5 to 50km), so it might be good for Melbourne but won't help the Outback. (I'm assuming AU rather than NZ et all) It also seems optimized / oriented for urban density and perhaps coastal but not actual marine, as well.

Re: public forecasts: The biggest issue is in all but the largest urban areas the forecast regions for public weather forecasts are so large, and that the forecasters have to report critical weather that may happen with the entire region, not just where you are. I live 7 km (~4 mi) from a weather observation site, and routinely have different local weather when I step outside my door versus that ob site reporting even at the same minute.

Weather is a non-linear and dynamic physical system - it is hard to predict by its nature, forecasters I know don't put much faith in forecasts beyond 72 hours, and that "extended" (5-10 days) forecasts are useless except for very general guidance / warning of crucial points to watch closely.

"Telescoping grids" - the more standard terminology is "nested grids" and US / European / Canada forecast government global and regional models all use nested grids (non-uniform I think in all operational cases nowadays).

Disclaimer: I've made a lot of speculation above, and I shouldn't be mistaken for being actually an informed opinion let alone being knowledgeable. Wait, this is the Internet, everyone's an expert.

Sean Gallagher / Sean is Ars Technica's IT Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland.