So … how well did those swine flu models do?

About a month ago, during the embryonic stages of the swine influenza outbreak, I wrote about some Northwestern University engineers who were modeling the spread of the disease.

I found the projections calming as, at the time, there was a fair amount of hysteria as we were still gathering information about the spread and virulence of the H1N1 strain. The model predicted that, in a worst-case scenario, Harris County would have 138 cases by May 27.

In this sense, then, the models verified. However, the models failed to predict that Cameron County, in the Valley, would have the most cases in the state. It evidently underpredicted the spread in more rural areas.

Brockmann

So how did they really do for all 3,000 U.S. counties? I contacted the project leader, Prof. Dirk Brockmann, and he said the team was “pretty happy” with their results. Based upon a statistical verification scale from 0 to 1 that I am not going to pretend to understand, Brockmann said the model scored a 0.91. Considering that a 0.50 or higher shows some predictive skill, that seems pretty good.

Right now this science of computational epidemiology remains in its infancy, way behind other predictive sciences such as meteorology.

To make a weather forecast you need excellent information about initial conditions (i.e. temperature, winds, air pressures) and complex computer models that mimic atmospheric conditions. It’s taken meteorologists decades to get to the point where they have good information on initial conditions and good models.

As a result, modern 3-, 4- and 5-day weather forecasting is reasonably good.

Brockmann and others in his field are still at the beginning of the curve: they have relatively poor information at the beginning of an epidemic, and their models need considerable refinement. But he says there’s no question this method will eventually pay public health dividends.

Brockmann also believes that the proliferation of mobile devices and social media will provide a wealth of information about how humans interact, which is critical to accurately modeling the spread of diseases.

“We’re not yet in a state where we can do what meteorologists do, where it’s common to make accurate forecasts,” he said “We’re just in the beginning. But I really believe this is the right path to take, and I think we’ve reached a turning point in our field.”

Great coverage on Dr. Brockman and his work. I doubted his report and your reporting on it based on the overwhleming reportage at the time. We obviously did not have (thankfully) the kind of rapid and dangerous spread the early hype predicted. Congrats on giving a cool head a great venue for getting the word out. Swine Flu is not an apocalypse, thanks be to all that is good. Dr. Brockman saw this, and stuck to his guns when it would have been much more popular to simply be part of the consensus.

One interesting thought – Are the researchers planning on running a set of models predicting more long-term data on the H1N1 flu? I have a sneaking suspicion that this flu will come back with a vengeance this winter.

I’m not buying it. This just takes into account confirmed cases. It’s been made pretty clear that the actual number of cases is greater, due to folks getting better without going to the doctor, medical professionals choosing not to test, etc.

This was a good example of Predictive Science but your analogy to weather forcasting was not valid, IMHO. The TV weathermen don’t know if it’s going to rain, might rain, could rain, or won’t rain from hour to hour. That’s why I call them weather guessers. Usually I can tell more from stepping outside and looking up and feeling the wind than they can tell me with their fancy charts and almost never right computers.

I noticed the extreme media hype around what I considered a nonepidemic YET…and was really waiting to see if anything would come of it. Glad to see someone is working on modeling for this! Great reporting glad for the follow up which is not done often enough!

Yes, I recall your post on this, and Brockmann’s forecasts were quite a common-sense contrast to all the media hype!

But if they are going to improve their models by computing vastly more detailed behavourial paradigms

— they will certainly need a lot of computing power! :^D

Just looking at my immediate suburban surroundings, I count 16 public and private school and college campuses, 22 churches, a dozen medical facilities and offices, hundreds of thousands of square feet of office, lab, shop and warehouse space and hundreds of thousands of square feet of retail outlet space and restaurants just within approximately a 5 mile radius of my home.

Even this tiny little slice of the Houston metro area is a high capacity incubator.

Of course, there are very well-known and very predictable large scale patterns of contact behaviour involved at all of these human meeting and interaction places, and that has got to help with the calculations.