Who's crystal clear and who's all wet when it to comes to Minnesota weather forecasters?

Tuesday, December 6, 2011

A New Weather Grader in Town

For those interested in seeing how local forecasters perform in select areas of the country (including Minneapolis/St. Paul), there’s a new service that can tell you just that. Weatherist.com, which launched in Beta in late November, aims to assess the comparative accuracy of television forecasters and select other sources.

A sample "Leaderboard" of Minneapolis/St. Paul forecasters from Weatherist.com

The site features a number of interesting features including the ability of individuals to post their own forecasts and have them analyzed alongside the pros. The site also develops a “most probable forecast” that’s derived from the performance of “hot” forecasters.

According to Dave Chung, co-founder at Weatherist, the goal is to bring accountability to the weather forecasting industry and to provide a critical voice for consumers in an industry that has never had one. We applaud Weatherist’s ambitious goal and hope it’s successful.

If you’re curious to see an approach that’s far more scientific than ours, do check it out. And, as a new startup, Weatherist.com is interested in gathering as much feedback as possible. Feel free to leave your comments below.

12 comments:

Bill,I'm guessing your the blog creator for minnesotaforecaster(since all the posts are posted by you)well I just stumbled upon this site and wanted to tell you how terrific it was,as a weather nut I find this all so fanscinating,the posts, the comments,the polls,the grading system you have,but I am most interested in this fantasy weather I've read in some of your readers comments,but I am looking all over this site and do not see how one can enter into it,I find myself guessing all those winter events(first snow,season snow,below zero days,etc.)and see how I do by seasons end,it would be pretty interesting to see how I fair aganist others.Also went to above website(weatherist.com)to me looks very confusing,like what does all the %'s mean,where were they derived from,is there away to see what they actually forecasted for a specific day and match it up to actual.?

Not sure what Weatherist is really telling us. Find it confusing and not very meaningful. Need more data to understand what's going on there. And we all know inaccuweather is not good so not sure how that rates well.

I can't recall the site out of Ohio that does a similar function and is monitored by several weather companies including TWC, etc. Interesting is that their findings consistently has Accuweather behind the NWS, TWC, Intellicast, etc. Should be interesting to see the rest of the winter results.

What masks MNForecaster very useable is its objective and subjective view which the average user/viewer/reader understands and relates to rather than percentages and exact values instead of ranges, and also changes (amendments) to a forecast.

@DDwx Thanks for the kind words. I guess I think of my "shop" as sort of a boutique. It would be nice to have a systematic way to capture temp forecasts, but I think we manage to cover the ground we need to.

I believe the site you're referring to in Ohio is Forecastadvisor.com. They do a great job and provide very interesting stuff for weather enthusiasts.

Dave from Weatherist.com here. Thanks everyone for your comments and feedback. As Bill mentioned in his post, we just released our beta last week and are listening to your feedback very closely. I just want to point out a few things to help clear up any confusion:

Our ratings are derived from comparing the forecasts with the actuals reported by NWS. We use a relatively simple formula to generate the % score that you see. I should note that the score does not mean the forecaster is "84%" accurate - it is purely a measurement we use to compare one forecaster to another. That being said, we believe that relative performance is more important than absolute performance - particularly since our goal is to generate The Most Probable Forecast.

The "overall" score that you see in the leaderboard is really just a crude way to combine the scores of all the metrics we collect. I find it actually more interesting to look at the ratings for each individual metric. When you do this, you can see that AccuWeather hasn't been very accurate.

We realize that we still have a lot of work to do. We are planning on adding more detail to our methodology and at the same time continuing to refine our process. We greatly value your feedback - so please, keep it coming.

Feel free to continue this thread or visit our site at www.weatherist.com and fill out the feedback form by clicking on the blue button in the bottom-right corner.

Speaking of grading weather casters for accuracy, Novak sees a chance for "significant" (I assume that means shovel/plow) snow next Wed/Thu. The Strib, in turn, says no significant snow accumulations through the end of next week.

Umm...I would equate that to one person answering "True" while the other says "False" on the same test question. Someone's going to be very wrong.