More years of data for Accuracy of Win Predictions

30Dec

My first reaction after I’ve finished post about Accuracy of Predictions for Wins in 2010/11 was this: “that’s nice but what about consistency of authors?”. Are writers and bloggers consistent over time or is it basically a semi-random outcome every year? Is it an educated guess or just a guess by educated people?

To answer this question I simply needed more data and because I didn’t want to wait a couple more years I went the other way – into the past. I collected some predictions for NBA standings before 2009-10 season and before 2008-09 season [btw, it’s not a closed list so if you want to add any predictions let me know in the comments]. Then I again compared them to actual results by Root Mean Square Deviation [read post about 2010/11 for explanation]. If you want to play with numbers or check any single entry just download NBA Predictions Accuracy 2008-2009-2010.pdf but if you are interested only in the results here they are…

Those lists are unfortunately shorter than I hoped for because:
– sadly APBR forum was hacked so their batch of past predictions vanished forever,
– Wages of Wins Network either didn’t exist or didn’t post any win predictions for 2009/10 or
I can’t quickly find them [they merged and relocated],
– surprisingly few authors even published their win predictions 3 years ago or google just can’t find them easily.

I didn’t get a big sample but it’s still bigger than just one year so
I think I can make a couple of additional points…

– last season had a higher level of accuracy than two previous ones. Was it a classic fluke year? ;-)

– Vegas odds-makers clearly know what are they doing.
So I’m just wondering… as a blogger without any good system shouldn’t you just copy their predictions with some small changes regarding those teams where you feel veeery strongly they are wrong?

– wisdom of the ESPN crowd had two really good years in a row – coincidence or this system works?
Again, I wish there was more data but they started it two years ago.

– John Hollinger had two top seasons and one average so either he is really good at this game or very lucky.

– WARP, Wins Produced and Win Shares did a surprisingly poor job in the last 3 seasons so even though it doesn’t tell us anything about those ratings I have a theory: you shouldn’t base your predictions on numbers from previous year and just plug them into a system because there are too many unpredictable variables… you really should get a little wild and add some changes not based on numbers. True or false?

– overall, it DOES seem to be at least somewhat consistent… although again I wish it was a bigger sample.
We will get there in a couple of years ;-)

I’ve calculated RMSE for your numbers and results look pretty bad (JohnH=11.64, DW21=11.29, MikeG=11.14, Mtn 10.51) but without any context it’s hard to tell whenever it was just a difficult year for predictions or it was a learning process or what.