Sunday, July 12, 2009

Variations on a Theme: Different Formulations of Quality of Competition

One very nice innovation on Quality of Competition came from Jonathan Willis at Copper 'n Blue. In order to calculate Quality of Competition, you normally need 1) TOI totals for every player; and 2) head-to-head icetime totals for every player combination. This data is only available for the NHL, and it's only available for the last three years. What Willis did was to use goal events as a proxy for icetime. That is, in the absence of TOI, Willis assumed that the total number of goals a player was on the ice for was a reasonable proxy for the amount of time he spent matched up against specific opposing players.

QC0 is Quality of Competition as calculated on my site. QC1 is the same TOI-based calculation, but it includes only the off-ice stats for games the player played in (for simplicity of calculation, QC0 includes all of a team's games.) QC2 assumes total TOI is known, but uses goal events (+ and -) as a proxy for head-to-head icetime. QC3 is the same as QC2, but uses games played as a proxy for TOI.

Some results are similar: everybody loves Horcoff and Moreau; everybody hates Stortini. But there are some substantial differences between these systems too. The TOI based ones favor Sheldon Souray, Dustin Penner and Tom Gilbert, while the proxy systems like Jason Strudwick, Ladislav Smid and Andrew Cogliano.

So my question is - given that I see maybe one Oilers game per season - which system seems to be closest to the right answer? Is the difference tolerable?

I think that the TOI system is going to be necessarily better. With that said, I've always been vaguely troubled by the fact that a player could be perceived as being more difficult competition if he was on a different team. As I understand your system, Crosby on the 05-06 Pens is going to have a higher ranking than he would on the 05-06 Wings.

That seems off to me if we're talking about his absolute level. The way most of us use this stuff though is purely for purposes of understanding how a player is used within his team, which isn't a criticism of the system, only a recognition of the limitations. If that's what we're using it for, the TOI way is probably better. Willis' system - which looks decent stacked up against this, which is good in terms of getting QoC measures for other leagues - is necessarily limited.

I tend to agree with Tyler for the most part. The time-on-ice method is better at evaluating which players are being trusted with more responsibility.

The proxy methods (QC2 and QC3) are helpful for looking at other leagues but there are some Oiler players that look a bit wacky using them. Smid, Staios and Strudwick, especially, seem like they are ranked too high in QC2 and QC3 while Hemsky and Gilbert are ranked far too low in QC3.

That said, I think Willis is using a different proxy method based around pts/gm instead of GF-GA. He awards "difficult minutes" to the players that score on/are socred on by forwards that have high pts/gm and "easy minutes" to the players that score on/are scored on by forwards that have low pts/gm. I'm not sure how that might change the results.

Poring over last season's numbers of my beloved habs, I couldn't help but notice two guys in particular, Bouillon and D'Agostini. Both were, from what I can recall, fairly reliable players defensively. But both Bouillon (.889) and D'Agostini (.895) had low save % while on the ice, compared to the team in general that is (.922). For both players, this added up to 10 extra goals against, thus driving down their RATING, hence driving down the QUALTEAM of those who shared lots of ice with them. Andrei Kostitsyn also had some kind of a raw deal, coming in at .907/-7, while Lapierre was kinda blessed at .936/+7.

I guess when it comes to QUALCOMP, those kind of blips even out, and thus my comment is completely off-topic... In fact, maybe this is by design and I just missed the point, but still: wouldn't QUALCOMP and RATING be more, uh, reliable were it based on Corsi or even shots F/A instead of goals?

I'll third Tyler and Scott's comments. The TOI system is almost certainly better by a substantial degree.

My system seems to do better with the forwards than the defensemen, which makes sense intuitively given that there's a greater spread up front (12 vs. 6) and also that MacTavish shuffled around the defensemen who faced the top matchups this year a fair bit.

TOI's the system to use, but I'm fairly pleased with how the goal events proxy stood up to this particular test.

The other difference is grading the competition - are the numbers as they're setup now shoing opposition graded by adjusted +/- or by points/gm?