Tuesday, October 15, 2013

Corsi, shot quality, and the Toronto Maple Leafs

The Toronto Maple Leafs had a decent season in 2012-13, finishing fifth in the conference and making the playoffs for the first time since 2004. But, perhaps, we fans of God's Team shouldn't get too optimistic. For months now, hockey sabermetricians have been arguing that the Leafs were still a bad team -- a bad team that just happened to get exceptionally lucky.But ... I've been fiddling a bit with the numbers, and I'm not sure I agree.Before I get to my own case, though, let me tell you why the consensus says what it says. First, Sean McIndoe has an excellent Grantland article that summarizes the issue. Second, when you're done that, here's my own summary, which is a bit more statistical. -------One of the new sabermetric statistics in hockey is the "Corsi" statistic. Corsi is much like the NHL's official "plus-minus", but, instead of goals, it counts shots. (Not just official shots on goal, but all shots directed at the net.) A player's Corsi is the difference between team shots for and shots against while he's on the ice in 5-on-5 situations. Applied to teams, Corsi is just the difference between shots taken and shots allowed. The idea is, that there's a lot of luck in terms of whether shots actually go in the net. So, instead of goals, you can better measure a team's talent by looking at shots. It's the baseball equivalent of using Runs Created instead of runs scored. In the baseball case, you eliminate "cluster luck" (as Joe Peta calls it), to get closer to true talent. In the hockey case, you eliminate "bounce in off a player's butt luck" (among other randomness) to also get closer to true talent.Corsi is a very good predictor of team success. In one study, Corsi correlated with team standings points at r=0.62, which is pretty high.So, the consensus is that if a team's Corsi doesn't really match their won-lost record, the difference is probably luck, and the team shouldn't be expected to repeat. Last year, Toronto did not look good in the Corsi standings. In 5-on-5 situations, they took only 44.1 percent of the shots (meaning their opposition took the other 55.9 percent). That was worst in the NHL.So how did the Leafs win so many games, finishing in the top half of the standings? Even though they took few shots, the shots they did take went in at an exceptionally high rate. The Leafs had a 10.56% shooting percentage (goals divided by shots on goal), the highest in the league. No other team was over 10. The league average was roughly 8, with a standard deviation of roughly 1, so the Leafs were well over 2 SDs above the mean.Now, you might be thinking: "Sure, the Leafs took fewer shots, but maybe it's just that they took BETTER shots, and that's why they did so well. Corsi counts all shots equally, whether they're weak shots from the point, or point-blank shots with the goalie out of position. How can you call the Leafs a bad team without also checking their shot quality?"The sabermetric community responds that, if you look at the evidence, shot quality seems to be luck, rather than a skill that varies among teams to such a large extent. If shot quality were actually non-random, a team with a high shooting percentage this year would tend to also have a high shooting percentage next year. But that doesn't seem to happen. One study, by Cam Charron, divided teams into five groups based on their shooting percentage this year. The following year, all five groups were almost identical! If you look at Charron's chart, there actually is a small effect that remains, but it's only about 10 percent of the original. In other words, you have to discount 90 percent of the differences between teams.Another study computed shooting percentages for individual players. There were substantial differences, but: (a) only two players had shooting percentages higher than the entire Leaf team last year; (b) there's still luck in the individual numbers, so even those players probably don't have that kind of talent; and (c) those players may be taking the team's higher-quality shots because of their role, rather than because they create those shots. So, it doesn't seem like the Leafs' 10.56% could be actual talent.So, the argument in a nutshell: -- the Leafs took very few shots-- teams that take very few shots are usually bad-- the Leafs weren't very bad only because of their exceptionally high shooting percentage-- an exceptionally high shooting percentage is usually luckTherefore, the Leafs were probably just a bad team that got lucky.------I'm going to argue that that's not necessarily right. There's another explanation that works just as well.I'll give you that explanation now, in case you don't feel like reading the numbers to follow. Actually, instead of the explanation, I'm going to give you an analogy, which might convey it in a more meaningful way.A company pays its commissioned salesmen in cash, normally in 20 Euro bills. The value of a Euro fluctuates, so the workers have learned that you can figure out who made the most money just by counting the bills. If Joe has 35 bills, but Mary has 38, then Mary made more money than Joe.One month, the paymaster happens to have some extra British currency he wants to get rid of, so he substitutes pounds for Euros in Mary's pay envelope. A pound is worth more than a Euro, so instead of 42 banknotes of 20 euros, Mary receives 35 banknotes of 20 pounds. Also, Mary wins the "salesperson of the month" award.The sabermetricians seize on this. "We have found that the statistic called 'Borsi,' which is the number of banknotes received, is one of the most realiable indicators of sales performance," they say. "But, this month, Mary's 'Borsi' was only 35.""Sure, Mary won the award because her 35 Borsis were more valuable than normal banknotes. But, our research shows that receiving British pounds is not a repeatable skill -- salespeople who receive pounds this month tend to revert back to Euros next month. Therefore, you can't credit that to Mary's talent. Therefore, she was lucky to make as much in commission as she did.""In summary:-- Mary had a low Borsi-- salesmen with low Borsis are usually not productive-- Mary did well only because her banknotes were worth more than normal;-- receiving high-value banknotes is usually just random chance.Therefore, Mary was lucky."See the flaw? Each of the four points above is actually true. But what the analysis doesn't consider is that, even though receiving high-value banknotes is luck, there is a real, non-random relationship between that luck, and the number of banknotes received. So, the analysis correctly adjusts for "high-value banknotes luck", but not the "too few banknotes luck" that corresponds to it exactly.I suspect the same is true for Corsi. It was luck that the Leafs scored on more than 10 percent of their shots, but that luck is actually tied to the fact that they took fewer shots. OK, here we go.--------If shooting percentage is almost all luck, the implication is that it's not something a team controls, or can even *choose* to control. It's like clutch hitting -- just randomness that looks like there's something real behind it.In that case, you'd expect every team to be around the league average of 8%, in all situations. Shot quality must be about the same for all teams. Intuitively, you might think some teams are good enough to have more breakaways and blind passes, while other teams take a lot of harmless shots from the point. But, the data show otherwise.Except that ... there ARE situations in which shooting percentage varies meaningfully from 8%. For instance, a team's shooting percentage depends heavily on the score. Here are the situational averages for the six years from 2007-08 to 2012-13, with every team's six year total weighted equally. 7.60% ... down 2+ goals7.75% ... down 1 goal7.52% ... tied8.40% ... up 1 goal9.19% ... up 2+ goals

(By the way, all the numbers in this post come from David Johnson's data pages at hockeyanalysis.com. Thanks, Mr. Johnson ... never could have figured all this stuff out otherwise.)

Why such big differences? My guess is ... when a team is behind in the game, it changes its style of play. It probably presses a little more in the other team's zone, trying for better opportunities -- which is why its percentage rises a little bit. On the other hand, when it presses more, that increases the chance of being caught behind on defense. That gives the other team more chances at odd-man rushes and breakaways. Which is why the opposition -- the team that's up 1 or 2+ goals -- sees its shooting percentage rise significantly, all the way to 9.19%.I just made that explanation up, off the top of my head ... some of you guys know hockey a lot better than I do, so there's probably a better description of what's going on that would be more plausible to a real hockey strategist.But, regardless of the details of the explanation: how you play does indeed seem to influence shot quality. It's not all just random.So: why isn't it possible that the Leafs' numbers are the result of style of play? Couldn't they be deliberately playing the "up 2+ goals" style of play all the time? If they're the only ones doing that, one team out of 30, it would be too small to show up in the statistical studies, and it would still look like shot quality is 90% luck.I'm not saying they *are* doing that, just that it's *possible*. ------Now, when you look at the above numbers, you might think: it must be the team that's UP that changes the style of play. Because, that's the team that looks like it gets the much bigger advantage in shot quality! The team that's behind probably doesn't like it, but has no choice.But then you'd ask the obvious question: if playing that style is so beneficial, why don't teams do it ALL THE TIME, instead of just when they're in the lead?The answer is: it's not that beneficial. The higher shooting percentage is offset by the fact that the teams in the lead take fewer shots -- that is, they have a lower Corsi. Here are the percentages of (Corsi) shots taken based on score:57.0% ... down 2+ goals54.1% ... down 1 goal50.0% ... tied46.0% ... up 1 goal45.1% ... down 2+ goalsThis makes sense too. If you're behind in the game, you have to concentrate on offense more than on defense. The higher offense means you'll be taking more shots. But, as we saw, the lower defense means you'll be giving the other team better quality shots. The quantity and quality factors go opposite ways, and they roughly cancel each other out. How do we know that? First, it just looks like it to the eye; if you put the numbers together, one goes up roughly at the same rate that the other goes down:57.0% ... 7.60% ... down 2+ goals54.1% ... 7.75% ... down 1 goal50.0% ... 7.52% ... tied46.0% ... 8.40% ... up 1 goal45.1% ... 9.19% ... up 2+ goalsBut, more empirically, we know from goal scoring, which remains roughly even between the teams regardless of the situation. Here are the rates of goals scored (team and opposition) per 60 minutes of even strength play:2.42 - 2.32 ... down 2+ goals2.39 - 2.26 ... down 1 goal2.21 - 2.21 ... tied2.26 - 2.39 ... up 1 goal2.31 - 2.42 ... up 2+ goalsIt's not perfectly even ... being down actually gives you a small advantage in future goals. That might be random, but it might be real. There's a plausible reason it might happen. The team that's ahead has an interest in limiting scoring by wasting time. It might be worth playing a style that gives the opponent a slight advantage in expected number of future goals, if that's offset by a lower probability of getting a goal in the first place. -------So, I can say again: isn't it just possible that the Leafs are deliberately playing the "up 2+ goals" style during tie games?Here's a comparison of the two sets of numbers. The first is the "leading by 2+" from the above chart, and the second is last year's Leaf team with the score tied:2+ games: 43.1% Corsi. Leafs: 43.8% Corsi.2+ games: 9.19% shooting. Leafs: 10.82% shooting.2+ games: 7.60% opposition shooting. Leafs: 8.64% opposition shooting.2+ games: 2.31 goals to 2.42. Leafs: 2.96 goals to 2.80.Three of the four numbers are roughly the right magnitude and direction: lower Corsi, much higher shooting percentage, slightly higher opposition shooting percentage. The goals thing goes the wrong way, though. Still, reasonably consistent with the Leafs choosing to play a "concentrate more on defense and jump on opposition mistakes" style of game. --------The situational numbers suggest that one way to lower your Corsi and raise your shooting percentage is to consciously choose defense over offense. But it can also happen randomly, because opportunities are random. Every player, every time he has the puck, has to decide whether to shoot or not. Some days, you might have cases where the best option is to shoot. Other days, you might have options where the best option is to pass, in hopes of a better shot.Imagine a player has the puck. If he shoots, he has a 5% chance of scoring. If he gets the puck to his teammate on the other wing, the chance goes up to 10%. Should he pass? If he does, there's the risk that the defense will intercept the pass and take over. Given these numbers, he should only choose the pass if there's a better than 50/50 chance it'll get through the defense. Now, options like that present themselves all the time, with different probabilities. Sometimes, you have only a 15% chance of completing a pass (across the slot through a bunch of legs, say), but, if it works, there's an 80% chance of the shot going in. Sometimes, it turns out nobody is open, and the 4% wrist shot from a bad angle is your best option.All that, to a certain extent, is random. Perhaps one day, by chance, everything is a shot. You may take 60 Corsi shots, at 5% each. Your shooting percentage will average 5% -- 3 goals in 60 shots.The next day, randomly, everything is a 50/50 pass to a 10% shot. You'll make 60 passes. 30 of them will be intercepted, and 30 of them will turn into Corsi shots. Your shooting percentage will average 10% -- 3 goals in 30 shots.Corsi will think your first day was a much better day: you had twice the shots! But ... it wasn't. One day you had more low-probability shots, and one day you had fewer high-probability shots. One day you had 60 five-dollar bills, and the next day you had 30 ten-dollar bills. My example is too extreme ... your swings won't be that wide, from 60 passes to 60 shots because of random changes in offense and defense patterns. But there will be SOME random variation in opportunities, which means there will be SOME random variation in Corsi that will move shooting percentage the other way. In this scenario, it is absolutely true that shooting percentage is not a repeatable skill. But that doesn't mean that you didn't earn the extra goals. You earned the extra goals because the "lucky" shooting percentage came from "unlucky" reductions in shots taken.--------Here's more evidence that Corsi and shot quality are inversely related.I again looked at the last six years of the NHL, 180 team-seasons. The correlation between shooting percentage and Corsi was -0.22. When I took only the 36 most extreme shooting percentages, the correlation was -0.37. In the regression, each point of shooting percentage decreased Corsi by 0.785. That means the Leafs' shooting accounted for around 2 points of Corsi, enough to move them from 44.1 (last) to 46.1 (third-last).I think that's not enough of an adjustment, that the Leafs are more extreme than the regression suggests. That would be possible, I think, if the Leafs are different from other teams in some way other than random variation in shots.But that's just my gut. And I may be improperly biased by knowing, in advance, that they had a positive goal differential. --------After looking at all these results, my overall view of Corsi is that it's decent enough to tell us something useful, but way too biased by situation and shot quality to be taken seriously on its own. As for the 2012-13 Leafs, I suspect shot quality is the biggest explanation for their low Corsi and high shooting percentage. Not that I'm dismissing luck -- any time you have an extreme result, without a full explanation, luck is probably involved somewhere. So, yes, I think the Leafs were a bit lucky in their shooting. But, the rest of it, I think, was something real. Specifically, I think it was some combination of:1. The Leafs playing a more defensive style, allowing them to capitalize better on opposition mistakes;2. Taking fewer low-percentage shots, and passing instead; and3. Random variation that made passes a better option more often.And I say that because, from all the results I looked at, it seems that Corsi and shot quality are indeed related to each other. If you accept Corsi, you can't dismiss shot quality. To a significant extent, they're opposite sides of the same coin. (There are seven parts. This is Part I. Part II is next.)

Adding to Cam's statement, as others have pointed out on twitter, it should also be noted that performance changes during different situations are almost certainly due to changes in BOTH teams' strategies.

In other words, it's certainly true that some teams will play different styles during games - for example: Ottawa was a high event team last year who was positive in possession, while the Devils were super low event but also positive in possession. But the 2+ lead situation is a product not just of one team opting to play defensive - see the Devils - but of the other team also having to take the greater risks because they NEED to score.

The up 2 team gets those high % shots because the other team is forced to risk more mistakes....but if the game is still close, that's not the case. This is why you don't see consistent high shooting % teams from year to year, nevertheless high shooting % AND badly outshot while the situation is close or tied.

First, teams don't care about shooting percentages, just goals. Why would they care if the goals come off lots of low % shots or fewer high % shots?

Second, you'd think some teams would still chose the high % strategy. That's a puzzler. I was kind of expecting someone to say, "hey, that makes sense, the Leafs were doing X Y and Z before, that I don't remember anyone else ever doing!"

So far, nuthin'. :)

3. As I say in the post, it could be just random luck that the Leafs took higher percentage shots. That is: yes, it's luck, but, no, that doesn't mean they overperformed. Because, the luck was that they got fewer $10 bills instead of more $5 bills.

I guess the counterintuitive point is: the high % could indeed be just random luck, without it meaning that, with different luck, they would have scored fewer goals.

Also Phil, Cam here actually tracked scoring chances during Leafs games, and found the Leafs to be outchanced as well as outshot. (Here's the 3/4 season post, i can't find the full one: http://theleafsnation.com/2013/4/4/maple-leafs-scoring-chances-through-36-games).

Moreover, the Leafs' coaches have insisted that "no the shot #s are wrong, we controlled possession", despite Leafs fans actually going back and timing time in the offensive zone.

Now it's possible that maybe the Leafs were getting even higher % chances than the opposition, even when they were being outchanced overall. But it certainly doesn't seem likely.

Probably because it still isn't necessarily a "good" strategy. The leafs were a slightly above average team last year, who's to say they wouldn't have been better if they employed a corsi-centric strategy? The underlying numbers would look a lot different certainly. But, I don't think Phil is arguing that the Leafs strategy is ideal, just that perhaps the Leafs weren't 'lucky' to be a top 12 team last year.

garik16: agreed, score effects are the result of both teams' actions. But it seems like one team could do SOMETHING, even if not exactly the same thing, that would have the effect of fewer but better shots.

Dave: yes, thank you. I should say that explicitly: I'm not saying the "high shooting percentage" strategy is good or bad. But I *AM* saying that it leads to a lower Corsi and higher shooting percentage, with little change in relative scoring.

"The up 2 team gets those high % shots because the other team is forced to risk more mistakes."

There's also the element that worse goaltending allows more goals, so a team up 2 is more likely to be shooting against inferior goaltending and a team down 2 is more likely to be shooting against superior goaltending.

It's similar to how some of the splits for baseball pitchers for when there are men on base versus no men on base can be accounted for by worse pitchers allowing more baserunners, so men-on-base situations are more likely to occur with worse pitchers.

This could be one of the reasons we should be tracking passes as well as shots. If teams are employing a strategy that is intended to produce better shots (in a trade off with more shots) we should be able to see that in how they move the puck around, especially in the OZone.

My best guess is that the Leafs shooting % in 12-13 was basically pure luck, but that their defense deserves far more credit than their 30th by far CA60 indicates. Not to mention that goaltending is a skill and Reimer was one of the team's best players last year.

If you actually look at where the Leafs sit in their Corsi Events for and their SH% by situation last year I'm not sure this makes sense.

Up2+: CF60=44.8 (17th), CA60=72.8 (29th), SH%=11.46 (6th)

Up1: CF60=45.8 (24th), CA60=67.4 (30th), SH%=9.17 (11th)

Tied: CF60=50.7 (24th), CA60=65.0 (30th), SH%=10.82 (1st)

Down1: CF60=60.4 (11th), CA60=59.0 (28th), SH%=12.59 (4th)

Down2+: CF60=60.7 (13th), CA60=49.8 (28th), SH%=8.65 (8th)

The main reason I'd suggest the team isn't "playing like they're up by 2" all the time is because if you look at the number of events they're allowing - they're atrocious no matter the game state.

Defensively this team is bad and it doesn't matter whether they're up1, up2+, down1, down2+, or tied. The correlation between CF60 and CA60 is minimal, so while I'd suggest riding the percentages offensively might make some sense, this team's brutal defensive results doesn't really inspire confidence.

On the flip-side of the coin (from that LACK of variation in their D results), is their offensive shot production. Their CF60 results are obviously far superior when they are down a goal or more... but the percentages don't shift drastically in terms of SH%.

In fact they actually shot BETTER in comparison to the rest of the NHL when down than when up... despite increasing their shot attempts, which if I'm understanding your logic sort of goes against your suggestion.

They were taking MORE attempts and scoring at a higher rate. I just fail to see how this holds up when you look at their results in detail.

My first thought, which two Anonymouses (anonymice?) pointed out, is that sorting by goal state probably sorts you into teams of different quality. Teams that are up by 2 are likely the better team that night and more likely to score. It's also likely to be later in the game as opposed to earlier if one or two goals have been scored, so empty net goals and holding the puck to run time (and thus not shooting) become issues for the team in the lead.

Corsi is only one aspect of overall performance. Goalie play and Special team's play can have a huge impact especially during the regular season. So there are two arguments getting merged into one discussion. Toronto's 5o5 shooting percentage was high for last season. That has a whole lot of randomness to it.

Toronto got really good goalie play and in particular Penalty killing goalie play last season. You can win more than your far share of games just on that. Toronto has not managed to maintain that high 5o5 shooting percentage in either the playoffs last season or the start of this season. Toronto’s goalie and in particular goalie penalty killing play is even better this season, though it probably won’t remain at these levels for the course of the season.

And the “shot quality vs. shot quantity” arguments are really annoying. A quality shot tends to generate more rebounds which generate more shots. A non-quality shot tends to be a turnover. Over the course of a season's worth of games it is next to impossible to win the quality scoring chances but lose the overall shooting chances. Anyone that has attempted to look at that has come to that conclusion. At this point, anyone making the shot quality argument needs to show proof.

As for "up two shooting percentages," a team down two tends to play a more desperate and aggressive style of hockey. This leads to odd man rushes for the team that is up which increases that team’s shooting percentage. Since you can’t force your opponent to play that aggressive when the score is close you can’t play an “up two” strategy at all times.

We are told that the leafs are the only team to be in double digits SH%. According to Yahoo stats 6 teams were double digit. Apparently the leafs leading the East in SH% at 11.5 , Pitt 11.3, TB at 11.1, Wash 10.8.

So the premise is wrong.

Secondly, Corsi as a predictor of points is a distant second GF/GA ratio.The Goals For Goals Against ratio has the best correlation. Surprise Surprise! LOL

The reason it is no surprise is that it includes all the statistical information in shots, Sh% Sv% etc.

Lastly, Washington had similar stats to leafs so another team figured it out. Another mistake.

According to Yahoo leaf SV% of 91.7 was 4th in the east at SA/G were 13th . Compare with Wash SV% was 91.6 ranked 5th and SA/G 32.3 ranked 14th. Washington had the same quality of goaltending according to the stats following the Authors logic.

Washington was very similar to the leafs in SV% SH% SF and SA. Apparently, this combination of stats generated 25% of the East’s playoff teams.

Another thought - if a team is consistently doing something to get better shots, then they might have fewer shots blocked (if you assume 'better' means less likely to be blocked). However, if I look at the Leafs from last year and calculate their blocked shot percentage by using Corsi shots minus Fenwick shots divided by Corsi shots, the Leafs were 4th in the league - their shots were blocked a lot. That doesn't necessarily strike me as a sign of a team that is reducing shot numbers in order to get better shots.

I haven't thought it all the way through, but there might also be an explanation for the correlation you see between shooting percentage and Corsi. Shooting percentage only counts shots on net. Both goals scored and shots on net will increase with Corsi, but if shots on net increases more (or is more reliably linked to Corsi), then you might get a negative correlation because shots on net is in the denominator for shooting percentage.

MaxPower417: Yup, that's a good point ... someone posted that at one of the other discussions. Turns out that last year is negative, the year before is negative, but the ones before that are slightly positive. IIRC.

Which is not what I expected ... but overall, it's still fairly negative. More investigation required.

(BTW, in a short season, you'd expect the correlation to be weaker, not stronger. Randomness brings it closer to zero. So, last year is perhaps more important, not less. But, that's not a big deal.)