Saturday, August 29, 2009

Some hockey fans think playoff wins are the best way to evaluate a goalie. The other stats don't matter, they say, because it doesn't matter how many saves you make but when you make them, and the teams that win are the teams that get important saves from their goaltenders at the most important times in the game. This means that stats are just excuses for losers. If those goalies played better when the game mattered then their team would have won.

I've already dealt with some of the "big saves" argument in this post that points out that most of the game of hockey is played with the score close. However, it remains at least theoretically possible that teams and goalies could optimize their goals for and against to improve upon their win/loss record. Hockey games are won and lost by goals for and goals against, but a team that wins the first game of a playoff series 8-0 and loses the second 2-1 has the same record as their opponents, despite a much better goal differential. We know from observed results that in the long run most of those differences even out and that records can usually be predicted quite accurately by goal differential. However, the possibility does remain, no matter how rare it might be, that a goalie could "stretch" their teams' goals farther than normal and win more games by playing better in key situations.

It seems unlikely to me in today's widely-televised, big-money NHL that a goalie would be goofing off when it didn't matter and bearing down only when it did, but what about the goalies of yesteryear who were playing for fun and had to work other jobs in the offseason? Did some of them only care about winning and adjust their effort accordingly? If that was the case, then maybe they could be legitimately considered better than another goalie with equivalent stats because they did more to help their team win hockey games. Is this something that goalies actually do or have done in the NHL playoffs?

The Pythagorean expected points equation (GF^2/(GF^2+GA^2)) allows us to predict what a team's record will be given their goals for and against. If a goalie has exceptional timing in terms of allowing goals, then they will allow more goals when it doesn't matter and fewer when it does matter. Doing this should allow the team to outperform its expected win total.

Let's give an example: If a goalie plays in 20 playoff games and his team scores 50 goals and allows 50 against, then it is difficult to claim that the goalie is a clutch winner if his team's record is 10-10. That is of course exactly where we would expect the team to end up. If they instead are 15-5, then that must mean they were winning small and losing big, and either they got lucky or the skaters and/or the goalie were doing more to contribute to winning than the basic stats would suggest.

For each playoff season where the goalie played over half of his team's minutes in the playoffs, I took the total goals for and against during that playoff season, used that to project the team's record, and then compared it to their actual wins and losses. Over their entire career we can see if their teams did better or worse than expected. I wasn't able to split out individual goalie performance because game summaries aren't always available, but this shouldn't have big effect since most would have played the vast majority of their teams' minutes.

This makes for a very simple analysis, as all we need are win/loss records and goals for and against. We can thus compare goalies throughout the entire history of the league, all the way from Benedict to Brodeur, including goalies from the early years for whom we have no shot data or save percentage stats available.

Great teams tend to overperform their expected win rate, especially in the playoffs. This is mainly because teams tend to play to the score more in the postseason. If they gave it a full effort for the entire 60 minutes, a dynasty team would likely blow out their opponents by an even greater margin that might be more representative of their actual dominance, but teams that are ahead usually trade offence for defence to increase the chances of holding onto their lead. Therefore goalies who played on very strong teams would quite possibly show good results here even if their own performance was consistent across different game situations.

Just to be clear, this metric has little to do with a goalie's overall performance. A good or bad result doesn't prove anything, but only tells us where to take a closer look if we are trying to see if some goalies contributed more to winning games than their basic stats would imply. A more in-depth analysis would be required to confirm or deny the hypothesis. I'm not at all convinced that any goalies have consistently shown good timing or an unusual ability to raise their level of play in high-leverage spots. I think the ability of the rest of the team to play to the score is likely far more important than a goalie's clutch performance. Some (maybe even most) of these results might be entirely because of team factors, and some of these results might be entirely because of luck. Be aware of that as you draw your conclusions.

Here are the results with the goalies grouped into tiers, sorted by number of wins above expected:

All of these guys are well known for their playoff successes. They were all probably above average in the postseason, but they all also had the benefit of playing most of their playoff careers on very strong teams.

I have already looked at some of Grant Fuhr's peak playoff seasons in Edmonton in some detail, as well as Patrick Roy's later career. I don't think the evidence suggests that either of them were making a huge impact on their team's win/loss record by making the so-called "big saves". I think the Oilers and the Avalanche were similar in that they were both high-percentage scoring teams that did not dominate on the shot clock and played to the score, which is the kind of team that might very well win more than their goal differential suggests. I would be interested to look at Roy's Montreal game results in more detail, but unfortunately most of those box scores aren't readily available at the moment.

Fuhr and Smith are goalies who are usually subjectively rated well above what the statistical record suggests they deserve. Just like Brodeur, that makes them interesting objects of study, to try to assess whether there is something extra hidden in the numbers. I'm still not entirely sure what to make of Smith, I'm always skeptical about goalies who have reputations that depend primarily on their team playoff successes. However unlike Fuhr Smith actually has a pretty strong save percentage record, although he played mostly in a diluted league and didn't outperform his backups which suggests that there was a strong team context.

Giguere has a win total that is far above expected, primarily as a result of his 12-1 record in OT games. I don't know whether he is going to be able to keep repeating that close game success, but it is impossible to deny that Giguere has had some great clutch performances to this point. Broda is considered to be one of the best playoff goalies ever, so I was expecting him to do well by this measure. Worsley does surprisingly well although his outperformance is nearly entirely on the Canadiens, which suggests that his team was helping, something we should also keep in mind when evaluating his Montreal predecessor Jacques Plante.

Considering his teams I don't think Cheevers' playoff record is all that special. Barrasso's position is entirely from the 1991 and 1992 Cup runs (+5.8 wins in '91/'92, -3.3 wins for the rest of his career). Who was the real clutch player, Barrasso or Mario? I know what my guess would be. I don't think many observers would consider Andy Moog clutch, but he won more than expected, suggesting that his Edmonton or Boston teammates were good at playing to the score.

I was expecting Hasek, Brimsek and Parent to finish a bit higher than they did. In fact, Hasek's relatively ordinary ranking suggests to me that these results have a lot more to do with the team than the goalie. With Hasek in net the Sabres won pretty much exactly what they were projected to win, even though Hasek had an astonishing .949 combined save percentage in Buffalo in overtime and in third periods that began tied or with the teams within one goal of each other. Hasek was also almost unbeatable when Buffalo was ahead late in the game (the numbers are all in this post). It's pretty hard to get much more clutch than that, and yet if you plug Buffalo's goals for and goals against into the Pythagorean formula you get a winning percentage that is within .002 of the actual observed result.

All of these guys won pretty much exactly as much as the goals for and goals against predict they should have. Again that doesn't mean they didn't play well, just that there is little evidence to suggest that their teams won more games than they should have with them in net. Therefore to me it makes sense to base our evaluation of them on statistical measures of their individual performance rather than their team success. Chris Osgood may have a lot of playoff wins and Cup rings, but that doesn't mean he is an unusually clutch player. The opposite would apply for Roberto Luongo.

Several of these goalies have reputations for playing poorly in the playoffs (especially Hall, Giacomin, Esposito, Cujo). I've looked at Joseph's results in some detail and I think that some of his underperformance is the fault of the scorers in front of him, although he has not done particularly well in some high-leverage situations such as playoff OT. Similarly, I think Glenn Hall's teammates are likely more to blame than he is. On the other hand, Giacomin and Esposito both had some weak playoff seasons on strong teams.

The Worst Clutch Goalie?Harry Lumley (-5.5)

Lumley's teams had a winning percentage that was .073 below what was expected based on their goal differential, which suggests something was going very wrong in those tight games. Whether he was a choker or merely the most unlucky one in this group I'm not sure, but given that he played for three different franchises and was mostly on good teams it's possible Lumley had something to do with it.

Tuesday, August 25, 2009

That lineup looks pretty good to me, it's nice to see they didn't pick a Turco or an Osgood just because of the "name" factor or because of team success. The only other guy who comes to mind that could have been there would be J.S. Giguere, who doesn't have much of an international track record and picked the wrong time to have a poor year. Carey Price was at one point considered the most likely candidate in the "young guy who's there for the experience" role but was leapfrogged by Mason.

I think the choices that matter (the starter and the backup) are pretty straightforward this time around: Just go with the same guys as before. Brodeur and Luongo have been the top 2 since the 2004 World Cup, and I expect both of them are already pencilled into the starting lineup. That leaves Ward and Fleury fighting it out for the chance to sit in the stands, unless I'm underestimating how much weighting will be put on the latter two guys' Cup rings and recent playoff heroics.

The debate over the third goalie on the team is one of those debates that are fun but pretty irrelevant in the big picture. Ward vs. Fleury is pretty close. I wouldn't want to have to make that decision right now, I'd prefer to wait a bit to see if the evidence suggests that Ward's 2008-09 second half was the result of improvement or merely a hot streak. If so I'd probably be leaning towards Ward, although I imagine Fleury's Cup ring likely gives him a bit of an edge with the selection committee at the moment. Again, that decision isn't likely to have any impact at all on the ice in Vancouver.

Mason had some good timing with his rookie year and he'll have the chance to make an impact on the Team Canada brass, although he's a longshot to get named. Mason's Calder Trophy has given him the coveted "young Canadian goalie of the future" mantle, although I wouldn't be at all surprised to see that flip around again this coming season if Mason hits a sophomore slump and Carey Price reverts to his 2007-08 form. It probably won't matter either way, as I don't think either have any chance at all to be wearing the Maple Leaf in Vancouver.

Wednesday, August 19, 2009

My recent look at Brodeur vs. Luongo in the 2000s was linked on a couple of message boards, and many of the resulting comments were some variant of "Brodeur is a winner, he has the Cup rings, he is obviously the better goalie." Brodeur's team did win 2 Cups, so I guess by most standard definitions it is fair to call him a "winner". It is the rest of the suggestion, the part where that alone makes him the better goalie, that I have a problem with.

Brodeur's team successes in the 2000s are concentrated in the first half of the decade. More recently he has not had much playoff team success. In the last 5 playoff seasons the Devils have combined for a 15-22 record. The only goalie who has played at least 1,000 minutes in the playoffs over the last 5 seasons and has a worse record than Brodeur is Jose Theodore.

This raises the question: What happened to Brodeur's ability to just win hockey games? Can we now use the same logic as before to call him a "loser"?

Brodeur's teams are mediocre now, you say? Hang on a second, you can't have it both ways. He can't be a winner for winning on strong teams and then get absolved from blame when playing on weaker teams. Either the team matters or it doesn't.

Let's compare Brodeur's playoff record in the last 5 seasons against his record in the previous 5:

What we see is that Brodeur's save percentage is pretty much identical relative to playoff average during both periods. In the more recent sample, Brodeur's GAA increased by 36%, and about 27% after adjusting for league scoring levels. New Jersey's shots against increased by 33%, and about 25% after adjusting for league scoring levels. It looks like the difference in New Jersey goal prevention did not have much to do with Martin Brodeur and a whole lot to do with the team in front of him.

The earlier Devils scored about 14% higher than the average playoff team, while the later Devils scored just 3% above average. Put that together with the goal prevention difference, and that explains why the Devils were so excellent earlier in the decade. Brodeur's shutouts are closely correlated with the number of shots against. His shutout rate was twice as high from '99-'03 as later on, again despite putting up an equivalent save percentage, which shows how much easier it is to record shutouts on a low shot team.

The popular perception of those early '00s New Jersey teams is that they were a defensive team that won a lot of grinding, close defensive battles because of great goaltending. Was that actually the case? Let's look at their record in playoff games that were decided by different goal margins:

Games Decided by One Goal:1999-2003: 19-182004-2009: 6-13

Games Decided by Two Goals:1999-2003: 11-102004-2009: 3-4

Games Decided by Three or More Goals:1999-2003: 22-52004-2009: 5-5

Turns out that wasn't true at all. The '99-03 Devils were a dominant two-way team that won more blowouts than one-goal squeakers. The more recent version does not have that same capacity to blow teams out. They have also either performed poorly or not had any luck in close games. That's probably a bit of a small sample size fluke given that the regular season Devils were usually very good at winning close games.

I think Brodeur deserves just as much blame for his team's performance from 2004-2009 as he deserves credit for his team's success from 1999-2003. After all, his play was probably quite similar over both periods. If you think he was a crucial element of the earlier teams, then applying the same logic means you have to figure that he was a major reason for the more recent flops. I don't give Brodeur much extra credit for the earlier team success (solid goaltending is the same solid goaltending whether the team wins or loses), and as a result I also don't assign him much blame at all for the last few years.

I'm sure more than a few readers are thinking about the trapezoid rule right now. That is an unknown factor, and it likely did have some small effect, but I don't think there is much evidence to suggest that the trapezoid rule had a huge effect on New Jersey's shot or goal prevention. From '98-99 to '02-03 Brodeur's backups faced 23.9 shots against per 60 minutes during the regular season, and from '03-04 to '08-09 their average jumped to 28.3. The team was very obviously not as strong defensively in the latter period.

I haven't thrown Luongo's playoff stats up yet, so let's just do that for completeness. I've included the league averages from the same period as Brodeur (2004-2009) for easy comparison:

We can't conclude too much from this small sample size, but the numbers show a team with below average offence and below average shot prevention that still managed to win as often as it lost.

Players don't win Cups, teams do. Martin Brodeur is one of many examples of a player who performed at a roughly consistent level, yet happened to have spectacular playoff success when his team was really good and very little playoff success at all when it wasn't. If Brodeur truly is a "winner", then it must have been pure coincidence that all his team success came when the team in front of him was strong.

In short, I don't see the logic of claiming that Brodeur's team successes alone justify his position ahead of Luongo in the "goalie of the decade" debate. Team success is by far mostly determined by circumstance. Evaluating individual players by team results makes little sense at all.

Sunday, August 16, 2009

Bruce over at Copper & Blue has laid out the case for Martin Brodeur as the best goalie of the 2000s. He picks Brodeur as the best combination of quantity and quality over the given time period, and describes him as a "win machine".

Brodeur's teams have won a lot of games, that is undeniable. However, the problem with using wins is that you are giving individuals credit or blame for team results. The best goalie is the one who did the most to help their team win, not the one who had their team win the most. I'm not picking Luongo as the best goalie of the 2000s because he ranks first in a few random statistics that I happen to like, I'm picking him because I think he did more than anyone else to help his teams win.

I have a method for quantifying a goalie's expected wins, based on their goal support, shots against, and team discipline. It compares the goalie's team to league average in each of those categories, assesses how many goals they scored or prevented compared to average, and then converts that goal differential into wins. Here are the averages of the key components during the period:

For example, since Brodeur's teams averaged 25 more goals per season than Luongo's teams, we would expect that to translate into more wins. The standard rule of thumb is that a 6 goal differential is equivalent to one win, which means that Brodeur's teammates' offence is worth about 4 extra wins per season.

It's not too difficult to see from those numbers that it was easier to win games in Brodeur's situation than in Luongo's. Brodeur had a significant advantage in all three categories. I calculate Brodeur's teams as having 410.1 expected wins and 465.0 actual wins, counting each tie or shootout/OT loss as half of a win. Luongo's teams had 296.1 expected wins and 353.5 actual wins. If we subtract the two numbers, we can see how each team did relative to their expected numbers. Brodeur's teams won 54.9 more games than expected, while Luongo's won 57.4 more. It is even more impressive that Luongo's teams outpaced Brodeur's teams since Luongo played 80 fewer games. That's almost an entire season's worth of weaker backup goalies in net, which would likely substantially drag down the overall results.

If we convert the expected numbers into winning percentages, we can compare directly with each goalie's actual winning percentage to remove games played by backups from the equation. Brodeur's winning percentage was .630 while his teams had an expected .556, for a difference of .074. Luongo's winning percentage was .498 compared to .401 expected (+.097).

That's not the entire story, however, as there are two major variables we have to still account for. One is shootouts, and the other is Brodeur's shot prevention effect.

From 2006-2009, Brodeur's teams have gone 33-18 in shootouts, while Luongo's teams are just 18-25. If we dig deeper, we see that Luongo and his backup goalies actually had a better combined shootout save percentage (.696) than New Jersey did (.679). The difference in win/loss record is entirely because Brodeur's teammates scored on 40% of their shootouts while Luongo's teammates scored just 28% of theirs.

To factor this in, we need to adjust the expected win totals to reflect the rest of the team's shootout performance. Brodeur's teams won 7.5 more shootouts than average, while Luongo's won 3.5 fewer. Adding that to the win totals, we get Luongo's teams at 60.9 wins above expected and Brodeur's at 49.9.

Finally we come to the tricky issue of shot prevention effects. It's much tougher to assign a specific number here, so I'm going to rely on a sensitivity analysis. The goal is to identify the shot prevention effect Brodeur would need to surpass Luongo, and assess whether it is reasonable that the actual difference was that large.

First, let's look at how New Jersey's overall team wins above expected would vary based on different estimates of shots against prevented by their goalie:

Since Luongo's teams were at +60.9, the goaltender shot prevention difference between the two teams has to be 1.2 or more for Brodeur's teams to end up ahead. I'd say it is reasonable that Brodeur's effect relative to Luongo's is that high or higher. Keep in mind though that we are comparing Brodeur's teams to Luongo's teams, and Brodeur played in 80 more games. With these numbers it is certainly possible to make a case that Brodeur provided more total value than Luongo over the decade, but let's look at the individual numbers for each goalie:

Here the break-even point is 2.5 shots per game. I'm willing to concede a shot prevention difference between them of 1.0 - 1.5 shots per game, and maybe even as high as 2 shots per game, but 2.5 seems a bit high to me. As a result, my interpretation of these numbers is that on a per-game basis Luongo was slightly more likely to help his team win than Brodeur. To be sure, it is very close. When the numbers are close, we can also go to qualitative factors to break the tie. I generally put a high weighting on peak play in my rankings, and in my opinion in the 2000s Luongo's peak was higher than Brodeur's.

Another consideration is that each marginal goal becomes slightly less valuable the more a team's expected winning percentage deviates from .500, . A goalie on a very poor or very strong team would not see each additional goal saved have the same impact on winning, because it is more likely to come in a blowout game. On the other hand, a goalie on an average team will spend a lot of the time in tied or close games, which means that their effect is magnified. After factoring in shot prevention, Brodeur's expected winning percentage is close to .500, which means that his marginal goals saved likely had a strong impact in terms of winning games. On the other hand, Luongo's expected winning percentage was just .396. I don't think this has a large effect on the numbers, but it is something that would be a slight advantage to Brodeur.

Depending on your assessment of the method and goalie shot prevention effects, you may agree or disagree with the evidence that Roberto Luongo helped his team win more games than Martin Brodeur. I'm still working with the method so some of the assumptions may need to be revised slightly to get a better fit. Regardless, I think this exercise should be strong evidence against the view that Brodeur far surpassed all his goaltending peers between 1999-00 and 2008-09. If Brodeur can be described as a win machine, then the same thing should be said about Roberto Luongo.

Saturday, August 8, 2009

"72% of all regulation play during the 2008-09 season was spent tied or at a one-goal differential."

This is why I think people are simply talking nonsense nearly all of the time when they start going on about goalies who make the big saves or who make saves at the key times. Most of the game is "key time", because there just aren't that many goals in NHL games. "Key" implies an important or pivotal time in the game, and it seems to me a bit improper to use that word to refer to a situation that includes nearly three-quarters of the saves a goalie might face.

Let's say we had an extreme case of a goalie who puts up a .920 save percentage in one-goal or tie games, but then completely tunes out and stops caring when the margin gets to 2 or higher (.890). If the shots against are distributed with the same 72/28 split, his overall save percentage is going to be about .912. Given what we know about how teams play to the score (shot rates tend to drop in blowouts and tend to rise in close games), that assumption is probably false, which means that there will be even less of a save percentage difference. In addition, that goalie might save a few more wins for his team in close games, but he'll also make it much harder for his team to come back occasionally from a 2 goal deficit, and he would also frequently put his team at risk of giving up 2 goal leads. As a result, I'm not sure there would be much of an increase in the team's win totals.

This shows that even an impossibly extreme split leads to an overall save percentage difference of just .008. Even if there are actually some goalies who consistently either raise their play in tight situations or lose focus a bit in blowouts, that is likely to have a very small effect over the course of an entire season. Let's say in the above example the goalie had a .910 save percentage in the blowout scenario, for a drop of .010 (which is about the difference between an average goalie and Roberto Luongo). That would leave the goalie's overall save percentage at .917, very close to the original .920.

The converse is also true, that a goalie who plays poorly during the 72% of the game that is "key time" is likely to have poor overall stats as well. It is not possible that a goalie can be so good in non-pressure situations that he makes up for having awful stats in close games. Let's change the example to a goalie who can't handle the pressure and can only stop 89% of the shots against when the score is tied or his team is within one goal. What save percentage would he need to have in all other situations to end up at the same .912 as in the first example? The answer is .970, and it obviously goes without saying that nobody is going to consistently stop pucks at that rate in any game situation.

It is not plausible to me that a goalie can make up a large statistical gap through timing his saves. Hockey is a fairly low scoring game, and as a result most of the time the score is close. Goalies who make a lot of saves also make a lot of key saves, and goalies who allegedly bear down when the game is close will spend about 3 minutes playing with extra focus and competitive fire for every 1 minute they are supposedly goofing off in a blowout. If one goalie has clearly better numbers than another goalie but a lower win total, then that is almost certainly not because of "making the key saves", but rather because of differences in the number of goals the two teams scored and the number of shots against each team allowed.

Note: Even though I've never seen much evidence to support it, I'm not ruling out differences in clutch play between goalies. I just don't think the difference would be very large and therefore we would need a huge sample size to be able to identify it. Even if we find some differences between goalies over a multi-season sample, we might still not be sure whether we are observing a difference of luck or a difference of skill. This uncertainty makes it something that is probably not really worth worrying about for the most part.

There certainly seems to be more randomness and/or potential team effects in this sample than in the even-strength one, as we see goalies like Roman Turek, Marc Denis and Chris Osgood popping up. The best guys (Hasek, Roy, Luongo) are still near the top, however. I think it is likely that playing at even-strength and playing on the penalty kill require different skill sets, at least to some degree, which would explain some of the discrepancy.

It does seem that it would be a mistake to entirely dismiss penalty kill performance in favour of even strength play and thereby fail to credit some of these guys for a significant contribution while their teams are down a man. I'm still not really sure how much of an effect the penalty killers have in front of the goalie, if special teams are more team dependent then that should be factored in. In any event, a goalie's contribution at even strength is much more significant than on special teams.

Tomas Vokoun drops pretty far down the list compared to his outstanding even strength results. Other notables who apparently don't do as well on special teams are J.S. Giguere, who is right behind Vokoun in 21st spot with an .870, and blog favourite Kari Lehtonen (.861).

About Me

I'm Philip Myrland, goalie stats blogger since 2007 and former writer for Hockey Prospectus. I call myself a contrarian because I rely mainly on stats-based analysis, think large sample sizes are more important than "big saves" and prefer to rank goalies based on save percentage rather than team success. In other words, pretty much the mainstream views of the hockey analytics community, but I've grown attached to the handle and plan to keep it as long as broadcasters everywhere keep bringing up GAA and wins. If you have any questions or comments contact me by email (contrariangoaltender@gmail.com). You can also follow me on Twitter (@tcghockey).