Monday, October 17, 2011

The Ecological Fallacy

I found a scholarly name for the tendency to rate goalies on winning teams as better than goalies on losing teams. It's the ecological inference fallacy. You can read all about it on Wikipedia, but in short it is the assumption that all members of a certain group share the same characteristics of the entire group. To quote Wikipedia:

If a particular sports team is described as performing poorly, it would be fallacious to conclude that each player on that team performs poorly. Because the performance of the team depends on each player, one excellent player and two terrible players may average out to three poor players. This does not diminish the excellence of the one player.

Nor does it boost the performance of an average player who happened to have great teammates. Avoid the ecological inference fallacy and give credit where it is due, based on an individual's contribution to the team effort regardless of the final result.

To see this type of thinking in action, go read one of the post-game report cards that fans put up after their team plays. In many cases, when a team wins there are As and Bs across the board. Yet when the team loses, everyone gets Cs and Ds.

Check out, for example, Vancouver blog Nucks Misconduct's report cards from last year's Cup Finals. In Vancouver wins, the average score for Vancouver fourth-liners was 9.1. When the Canucks lost, the average score for fourth-liners was 7.0. Can the Canucks' losses be blamed on a line that barely played and had little impact overall on the series? Of course not. It just so happens that when Roberto Luongo was making saves and the other forward lines were scoring then Victor Oreskovich, Tanner Glass, Jeff Tambellini et al looked better by association. In contrast when Luongo got shelled and the Sedins were shut down, the same guys playing their usual 6 or 7 crash-and-bang minutes without a goal for or against ended up getting hung with the same mediocre grade as the stars who were actually driving the bus. That's the ecological fallacy in action.

I'd say this logical error explains quite a few of the most common mistakes made in rating goalies. Add in the base rate fallacy that causes people to exaggerate their praise or criticism for a goalie's performance by not properly factoring in the play of a typical replacement ("Without Lundqvist, the Rangers would have lost at least 10-1!"), plus the fundamental attribution error which makes people lean towards personality-based explanations for team successes or failures ("Carey Price's teams will never win in the playoffs because he lacks mental toughness"), and finally availability bias ("I don't remember any of Mike Liut's career except for that Canada Cup Final where he let in 8 goals, but that game proves he was an awful clutch performer"), and you've probably covered 95% of the rest of them as well.

I think in general you are right and make some good points. But the 4th liners you mentioned played worse in the Canucks losses. All 3 mentioned were even players in the wins and minus players in the losses.

I'm not saying they had a meaningful impact on the game but to the average observed there did not play as well.

A lot of things could go into this which is beyond their control.

1) Match-ups: the nucks won 3 home games and lost 3 road games. The 4th liners were likely matched up against better competition in Boston which makes it more likely they will get out-played and appear worse (and showed up in the scoreboard).

2) Play time: these guys all played more in wins (6-8 minutes) than in the losses (4-6 minutes). This might be related to the point in (1).

3) Score effects: The canucks losses were blow-outs. Its tough to play hard and look good when you are down 6-1.

I disagree that they played worse in Boston's wins. You're arguing based on plus/minus, which is pretty much backing up what I wrote about how they looked good when Luongo was making the saves and bad when he wasn't. My argument is based on watching the games and on the scoring chance numbers from Copper 'n Blue, which are as follows for Malhotra, Oreskovich, Glass and Tambellini* combined in the Stanley Cup Finals at even strength:

Home: 66.4 minutes, 9 SCF, 17 SCARoad: 76.4 minutes, 15 SCF, 21 SCA

*-Does not include game 7 numbers for Tambellini, as he wasn't on the Canucks' fourth line in that game.

Your ice time stats aren't right either. All four of those guys averaged more ice time on the road than at home. Which makes sense, since if the game's already lost there is no reason not to spread out the ice time more than if you are trying to protect a slim lead.

Malhotra was also +1 in wins -2 in losses. But wasn't included int he original post so I wasn't looking at him.

That said espn.com doesn't list minutes for Glass in games 1-2 and Tambellini for games 5-7. I was guessing that meant they didn't play those games but maybe the data was missing and I can't remember from watching.

"All four of those guys averaged more ice time on the road than at home."

That doesn't seem to be the case above.

"Which makes sense, since if the game's already lost there is no reason not to spread out the ice time more than if you are trying to protect a slim lead."

I think we might be saying the same thing with this part. What I mean is:

If the game is close (like the Vancouver wins) the 4th liners sit while the stars go out and try to keep/extend the lead.

If the game isn't close (Boston's wins) the stars sit and there is more ice time for everyone else.

My argument is based on your comment of:

"the same guys playing their usual 6 or 7 crash-and-bang minutes without a goal for or against ended up getting hung with the same mediocre grade as the stars who were actually driving the bus"

The 4th liners are supposed to do exactly what you are saying there. There are supposed to go out and give you a shift or 2 to rest the other lines. You would rather them play 0-0 than 1-1. Goals against by the 4th line are a huge no-no and will earn them a lower evaluation.

Maybe saying they played worse is a poor choice of words. They didn't do their job as 4th liners in the losses as well as they did in the wins is the point I am trying to get across.

About Me

I'm Philip Myrland, goalie stats blogger since 2007 and former writer for Hockey Prospectus. I call myself a contrarian because I rely mainly on stats-based analysis, think large sample sizes are more important than "big saves" and prefer to rank goalies based on save percentage rather than team success. In other words, pretty much the mainstream views of the hockey analytics community, but I've grown attached to the handle and plan to keep it as long as broadcasters everywhere keep bringing up GAA and wins. If you have any questions or comments contact me by email (contrariangoaltender@gmail.com). You can also follow me on Twitter (@tcghockey).