Yesterday I discussed some of the entrenched misperceptions about statistical analysis in hockey and why they are misguided. Today, in a post called "Hockey's Counting Problem", Cam Charron looked at some of the significant obstacles standing in the way of effective, evidence-based analysis truly getting a foothold in the NHL's upper offices.

Two challenges he touches on that I want to discuss in greater depth are top-down vs bottom-up processes and the clash between long-term decision making and short-term incentives.

Information Processing and Flow Problem

Part of the advantage with having peer-reviewed research in the public sphere is that other people can check your work for inaccuracies.

Cam touches onwhy "advanced stats" have proliferated in blogs, comment sections and messagboards over the last decade or so: the benefits of an open, non-centralized structure are iterative, self-correcting processes. Meaning, the spontaneous order of a bunch of independent minds conducting experiments and making arguments results in a sort of "wisdom of crowds" effect, where evidence and truth eventually win out. This bottom-up process means things that can skew decision making, like authority, hierarchy and adherence to convention are lessened*.

Hockey teams, in contrast, tend to be rigidly top-down, hierarchical organizations. In addition, the league is very much an "old boys club" consisting largely of same-names and cultural warriors who are committed to defending the status quo. As result, there are entrenched, institutional barriers to innovation, both within teams and throughout the entire league.

Of course, there are obvious reasons teams no doubt strive to improve their player evaluation and scouting - everyone is trying to get the upper hand, after all. The problem is the assumptions of authority figures aren't as easily challenged in this sort of setting: like water, information doesn't readily flow uphill, especially if it runs counter to the authority of the guy in charge.

Teams are "bunkered" as well, meaning they don't tend to share data or breakthroughs with each other. Think of it like a collection of prisoners separated by foot-thick concrete walls with no windows. Each guy is trying to find a way to break out of jail and each guy has certain, unique bits of information. If they could share their data and collectively generate an escape plan, their chances of breaking out drastically increase. Instead, each prisoner wiles away his hours gazing at the walls, foiled by the unknown unknowns.

As a result, there's limited data sharing and feedback on top of a greater risk of convention, authority and groupthink obscuring efforts to advance knowledge.

The Game Value Problem

Basically what you have is teams struggling with concepts they can’t prove, all while working within a very narrow window to succeed. I can imagine if I were a team owner and for two years I was paying money for an analytics department to wrestle with the concept of zone entries while the team lost games on the ice and nobody was paying for tickets, I wouldn’t have too much patience with the direction of the club.

As Cam notes above, the other reason hockey analysis has evolved much more rapidly amongst "outsiders" like bloggers is because it's easier (that is, less costly) to be publicly wrong versus your average coach or GM. In hockey, the vast majority of "new" analysis is long-term, broadbrush stuff - looking to tease apart the skill signal from the vast, competing noise can take a lot of time/shots/games etc. Amateurs and fans engaging in this analysis outside of the incentives and pressures of actually running a team have the benefit of time and endless iterations to eventually get things right. Guys in positions of power in the NHL? Not so much.

Unfortunately, the informational and predictive value of a single hockey game is incredibly small relative to the value of a game in the eyes of fans/players and decision makers. For example, it can take between 3000-4000 shots to get a feel for a goalies true talent level, meaning an average 30-shot game is about 0.01 (1%) of the info you need to make a truly informed decision about a puck stopper. In contrast, NHL executives and coaches are judged on individual wins and losses as well as clusters of success or failure that can have very little to do with the actual quality of their roster or their decisions. I've argued previously that coaches are often sacrificed before the percentages; that is, fired because of natural variance.

NHL decision makers are almost always under pressure to win now - and if they aren't winning now, to figure out why they're losing as quickly as possible. They are frequently dealing with only small snippets of useful information, often awash in a sea of competing signals and unpredictable or uncontrollable factors like schedule strength, officiating and injuries, to say nothing of the natural ebb and flow of human performance. The catch-22, then, is that the concern for the short-term can override the need to be patient and sort out true talent signals from the dead-ends and blind-alleys. And sometimes even patience isn't enough - managers and bench bosses will occasionally have to make decisions about players based on less than a full season of data.

Conclusion

There aren't any obvious or easy solutions to the issues posited above. Different clubs will have different corporate cultures, so openness to new ideas will vary to some degree depending on the people and management in place (although everyone will continue to suffer from the "bunkering" effect, or the inability to share, check on and improve findings). Encouraging experimentation and the challenging of even long-standing assumptions inside an organization would probably help.

In addition, GM's will always have to battle the urge to overly weight and react to short-term results given how much time/data it takes to truly separate the wheat from the chaff in the NHL. It would help to have stable, patient ownership as well.

I find there is a general problem with these stats - be it +- or corsi or GVT or what have you. People have expectations - for example Phaneuf is a good dman. Then when the data doesn't match their expectations they blame it on their team mates or the goalie or the fact he takes so many shifts. Now Phaneuf is a fine Dman but lots not kid ourselves his fancy stats show that he is pretty much average first pairing defender with serious defensive problems but nothing special.

The way they abuse advanced stats to support their position is as bad as those that ignore stats because they demonstrate that a knucklehead like Orr is bad player. Point being - people have already formed an opinion and look for stats to support their position or when stats don't look for excuses why the stats are incorrect. There is little credibility in the stats arena - just more formalized and analytical opinions supported by data. They are not fooling anyone just because they can add, divide and run a regression.

I have some appreciation for the position that coaches are put in, with respect to the pressure to win now. Coaches are also saddled the players given to them with little input on the players they have to choose from. While advanced stats are useful tools that should be used by coaching staffs, I can see why they might fear a different way of looking the game. The problem with being a trailblazing pioneer, is that those are the very people who get shot first.

GM's on the otherhand have much less to fear. Frankly, the secretive nature of the business would guarantee that a GM who embraced advanced stats would look no different than anyone else. It's also a GM that chooses the coach, who's style of play reflects his own vision of how the game should be played.