You give stats a bad name - how I’ve failed to become a statistical Bon Jovi

TJ JohnsonJun 18, 2011 1:00 PM

I long ago lost my faith in the quality of Bon Jovi's work.

Yet I can't recall the exact game I stopped believing in NFL statistics. Perhaps it's best if I tell you when I began to lose my faith.

It was last year's game when the Broncos were pulverized (yet again) by the Baltimore Ravens. You probably remember the statistical headlines from the game. Kyle Orton threw for over 300 yards and Brandon Lloyd went for more than 100 yards receiving. I had already posted my Gut Reaction to the game, which like all Gut Reactions, are usually posted within fifteen minutes of a game's finish. It was well-written, yet there was something that still haunted me.

So I hit the game tape over and over again. It didn't take more than a few series to see the horror that was Haloti Ngata.

Ngata was dominant that day, even though he finished with only three tackles. Play after play he dominated the Broncos' offensive line. Ngata got gap penetration; he used the same swim move what seemed like half a dozen times (which I promptly taught to a pee-wee football team the following week); he pushed the line back into the backfield two or three yards; he was a nightmare that, to this day, probably causes Ryan Harris to wake at night in cold sweats.

I had already known that stats often hid the truth in a game requiring eleven giant men to synchronize their movements against another eleven. Still, it was this game that was the Marcus Nash/Jarvis Moss moment for me. Ray Lewis, Joe Flacco, and Ray Rice were getting the stats.

But Haloti Ngata was the guy beating the living hell out of everyone.

At the same time I reread the Nassim Nicholas Taleb classic Fooled By Randomness, except this time I actually read it. The primary message of the book was quite profound: the more complex a system, the less likely we can describe it with models (i.e., statistics). While reading the book made me realize I might have wasted money getting an MBA in Finance (I had always been slightly annoyed that my financial models failed to predict the movement of securities), I was more than happy to apply Taleb's thoughts to the NFL.

Stats couldn't really account for Haloti Ngata. He was disrupting plays, yet he wasn't credited with a tackle. He was forcing double teams so that others could get a sack. He was eating up space so that Ray Lewis could sell body wash.

I began focusing on tape, which was more interesting anyway, which led to the the popular Playbook Abides series. It opened my eyes and took me back to my playing days. Rather than focus on the numbers, I began to see what the players were reading and what they were doing. If Ryan Clady got beat on a spin move and forced Kyle Orton to throw the ball away, I knew there was an explanation for that incomplete pass. If Orton correctly identified the Mike linebacker on a pre-snap blitz read and Knowshon Moreno still missed his blocking assignment, I had a reason for the sack. In short, I began to at least attempt to put the numbers into context.

Further, I realized all of the biases I had been bringing to my statistical analysis:

Confirmation Bias - I was using stats to confirm my own beliefs about the Broncos. For example, I could easily quote Kyle Orton's stats on 3rd down rather than watch the tape if I wanted to push my personal view that Orton was a bad option for the Broncos (as a side note, the tape supports the numbers).

Outcome/Hindsight Bias - I was judging decisions the Broncos made by their outcome rather than their context at the time. This lead to an eventual weekly piece called Huge Decision, in which I tried to evaluate weekly coaching decisions in context rather than their eventual result. So that 4th and 2 that the Broncos failed on? Rather than assume it was a mistake, I attempted to see if the numbers supported the decision at the time it was made.

Selection Bias - We've all made this mistake. Pick a set of data that supports your view and ignore the rest.

Illusory Correlation - Here, I'd try to find a correlation coefficient between two events and hint at a strong relationship. I finally stopped doing this when I realized that there was absolutely no correlation between teams coming off a short week and their records the following week. But I so badly wanted to explain what I had mistakenly perceived as a pattern (as a side note, it doesn't take stats to find a correlation between Raiders fans and dropout rates).

The list could go on and on (counting stats bias and sample size bias are two more), but there's simply no reason to point out all of my flaws. I plan to use statistics in the future, after all. Let's hope I use them in a better way.

One of the reasons that I've become a convert of Brian Xanders recently is that he at least has a feel for how to use numbers. From his interviews, he spends a lot of time with tape (which I've made fun of from time to time), but when it came time for the draft, he wisely stockpiled quality picks, which improved the Broncos' chances of landing quality starters. Although this might seem obvious, many teams simply junk their drafts trying to move up to get players. Those teams, as Taleb might say, will benefit from more randomness.

The same sort that had the Broncos move up in the 2006 Draft to get Jay Cutler. One pick later, the Ravens took Haloti Ngata.

Jay Cutler has a higher quarterback rating; Haloti Ngata has more wins.

You know, I can&#8217t really refute JoePlummer&#8217s point on fantasy football (this as an admitted fantasy player). That said, I think there&#8217s another issue here: the game of NFL Football has changed.

Now, mind you, I&#8217m fairly young. But it feels like the older version of the NFL was a bit more straight-forward: beat your man. This isn&#8217t to take away from the skill involved with that game: they would stand up just fine today. I&#8217m thinking more about the evolution of the game and scheme complexity. It just feels like the number of reads, the size of the playbooks, the constant change of personnel groupings&#8230it makes it harder to know what a player is really supposed to be doing, and whether his actions were &#8220successful.&#8221

I keep thinking to Robert Ayers here. While most outlets were beginning to use the &#8220bust&#8221 label his rookie year, Denver coaches were impressed. Sure, he isn&#8217t exactly racking up the sacks, but he IS a force on defense, creating exactly the kind of headaches he was originally drafted for.

Posted by DiscoStu on 2011-06-21 02:43:40

TJ, many excellent points. I really enjoyed this article. I think fantasy football has unintentionally led to dumbed down NFL analysis - too much focus on good stats and not enough on good plays.

Posted by JoePlummer on 2011-06-20 04:03:59

Shanny believing he could mold the next Elway (Cutler) lent a blind eye to the needs of the defense (Ngata) and opted for a bust in Dwayne Robertson who was a former #4 pick in the first round&#8230Forgeting or putting on a shelf an integral part of team chemistry (defense) Shanny knows how him do&#8230

Posted by bfree2bronc on 2011-06-19 21:34:46

Excellent piece TJ. The best use of stats is in combination with observation. One of the most successful NBA bettor uses a quantitative approach but watches every single NBA game to verify that what his stats are telling him is accurate. He also observes what coaches do when way ahead and way behind in terms of changing the pace of the game and uses that in conjunction with his statistics.

Also, a lot of what you described is a flaw in the fact that the statistics being used are box score statistics. When I see someone like Ngata dominate a game and don&#8217t see it reflected in any of the PBP or box score, I generally start thinking that we need a new stat for what he brings to the table - QB hurries might be one or zone stats similar to baseball that measure how much distance a guy can cover.

Posted by chantech on 2011-06-19 18:07:41

Warmick,

I think you can with say the Falcons this year and their trade up. But I think you&#8217re right that it&#8217s not junking a draft if you&#8217re moving a spot or so to jump ahead of a team.

I have to say that I base a lot of my confidence when it comes to the draft by how much I wanted Ngata and not Cutler. I was so pissed that day when I heard we moved up and then shanny picked Cutler.

Posted by Fan in Exile on 2011-06-19 02:17:41

Thanks DiscoStu

After submitting my post I got to thinking that was probably what it meant. I&#8217m not sure you can call the strategy &#8220junking the draft&#8221 though. That may happen either way.

Posted by warmick on 2011-06-19 01:48:01

@warwick &#8220Whats that mean?&#8221

I could be wrong, but I think TJ means: teams will use a bunch of picks to move up and grab a specific player. That player may be good&#8230but maybe not. You may end up with Jay Cutler, or Jarvis Moss. Maybe you take Jay Cutler, but Haloti Ngata was available and would have been a MUCH better pick (and that&#8217s not even a knock on Jay&#8230Ngata is a BEAST). With the uncertainty in the draft (it&#8217s not a complete crap-shoot, but there is a good bit of randomness involved) it&#8217s better to use good scouting coupled with a greater number of picks, than target THE ONE.

That was one thing I admired about the McDaniels drafting strategy: when the Broncos did move up to grab a guy, the move was either A) trade one pick from next year for one now or B) move a bunch, so your &#8220draft points&#8221 go down, but your number of picks stay the same(ish). It&#8217s not quite New England (they have 30 picks this year, right?), but it gives you players you want AND a good quantity of picks. Now, we can argue about results (but let&#8217s not right now), but the statistical premise was solid.