You’re probably hearing a lot right now about how Matt Schaub is not a primetime player — literally. Schaub and the Texans struggled in embarassing losses on Sunday Night Football to the Packers earlier in the season and on Monday Night Football two days ago to the Patriots. Schaub posted terrible numbers in a defensively-driven 13-6 win over the Jay Cutler/Jason Campbell Bears.

For his career, Schaub is 2-5 (0.286 winning percentage) in night games and 41-31 (0.569) in day games. Among the 24 quarterbacks studied (more on that below), that drop in winning percentage is the largest such decline. You might think this is due to facing better defenses in night games, but that’s not really the case.

Schaub has averaged 7.8 Adjusted Yards per Attempt during day games and 6.0 AY/A during night games; that difference of 1.8 AY/A is the second largest among the twenty-four quarterbacks.

So yes, there is no debate: Schaub has been noticeably worse during night games in his career.

The table below shows all quarterbacks who have started a game this season and that have started at least five night games in their career. The data consist of all games throughout their career in which they were the starter. To make it a little easier to read, I’ve shaded the day and night categories differently:

There is a simple explanation of why Schaub has played poorly at night. It’s because his real name is Choker McChoker. At least, that’s one way of looking at the data. Here’s another. Schaub has started 79 games in his career. Over those 79 starts, he has averaged an AY/A of 7.6 and has a standard deviation of 3.0. Therefore, there is a 30% chance that, purely by chance, over a random 7-start sample his AY/A would be 6.0 or worse.

Hey Gary, ain't that the dang sun setting?

But let’s say you hate math. Consider: there are 24 quarterbacks on the above list. Considering the large number of quarterbacks and the small number of night games, there was an excellent change that somebody was going to look like the captain of Choketown, USA.

Longtime readers will make the connection between this post and two of my favorites posts of all-time. In Splits Happen, I explained how if you look hard enough, it’s very easy to find crazy splits. For example, in 2011, Reggie Wayne had nearly twice as many fantasy points per game against teams that had blue in their uniforms. There are an endless number of silly splits you could find. I concluded with the line:

Despite what our brain tells us, there need not always be an explanation for a crazy split. Sometimes, splits happen.

We see Schaub’s nighttime/daytime splits, and think there must be an explanation. But that’s not the case. Sometimes, splits happen.

In “What Are the Odds of That?” — my favorite post ever — I looked at an obscure Jacoby Jones “stat.” In 2011, Jones gained three times as many receiving yards against teams at the back end of the alphabet as he did against the teams he faced in the front of the alphabet. Then I asked, “what are the odds of that?”

This is a very good reason why it’s often inappropriate to apply standard significance tests to football statistics. Surely Jones’ splits would pass any standard significance test, signaling that his wild split was in fact “real” even though we know it wasn’t. With a large enough sample, you would expect to have false positives, which isn’t a knock on standard significant testing. If something is statistically significant at the 1% level, that doesn’t mean you shouldn’t expect to see a false positive if you have 100 different samples…
…
Some in the statistical community refer to this as the Wyatt Earp Effect. You’ve undoubtedly heard of Wyatt Earp, who is famous precisely because he survived a large number of duels. What are the odds of that? Well, it depends on your perspective. The odds that one person would survive a large number of duels? Given enough time, it becomes a statistical certainty that someone would do just that. Think back to the famous Warren Buffett debate on the efficient market hypothesis. Suppose that 225 million Americans partake in a single elimination national coin-flipping contest, with one coin flip per day. After 20 days, we would expect 215 people to successfully call their coin flips 20 times out of 20. But that doesn’t mean those 215 people are any better at calling coins than you or I am. The Wyatt Earp Effect, the National Coin Flipping Example, and my Splits Happen post all illustrate the same principle. Asking “what are the odds of that?” is often meaningless in retrospect. If you look at enough things, enough players’ splits, enough 4th quarter comeback opportunities, enough coin flips, or enough roulette wheel spins, you will see some things that seem absurdly unlikely.

And if you look hard enough, you will find a split where Matt Schaub, or any other quarterback, performs poorly. Of course, that’s just my perspective. Like all things, the conclusion rests in the mind of the reader. If you’re the type of person who was inclined to believe that Schaub was the incorporator of Chokers ‘R’ Us, Inc., well, I doubt this post is going to change your mind.

Good info regarding the chance occurrence of signficant but meaningless correlations. Any stat professor will tell you that if you look for enough associations between factors, you will find odd-ball significant correlations that are not a true relationship.

In grad school I undertook a large project involving a repeated measures design that looked at correlations among 20 different factors and all the possible combinations. Found all kinds of stuff signficant at 0.01 but a lot was junk. A big fishing expedition that my thesis committee rightly pointed out.

Chris at Smart Football has recomended two books over the years related to cognitive psy—Daniel Ariely’s on Irrationality and Kahneman’s “Think Fast, Think Slow.” A point made in both (and in many other books about human reasoning) is that we look for patterns in all types of information from our general environment. This “coherence seeking” leads us to make many false cause and effect conclusions.

In the latest, presidential race I thought it telling that people talked about the football score correlations (I believe it involved the Redksins) as “predictions” regarding the outcomes of elections. And of course this latest election bucked the trend. In the meantime all these “knowers” ignored Nate Silver and his actuarial model (in which he was right).

Thanks Chase for the post. Can’t wait to see how the intuitive thinkers weigh in with disagreement.