Comparing Quarterback Rating Metrics

There is no topic amongst NFL fans that sparks more debate than quarterback rankings. As a result, there have been countless QB rating systems devised by people from every corner of the internet. Today I want to look at four of the most popular and divergent QB ratings metrics, and compare the way they rank QB’s through the first quarter of the 2015 season.

In the table below, you will see the ranking for each of the 34 qualified quarterbacks in NFL Passer Rating (PR), Adjusted Net Yards Per Attempt (ANY/A), Football Outsiders’ Defense Adjusted Value Over Average (DVOA), and ESPN’s Total QBR (QBR). Also shown are each QB’s average ranking among the four metrics, and the standard deviation of their rankings (Dev).

There appears to be a consensus on the top four QB’s so far this season, with Dalton, Roethlisberger, Brady, and Rodgers ranking in the top four in every metric. Beyond those four, the metrics begin to disagree more sharply. Philip Rivers, Marcus Mariota, Alex Smith, and Cam Newton fare significantly better in the box score metrics (PR and ANY/A) than they do in the advanced systems (DVOA and QBR). Why?

Rivers and Smith have padded their traditional stats with short completions and a ton of YAC from their receivers, artificially inflating their completion % and Y/A. Mariota has fumbled liberally, taken a lot of sacks, and padded his numbers with empty yardage on failed third down attempts. Newton is a bit of a mystery as none of his box score stats stand out, but apparently play-by-play and game charting reveal flaws that traditional stats can’t see.

There are a few other interesting patterns I’d like to highlight. For example, Teddy Bridgewater is a fascinating case, as he fares markedly better in QBR (11th) than DVOA (29th).

The three major differences between QBR and DVOA are the treatment of defensive pressure, splitting air yards and YAC, and game context. QBR credits quarterbacks who makes plays facing pressure, whereas DVOA does not. QBR gives quarterbacks more credit for air yards than YAC, while DVOA counts all yards the same. And QBR significantly lowers the weighting of plays occurring in garbage time, while DVOA has a lower success baseline in garbage time, which actually makes it easier to gain value in garbage time than in tight games.

Bridgewater has faced a gauntlet of pressure in every game, so QBR realizes that most of the sacks he’s taken were not his fault, and gives him extra credit for completing passes under such difficult circumstances.

Peyton Manning has been raked through the coals for his play in 2015, but QBR likes him much better than the other three metrics. Similar to Bridgewater, Peyton has dropped back behind a very porous offensive line, and he’s benefitted from less YAC than just about any QB in the league.

On the flip side, Brian Hoyer looks pretty average in every metric except QBR, which ranks him dead last. He threw 30 passes in Atlanta after his team fell behind 42-0, which is the very definition of garbage time. Despite performing well in that situation, QBR gives him almost no credit. In his only start in week 1, Hoyer missed a bunch of open receivers, took four sacks, and threw a horrendous interception in the shadow of his own goal line. QBR sees this as his only meaningful action in 2015, and drags down his rating accordingly.

I’ll close with a table displaying the correlations for each pair of metrics:

Metric 1

Metric 2

Correlation

PR

ANY/A

0.94

ANY/A

DVOA

0.92

PR

DVOA

0.86

DVOA

QBR

0.75

PR

QBR

0.69

ANY/A

QBR

0.68

As you can see, Passer Rating and ANY/A spit out very similar rankings, which is not surprising consider they’re both conceived from basic box score statistics. The main difference between them is that ANY/A counts sacks while Passer Rating does not. Surprisingly, ANY/A and DVOA also correlate very strongly.

Despite DVOA being proprietary and coming from play-by-play data adjusted for a multitude of situational factors, it doesn’t add much to the simple formula of ANY/A. Really, the main advantages of DVOA are crediting third down conversions and adjusting for opponent strength, but those things tend to even out for most QB’s over a full season. The outlier in this study is ESPN’s shiny new toy, as QBR diverges noticeably from the other three systems. I see this as a very positive feature of QBR – it tells us things the other stats can’t.

Personally, QBR jives with my intuition and the eye test far better than the others. It quantifies what I’m seeing very accurately. Despite the hand wringing about its complexity and being hidden within ESPN’s secret black box, QBR does its job better than anything else out there.