QBR: ESPN's Deeply Flawed Made-For-TV Stat

Imagine you’re a life-long Steelers fan sitting nervously on the edge of your seat during Super Bowl XLIII. Ben Roethlisberger takes the field at his own 22 yard line with just 2:30 left in a desperate effort to erase a 3-point Arizona lead.

Now imagine your buddy sitting next to you turns to say, “You know, I’d feel a lot better about Pittsburgh’s chances if Dan Orlovsky or Tyler Thigpen was calling signals because both have a better QBR than Big Ben.”
You would have looked at him and said, “You have picked a spectacularly bad time to go completely *@%!! nuts! … And what in the wide, wide world of sports is QBR?!”

QBR is an acronym for “Total Quarterback Rating,” the very first made-for-TV sports “statistic” brought to you by [Insert Sponsor’s Name Here] and to be televised and discussed wall-to-wall on ESPN.

Well, in 2008 Orlovsky QBRed the Lions to 10 losses in 10 games as Detroit became the first team in history to finish 0-16. Thigpen was only a little better, QBRing the Chiefs to only one win in the 14 games in which he participated.

Roethlisberger QBRed Pittsburgh to 12 regular-season wins, two more wins over San Diego and Baltimore in the AFC playoffs, and concluded the season with a majestic two-minute, 78-yard, fourth-quarter scoring drive capped by a perfect TD toss to secure a Super Bowl win over the Cardinals. Cold, Hard Football Facts named that effort the third-best drive in Super Bowl history.

But according to ESPN’s QBR, in 2008, both Orlovsky (51.1 QBR) and Thigpen (50.8 QBR) contributed more to winning NFL games than Super Bowl hero Roethlisberger (46.4 QBR).

Does anyone outside the film screening rooms of ESPN really believe Pittsburgh would have won that Super Bowl with Orlovsky or Thigpen under center? Made the playoffs? Finished .500?

Conversely, if Roethlisberger had played in Detroit, does anyone believe the Lions would have lost all their games? If Big Ben had been QBRering in Kansas City, would the Chiefs have doubled their total to four wins?

Oliver and the rest of the folks at ESPN clearly have issues with the NFL’s current passer rating model that was devised primarily by Don Smith back in the 1970s and, in their zeal to market QBR, have at least insulted and possibly defamed Mr. Smith’s incredibly effective passer rating model.

ESPN’s Chris Keating wrote that the NFL’s “passer rating doesn’t attempt to weight its categories by their importance to winning football games” and that the NFL’s passer rating “yields deeply flawed results. It’s always been a jury-rigged stat just waiting for the sabremetric revolution to kick out its struts and build something better in its place.”

Fear not ESPN, you are not alone in your negativity toward passer rating and the statute of limitations long ago ran out on criticism—misplaced or otherwise—of Mr. Smith’s model. Former Sports Illustrated columnist Paul “Dr. Z” Zimmerman, in particular, argued against Mr. Smith’s model, albeit with better reasoning underlying his arguments than those offered by ESPN.

With all due respect to Dr. Z and ESPN, passer rating may not be not flawless, but clearly it is not “deeply flawed.” In fact, if anything, passer rating is underappreciated and underutilized, as Cold, Hard Football Facts Potentate of Pigskin Kerry J. Byrne pointed out last week on Sports Illustrated (and many times before that).

To the best of QuantCoach’s knowledge, passer rating does not put a winless quarterback like Orlovsky ahead of a proven winner like Roethlisberger – one of the most effective and prolific passers in history, whether measured by yards per attempt (No. 5 all time), passer rating (No. 8 all time) or onfield success (three conference titles in seven NFL seasons).

It’s no coincidence that Pittsburgh’s return to Super Bowl form after 25 years wandering the post-Steel Curtain desert came with the ascension of highly rated, highly effective Roethlisberger as its starting quarterback.

Passer rating once again proved its merit in 2010: the NFL team that posted a higher passer rating using the current NFL passer rating model won 79.3 percent of the time (203-53, according to our new “Correlation to Victory” numbers at CHFF Insider).

Further solidifying the importance of passer rating in 2010, the Super Bowl champion Packers ranked No. 1 in both Defensive Passer Rating and Passer Rating Differential, while Green Bay quarterback and Super Bowl MVP Aaron Rodgers last year became the No. 1 quarterback in NFL history in both regular-season and postseason passer rating.

Passer Rating’s Imperfections
With all that said, we agree with critics that the current passer rating model is not perfect. Specifically, it has two characteristics with which one could take issue. But neither problem diminishes the model’s ability to effectively distinguish winning QBs from losing QBs throughout all of NFL history.

And neither criticism takes away from the fact that passer rating is a real indicator of success based upon Cold, Hard empirical on-field production, while QBR is at least a partially subjective number that is even harder to comprehend than passer rating and that considers factors other than irrefutable empirical data.

Issue 1: Passer Rating’s Bias in Favor of Modern QBs
The first issue with the NFL’s passer rating model is that it favors post-Bill Walsh Era passers such as Joe Montana, Steve Young and Tom Brady over pre-Bill Walsh Era passers like Otto Graham, Sid Luckman and Norm Van Brocklin.

The reason the NFL model favors modern passers is simple: By including both completion percentage and yards per pass attempt, the NFL model essentially counts completions twice, which boosts the rating of the high-efficiency, low-risk, high-percentage passers of the post-Walsh Era.

“Our objective was to make 25 first downs a game and control the ball with short passing and selective running,” Walsh wrote of his passing technology. “Our argument was that a chance of a completion drops dramatically over 12 yards. So, we would throw a 10-yard pass.”

A comparison of the NFL’s all-time leaders in passer rating vs. the all-time leaders in yards per attempt confirms passer rating’s bias: The passer rating leader-board is the exclusive province of post-Walsh Era passers, while the list of all-time leaders in yards per pass attempt is still topped by Graham, Luckman and Van Brocklin – the most recent of whom, Van Brocklin, last played 51 years ago.

It is passer rating’s bias in favor of modern, technologically aided, short-tossers and against vanishing (and virtually extinct) long-ballers that so vexed SI’s Zimmerman.

“Passers weren't dinkers and dunkers,” a crusty Dr. Z wrote in 2007 of the pre-Walsh Era quarterbacks. “They threw downfield and took their chances.”

However, unless you pine like Dr. Z for the glory days of long, high-risk passes with plenty of interceptions, or unless you’re worried about the “Ghost of the Dutchman” rising from the grave to painfully “kick out” your sabremetric struts, this bias toward post-Walsh Era passers is irrelevant for two reasons.

The first reason this criticism is irrelevant is that every team in the NFL now has the knowledge that Walsh sought and attained and every team now uses design variations of the plays that Walsh researched and developed into a genuine new technolgy for moving the football safely toward the opponent’s goal line.

As a result of the homogenous diffusion of Walsh’s technology throughout the NFL, the current NFL passer rating model is very effective at distinguishing winning quarterbacks from losing quarterbacks without the unnecessary cost of the bizarre “winless Orlovsky > champion Roethlisberger” bug that lurks beneath QBR’s graphical user interface.

The second reason this criticism of modern bias is irrelevant is that the winningest teams are always those teams among the league leaders in passer rating on both sides of the ball – no matter which era in which they played.

Teams today certainly boast higher passer ratings than they did in the 1940s, 1950s and 1960s. But the best teams in the 1940s (Bears), 1950s (Browns) and 1960s (Packers), just like the best teams today, were those that dominated passer rating relative to the competition.

Issue 2: The Imperfect Number of 158.3
The other so-called issue with the NFL’s current passer rating model is nothing more than a cosmetic appearance or what some people call the “QWERTY” problem. The so-called “perfect 158.3” passer rating under the NFL model just doesn’t look or right.

But neither does the “QWERTY” arrangement of letters on the keyboard on which QuantCoach typed this masterpiece. For all its awkward-looking flaws, the QWERTY arrangement has defeated all comers who have tried for decades to convince keyboard users that an alternative arrangement is better.

Like the QWERTY keyboard, the NFL’s passer rating model may not look like the most efficient approach to the viewer, but users know that it is in fact very efficient and effective.

To deal with passer rating’s QWERTY problem, ESPN merely applies some statistical cosmetics to beautify the ugliness of 158.3. Indeed, Oliver has said that the “last step” in bringing QBR to life is to transform the formula’s final output to a number between 0 and 100.

“This is just a mathematical formula with no significance other than to make it easier to communicate,” Oliver wrote. “A value of 90 and above sounds good whether you’re talking about a season, a game or just third-and-long situations; a value for four or 14 doesn’t sound very good; a value of 50 is average, and that is what QBR generates for an average performance.”

Obviously, a statistic that sounds good and looks good is ideal if you are in the television business like ESPN. After all, every supermodel (the Sports Illustrated type, not the mathematical type) probably has an imperfection somewhere if you look hard enough for it. But that knowledge is not going to be enough to make the QuantCoach change the channel.

Television makeup artists have buckets of experience masking, hiding, covering, polishing, and flat-out diverting attention from the occasional super model blemish. QuantCoach hopes for the sake of all super models that any such imperfection is nowhere near has hideous as an Orlovsky wrinkle or a Thigpen scar.

The reality, meanwhile, is that ESPN has not solved any issue: 100 may sound and look better than 158.3. But rounding off the number does not change the fact that both numbers are the product of syntheitc formulas that are not absolutely equivalent to the yards gained and points scored we see on the football field. Both QBR and passer rating are descriptions of reality, which is always something at least a few percentage points different than reality itself.

The World’s First Made-For-TV Stat
If anything, ESPN’s QBR is even more synthetic than passer rating. At the end of the day, passer rating is very easy to explain: it measures a quarterback by his completion percentage, his average per attempt, his touchdown percentage and his interception percentage. QBR cannot be explained so easily.

Plus, you could just as easily round off existing passer rating, too.

Indeed, Smith said that when he created passer rating he viewed 100 as the mark of excellence. "I think our attitude was that 100 was an A," he recalled, "And anything above 100, that was an A-plus.”

Before Walsh gave quarterbacks the equivalent of calculators to help them solve problems with defenses, Smith’s approach worked just fine according to Zimmerman.

“In 1973, the first year of the system, Roger Staubauch led the NFL with a rating of 94.6. Fran Tarkenton was second at 93.1,” Zimmerman wrote. “And they were the only ones over 90, which was the original intention. A grade of 90 was superb, 100 was spectacular.”

The idea that there is one score that reflects a “perfect" passing game is a myth in any instance. There is no such thing as a “perfect” passing game under either formula: a 100 QBR is no more perfect than a 158.3 passer rating. Cold, Hard Football Facts contributor Luis DeLoureiro tackled the myth of the “perfect” passing performance last year by simply removing the artificial constraints off of passer rating to offer “uncapped” passer rating.

Nevertheless, if ESPN can get its NFL analysts to enter into a monogamous relationship with QBR, ESPN might score big with the indicator and draw a few viewers away from the NFL Network’s pre-game and post-game shows that do not have this self-described “game-changing” statistic.

Ultimately, as a television network, ESPN’s purpose in creating QBR is to attract viewers and that is the biggest difference between QBR and the NFL’s passer rating model.

QBR was created specifically for passive viewers. Don Smith created the NFL passer rating model for active users. Which statistic appeals to you most likely will depend upon which demographic most accurately describes you.