Wages of Wins author and loyal Detroit Lions fan Dave Berri recently asked me about research on whether rookie quarterbacks are better off standing on the sideline all season. One of the big questions in Detroit this year will certainly be whether overall number one pick Matthew Stafford should start at QB. The central issue is whether starting a rookie QB somehow harms his long-term development. Does a year holding a clipboard allow rookies to adapt to the NFL and boost their prospects for a successful career?

Names like Boller, Harrington, Couch, Shuler, and Carr underscore the danger of starting rookie passers. But there are also names such as Manning, Roethlisberger, Marino, Elway, and Aikman that indicate that starting as a rookie may not be so damaging to a QB's development.

This is a question I get often but haven’t looked into it because of complications associated with the issue. First, there aren’t that many top QBs to analyze—the sample size is fairly small. Any inference we make needs to keep in mind the small sample. Second, there is a problem of bias in the data. The better QBs would be the ones to earn starting jobs their rookie year, and would also likely tend to be the ones to enjoy successful careers.The trick would be to properly account for underlying QB potential, which would be quite a trick. If we knew that, that’s all we’d ever really need to know about a QB. There's no perfect way to measure that, but in the end, I think the most reasonable variable to indicate potential is overall draft pick number. It’s something that is established prior to any decision to start or not as a rookie. Also, although it is often an unreliable predictor of career performance for individual QBs, it correlates very well for QBs as a group. In other words, the number one QB might not pan out or even be better than the second QB taken in any particular draft year. But as a whole, the top picks reliably tend to outperform subsequent picks.

Data

I looked at first and second round QBs drafted between 1980 and 2004. I chose 1980 because it is roughly the dawn of the modern NFL passing rules. I chose 2004 to allow at least four years to asses each QB’s performance. Data are from Pro-Football-Reference.com. To measure career success, I used Adjusted Yards per Attempt (Adj YPA). This is passing yards minus 45 yards for every interception, per pass attempt. For QBs with very low pass attempts and spurious Adj YPA stats, I replaced their actual Adj YPA with 3.0 YPA, generally the floor for QBs with a reasonable sample size of attempts. (There is no reason to expect 1989 Chief’s 2nd-round pick Mike Elkins to lose 20 yards for every pass based on only 2 attempts.)

Methodology

The first step is to account for potential using overall draft pick number. The graph below plots career Adj YPA by pick number. The scatterplot is pretty random for any individual QB, but as a whole there is a predictable trend. Top draft picks tend to end up as NFL top passers. I estimated the expected Adj YPA based on pick number. A QB from the top of the first round would be expected to average 5.0 Adj YPA, while a QB from the bottom of the second round would be expected to average 4.1 Adj YPA.

We can see a clear trend. Unsurprisingly, top picks would be expected to end up with better career passing stats than later picks. A linear regression estimates what the baseline expectation should be for each slot in the draft.

Next I calculated the 'Adj YPA above expected' for each QB. Now we can compare QBs who started their first year to those that didn’t, while holding “potential” equal.

But how do we define “started their first year?” How many starts qualifies--4, 5, 11? I don’t know, so let’s start by looking at the whole picture.Results

We can use that baseline to compare each QB's career Adj YPA. Some QBs did better than expected given their draft slots and some did worse. We can now test if there is a connection between better than expected performance and the number of rookie year starts.

As it turns out, it appears that QBs with more rookie starts tend to enjoy greater career success, even accounting for draft order.

Some of you might have noticed that what I've really done here is a crude multivariate regression. Holding for draft order, I estimated the effect of games started. What if we just do the regression directly?

As expected we get a small negative effect with draft order. (The higher the pick number, the worse the expected stats.) The Games Started variable is positive and significant at p=0.03. The model as a whole has an r-squared of 0.15--small in absolute terms, but considerable given the highly random individual variance in QB careers.

But this is only one way of looking at whether a QB was a starter or not. What if we draw a line at say, 5 rookies starts--below 5 starts he's not a rookie starter and above it he is. The group of QBs with 5 or less starts averages -0.01 Adj YPA above expected, and the group with 6 or more starts averages +0.3 Adj YPA above expected. If we define it at zero starts, those with no rookie starts averaged -0.03 Adj YPA above expected, while those with at least one start averaged +0.02 starts above expected. In fact, no matter where I chose the endpoints of the groups, from 3 to 13 starts, the group with more starts outperforms the group with fewer starts by about 0.4 Adj YPA.

Conclusion

Does this mean teams should rush their rookies out to face the onslaught of NFL defenses to somehow make them better? I really doubt it. If I had to bet, I'd say that we simply haven't fully accounted for QB "potential" using draft order alone. I think the better QBs, those with the best chances of career success, often gain and maintain starting positions earlier.

But at the very least, we can say this: Given this analysis, there is no reason for a coach to arbitrarily keep a rookie QB on the bench. He should start his best QB, rookie or not, and not worry about incubating him under a ballcap on the sidelines. In the end, it should be the coach's qualitative judgment on the readiness of the player.

Here are the QBs and their stats I used for this article. (It's interesting just to see who the QBs are who exceeded expectations. There are some surprising names--Pennington and Batch at the top, for example. And Is Eli Manning really worse than David Carr? Wow.) You can sort the table by clicking on the column headers.

19 Responses to “Should Rookie QBs Start?”

Looks like the effect is still mostly selection bias. Since teams likely change their evaluation of rookies significantly between draft day and the middle of the regular season, accounting for draft order doesn't eliminate the bias. High draft picks that look great in preseason/practice are more likely to succeed and get more starts than high draft picks that struggle.

Good work, as usual, Brian. I actually started working on this right after the draft for the same reasons you did, but trashed the project after seeing pretty similar results. I essentially got a small sign that starting early is better, but like you, I am hesitant to decide which way the causation arrow points.

One general question -- I assume you are not putting in actual draft order into your regression? Since draft potential, as measured by draft order, is not linear, you wouldn't want to do that. It's the exact reason I came up with this post (http://www.pro-football-reference.com/blog/?p=527). So whenever I'm doing a draft analysis like this, I replace actual draft order with the numbers in those posts.

It's a very difficult question to answer. Even accounting for draft position, coaches get to see a full training camp out of a guy. Those who aren't very good will have no chance of starting; those who look good will have a chance. So we already have some data (albeit one with lots of noise in it) on a QB besides his draft position by game 1 -- was he an opening day starter?

In the end, I suspect there's no career difference between a guy starting and not starting as a rookie. And certainly, we haven't seen any evidence to the contrary.

Thanks, Chase. I should have been more clear on the regression. The dependent variable was Adj YPA (not 'Adj YPA above expected'), and the predictors were draft order and rookie games started.

I agree with you on the non-linearity of potential, but strictly for Adj YPA, it does look linear, at least for the first 2 rounds. (See First graph above). Maybe Adj YPA doesn't really capture how truly 'good' a QB is. Or, the utility function of YPA isn't linear. In other words having a 5.0 YPA QB is more than 25% better than having a 4.0 YPA QB.

I don't think this proves anything, just that using this methodology we can't really detect any advantage to waiting. And even if the methodology doesn't capture every little nuance and bias, the true effect must be small enough to be engulfed by other effects, and is therefore probably not big enough to worry about.

If I have time today, I might rerun it with your AV numbers instead of raw draft order, and see how it turns out.

Your site is fast becoming one of my personal favorites. I will add your site to my blogroll at TheSportsDick.com.

If you disregard draft position and look at what really matters - the adj YPA, four of the top six and 10 of the top 15 performers started four games or less during their rookie season. Conversely if you look at the worst performers, only three of the bottom 20 started more than four games. From that data alone you can conclude that starting a rookie slightly improves their likelihood of a successful career.

The problem with coming to any conclusing by analyzing this data is that it's largely irrelevant. The factors that determine a successful quarterback include the talent around the rookie going in, the quality of coaching he receives, whether he earns the job or inherits it due to injury, etc. How many of the top QB's would still be at the top of the list if they had been drafted by Detroit? I submit that not all of them would have enjoyed the same success as they did.

It'd be extremely difficult to quantify the intangibles. I pose this question. Is it safe to assume that the fact that a team drafting a QB in the first two rounds indicates that the franchise is coming off of a bad year? No one drafts a QB to "take the best player remaining". Generally a quarterback taken on one day one is to fill a need. A team that needs a QB usually isn't in good shape.

I know this is a bit of a stretch. To support this, consider that of the 51 QB's taken in the first round, only 10 of them were taken in with one of the last 10 picks of the round - the picks owned by presumably strong teams (I know there are anomalies caused by trades and such).

Of the 22 2nd rd QB's taken, only five were among the last 10 of the round.

It would appear that the median number for avg YPA is around 4.8 or so. Of the 36 QB's below average, only eight of them were drafted with the last 10 picks of either round. Of those eight, none of them started more than five games their rookie year.

Looking at the number of games started the rookie year of all the players on the list, the median number is just less than 4. 13 of the 36 teams above that threshold were below the 4.8 adj YPA metric. Of those 13 'busts', seven were taken at the end of their respective round.

To clarify, of 15 late rounders, seven were busts that started four games or more.

If 4.8 is the threshold, than only one #1 overall pick failed to meet that standard. Of those 11 #1's, only three of them failed to start seven games or more. Interesting enough, of the eight players selected # 2 or 3, five of them were busts.

Of all of the busts, almost none of them were drafted by teams that had ever won a Super Bowl.

I think that's about the only conclusion you can come to. That bad teams draft badly. The performance of their rookie quarterbacks is nothing more than a reflection of the ineptitude of the team that selected him.

It would be interesting to see the results if you compared players who started, say, 10+ games to those who started 0 games in their rookie year. This would (mostly) filter out rookie QBs who were benched after a few bad starts, presumably because they were performing poorly. With these benched rookie QBs in your dataset, it's not surprising that players with more rookie-year starts appear to have better careers: they performed better early in the season because they were simply better players (at least on average) to begin with!

Brian,Really interesting study here. As a Jets fan, I've been wondering about this recently: historically, it seems to me most quarterbacks drafted in the first round will start early if they have inexperienced QBs on the team already (Ryan, Flacco), but will sit and learn for a while if there are decent, veteran QBs on the roster (Leinart/Warner, Rivers/Brees...). I agree that - using the Jets as an example - if Sanchez turns out to be better than Clemens, et al (with 8 career starts), he should start right off the bat and will benefit from a good team around him and the experience gained in the long run. On the flipside, if they happened to have a journeyman QB like Jeff Garcia, showing him the ropes, I wouldn't be surprised if sitting for a year and learning the system & the playbook helps a young QB over time.

I'm not sure I follow. I can see how that would affect number of rookie games games started, but I don't see how that affects the career prospects of the rookie. In other words, it would affect the independent variable, but not the dependent variable I'm interested in.

My hat is off to you Brian, you did an amazing job with this study. My only concern is that the primary measure of success, yards per attempt, is influenced by the style of offense a quarterback plays in. For example, a system that runs on first down, runs on second down, and then asks the passer to make a play on third and long (i.e. Eli Manning's Giants) would artificially hurt a ranking, while an offense that regularly passes on first down - when most teams play to stop the run - would artificially help it (i.e. Daunte Culpepper's years in Minnesota). But this is a small flaw that would not significantly improve variance prediction, and would be hard to quantify.

As a Lions fan, the question of how soon to play a rookie QB is pretty pressing right now. We screwed up Joey Harrington by turning him into a check-down machine while running for his life, and the general thought is that he started before he was ready. On the other hand, Chuck Long sat for the better part of three years before he saw the field and just couldn't do it on the field by then. It was like he forgot how to play in a game.

So, with Matthew Stafford and still not a very good offensive line, this data suggests that the Lions should go ahead and start him as soon as the coaching staff deems he's ready. There are successful QBs and busts that didn't start as a rookie, or that started full seasons. He'll either be good or not based on his talent and preparation. A long incubation doesn't help.

I just came across your blog today coming from the NYT and I'm really happy I did. It is great to see what you are doing.

I would like to suggest a couple of things for this particular analysis. You can approach it as using Analysis of Covariance (ANCOVA) to account for the draft order.

This approach would be more powerful because I'm assuming that the first model you use to define your response variable (above or below average) is not very strong to begin with.

As other people suggested, I'm not sure that YPA is the best choice because it depends on the type of offense. Maybe QB ratings could work better? Maybe adjusting for number of games played.

Also the choice of the variable number of games started as a rookie is the best. You found that those QB's who started more games are a little better than those who started a smaller numer of games. That is something to be expected, because if the QB had sucess in the first games he plays, that means the coach would be more likely to start him again.

This is a great blog and I'll be following your work. I hope this will encourage more people to learn and enjoy math.

I've looked over QBs drafted just in the first round (and Drew Brees since he's basically a first rounder) since Manning-Leaf up till Young-Lineart. Rookie first rounders who got the starting job before game four or five tend to bomb except for Manning, Big Ben and Leftwhich. Of those three, Big Ben and Leftwhich were playing for quality teams. The only QB I'm not sure about is Akili Smith since I haven't been able to find what games he did start in his rookie season.

I'm currently using your data for my Junior Research Project based on the thesis that Young QB's take longer to develop than runningbacks, due to the new offense,recieving cores, and defenses. Can you think of any other factors that can make or break a rookie quarterback?

@BBurkeESPN

ANS COMMUNITY

Interested in publishing your own football research, analysis, or stat-based commentary? Advanced NFL Stats Community is the site to share your thoughts and ideas. There's plenty of data available to get started. All submissions will be accepted and published. Check it out!

Support Military Families

If you enjoy Advanced Football Analytics, please consider a small donation to The Fisher House, a place where families of injured servicemen can stay while visiting their hospitalized heroes.