Most Recent FO Features

There's a serious need for defensive help in Kansas City, Los Angeles, and Oakland. In Denver, meanwhile, the Broncos must determine whether or not Case Keenum can really be a long-term solution at quarterback.

Most Recent Extra Points

Introducing QBASE

by Andrew Healy

On Thursday, a quarterback is almost certain to go No. 1 overall in the NFL draft for the 13th time in the last 20 years. And while there was a great deal of optimism about all of those quarterbacks at the time they were drafted, the numbers suggest that optimism was only sometimes warranted. For all their throws that leaped off the film, red flags should have jumped off the stat sheet for previous No. 1 picks such as Tim Couch and David Carr. Those same warning signs apply to Jameis Winston. Our new Quarterback Adjusted Stats and Experience (QBASE) system finds that, even ignoring his off-field concerns, the odds are against Winston ever becoming an elite quarterback.

Marcus Mariota, on the other hand, has all the statistical markers that previous elite quarterbacks have had. Other college quarterbacks whose numbers looked as good but who failed in the NFL have had at least one statistical weak point that marked them as potentially fraudulent according to QBASE. By the numbers, Mariota has no such weakness. While the numbers may mean a little less in his case due to Oregon's system, highly-drafted quarterbacks with his statistical resume have been the best bets to succeed in the NFL. QBASE does not call Mariota a sure thing. In contrast to Winston, Mariota would be a worthy pick at the top of the draft.

To predict NFL success for this year's quarterback class, QBASE looks at a range of statistics that we describe in detail below. Those statistics account for the opposing defenses that each quarterback faced in college and the quality of his offensive teammates. Based on those adjusted stats, QBASE conducts 50,000 simulations to estimate the passing Defense-adjusted Yards Above Replacement (DYAR) each quarterback will generate in the third, fourth, and fifth years of their NFL careers.

QBASE finds that only two quarterbacks in the 2015 draft are better than even-money bets to avoid being NFL busts. The presumptive No. 1 pick is not one of them.

Methods

To project NFL success for the 2015 quarterback prospects, we build on our earlier college quarterback projection systems that started with David Lewin and his Lewin Career Forecast. Our new model, QBASE, considers a wider range of numbers that measure a quarterback’s performance before boiling it down to just one number. The model then uses that number along with experience and projected draft position. At that point, limiting the regression to three variables reduces the risk of overfitting previous data.

Important to note is that we are projecting passing DYAR in Years 3-5. Rushing DYAR are not part of the model, and we skipped the first couple of years of a player's career because quarterbacks tend to develop slower than players at other positions and often don't play significant time until their second or third seasons.

Right now, we limit the model to top 100 picks since doing so produces the best model for predicting quarterbacks who get NFL playing time. We also limited this model to just FBS players so that we could use the same defensive corrections for everyone. Therefore there is no QBASE projection for players such as Joe Flacco or Tarvaris Jackson.

Each prospect's QBASE projection is based on three main ingredients:

1. College performance, adjusted for opposition and teammates: The strongest predictor of NFL success. To account for a changing college game, we look across three aspects of performance:

The best prospects succeed across all three of these areas, so we take the minimum performance across the three areas. A quarterback who has an inflated completion percentage because of a screen-heavy offense (e.g., Brandon Weeden) will score lower in QBASE due to his lower adjusted yards per attempt.

We make adjustments for team passing efficiency for 2005-14, the years for which passing efficiency data is available. Most importantly, team passing efficiency is the model component that adjusts for a quarterback's propensity for taking sacks; sack avoidance is an important part of an efficient passing game. While official college football statistics treat sacks as running plays, the S&P ratings correctly view these plays as passes. Last week, our projections on ESPN Insider for the 2015 draft class did not account for 2014 team passing efficiency. The final projections reported below do. Two projections (Brett Hundley and Sean Mannion) are now lower as a result. We scale the adjustment to account for quarterbacks who didn't play a full season.

QBASE makes two important adjustments to the raw numbers, accounting for the strength of opposing defenses and the quality of a quarterback's teammates. The first of these corrections makes our measure more accurate for a player such as Josh Freeman, who faced the 99th-best set of defenses in his last college season. The strength of opposing defenses back to 1995 was measured in the same way as Pro-Football-Reference's Simple Rating System.

The second correction gives more credit to players such as Philip Rivers, who succeeded in college without great surrounding talent, as opposed to a player such as Matt Leinart, who played for an NFL finishing school. We measured teammate quality based on the draft value of offensive teammates in both the player's draft year and the following year. For this year's prospects, these measures are based on mock drafts for both 2015 and 2016.

2. College experience, adjusted for quality: Our previous measure of total games started has been replaced with a measure that counts seasons with at least 150 attempts, with adjustments to count poor seasons less than good ones. Experience counts, but successful experience counts more. This variable can capture the better quality of players who get more starts and more opportunities to improve, as well as the underappreciated idea that players who succeed in a smaller sample may not live up to that short-term success. Mark Sanchez could have gotten lucky to do so well in his one season as a starter. Four-year starter Russell Wilson? Not so much.

3. Projected draft slot: Based in large part on scouts' ratings of quarterbacks' intangibles, accuracy, and other attributes for recent years, the draft slot accounts for how scouting information predicts players' NFL success. We used mock drafts to project draft slot for this year's quarterbacks.

Historical QBASE Projections for Passing DYAR

Among the 65 quarterbacks drafted in the top 100 between 1997 and 2010 (i.e. those who have completed Years 3-5), the players with the most overprojection are colored red and the players with the most underprojection are colored green.

QBASE Projections for Top 100 Picks Since 1997

Player

Drafted By

Pick

Year

Predicted DYARin Years 3-5

Actual DYARin Years 3-5

Philip Rivers

SD

4

2004

2317

2679

Carson Palmer

CIN

1

2003

2266

2268

Donovan McNabb

PHI

2

1999

1946

1075

Russell Wilson

SEA

75

2012

1561

503*

Robert Griffin

WAS

2

2012

1519

-374*

Peyton Manning

IND

1

1998

1463

3922

Byron Leftwich

JAC

7

2003

1200

369

Aaron Rodgers

GB

24

2005

1198

1891

Ben Roethlisberger

PIT

11

2004

1193

1381

John Beck

MIA

40

2007

1151

-143

Matthew Stafford

DET

1

2009

1125

3021

Andrew Luck

IND

1

2012

1076

879*

Chad Pennington

NYJ

18

2000

1069

2631

Christian Ponder

MIN

12

2011

1061

-188**

Daunte Culpepper

MIN

11

1999

1061

1620

Player

Drafted By

Pick

Year

Predicted DYARin Years 3-5

Actual DYARin Years 3-5

Cade McNown

CHI

12

1999

972

0

Teddy Bridgewater

MIN

32

2014

945

--***

Jay Cutler

DEN

11

2006

936

831

Danny Wuerffel

NO

99

1997

928

-160

Matt Leinart

ARI

10

2006

915

-56

Eli Manning

NYG

1

2004

892

1179

Jason Campbell

WAS

25

2005

891

666

Brian Brohm

GB

56

2008

853

0

Tim Tebow

DEN

25

2010

849

-9

Geno Smith

NYJ

39

2013

839

--***

Kevin Kolb

PHI

36

2007

830

33

Kellen Clemens

NYJ

49

2006

800

-92

Jake Plummer

ARI

42

1997

790

266

Cam Newton

CAR

1

2011

781

316**

Alex Smith

SF

1

2005

771

-763

Player

Drafted By

Pick

Year

Predicted DYARin Years 3-5

Actual DYARin Years 3-5

Drew Brees

SD

32

2001

737

1822

Vince Young

TEN

3

2006

690

616

Derek Carr

OAK

36

2014

641

--***

Sam Bradford

STL

1

2010

617

692

Colt McCoy

CLE

85

2010

610

-19

Andrew Walter

OAK

69

2005

583

-227

JaMarcus Russell

OAK

1

2007

535

-834

Johnny Manziel

CLE

22

2014

486

--***

Blake Bortles

JAC

3

2014

471

--***

Matt Schaub

ATL

90

2004

437

1181

David Greene

SEA

85

2005

432

0

Tim Couch

CLE

1

1999

428

-366

Nick Foles

PHI

88

2012

392

264*

Charlie Frye

CLE

67

2005

368

-271

David Carr

HOU

1

2002

365

-215

Player

Drafted By

Pick

Year

Predicted DYARin Years 3-5

Actual DYARin Years 3-5

Brandon Weeden

CLE

22

2012

325

21*

Brady Quinn

CLE

22

2007

262

-207

Drew Stanton

DET

43

2007

257

174

Quincy Carter

DAL

53

2001

248

263

Chad Henne

MIA

57

2008

220

183

Matt Barkley

PHI

98

2013

208

--***

Akili Smith

CIN

3

1999

198

-55

Shaun King

TB

50

1999

183

-102

Joey Harrington

DET

3

2002

178

-149

EJ Manuel

BUF

16

2013

170

--***

Ryan Mallett

NE

74

2011

168

120*

Matt Ryan

ATL

3

2008

158

3438

Jimmy Clausen

CAR

48

2010

153

-5

Brock Huard

SEA

77

1999

139

-8

Andy Dalton

CIN

35

2011

138

778**

Player

Drafted By

Pick

Year

Predicted DYARin Years 3-5

Actual DYARin Years 3-5

Ryan Leaf

SD

2

1998

105

-727

Jim Druckenmiller

SF

26

1997

54

0

Colin Kaepernick

SF

36

2011

22

882**

Blaine Gabbert

JAC

10

2011

14

-429**

Jake Locker

TEN

8

2011

2

-102**

Chris Redman

BAL

75

2000

-45

-67

Rex Grossman

CHI

22

2003

-82

-175

Dave Ragone

HOU

88

2003

-114

0

Pat White

MIA

44

2009

-155

0

Brian Griese

DEN

91

1998

-172

2004

Mark Sanchez

NYJ

5

2009

-184

-649

J.P. Losman

BUF

22

2004

-192

-310

Josh Freeman

TB

17

2009

-194

-154

Kyle Boller

BAL

19

2003

-222

56

Brodie Croyle

KC

85

2006

-226

-62

Player

Drafted By

Pick

Year

Predicted DYARin Years 3-5

Actual DYARin Years 3-5

Patrick Ramsey

WAS

32

2002

-234

-169

Chris Simms

TB

97

2003

-318

-166

Marques Tuiasosopo

OAK

59

2001

-348

-49

Charlie Whitehurst

SD

81

2006

-358

-141

Michael Vick

ATL

1

2001

-446

-518

Mike Glennon

TB

73

2013

-486

--***

Kevin O'Connell

NE

94

2008

-499

0

Charlie Batch

DET

60

1998

-530

59

Trent Edwards

BUF

92

2007

-647

-564

Ryan Tannehill

MIA

8

2012

-683

630*

Brock Osweiler

DEN

57

2012

-791

-5*

Josh McCown

ARI

81

2002

-1304

-102

* Year 3 only** Years 3 and 4 only*** Not yet reached Year 3

QBASE's all-time favorites are Philip Rivers and Carson Palmer. Each was a four-year starter who excelled across the board statistically as a senior. Donovan McNabb and Russell Wilson also fit that bill. Robert Griffin projected more as the first-year player he was in the NFL than the one we have seen for the last two seasons. If QBASE was the GM of a quarterback-hungry team, it would probably be trying to steal Griffin now for a mid-round pick.

Peyton Manning shows the importance of opponent adjustments. As a senior, Manning faced the strongest schedule of opposing defenses in Division I. Accounting for that strength of schedule pushes his projection to the sixth highest since 1997. Some responses to the original QBASE article were critical of the system for "missing" on Manning because of the huge gap between his projection and his actual performance, but no system based on regression analysis would ever project a player to be as statistically dominant as Manning was between 2000 and 2002. Manning's projected DYAR in comparison to the projected DYAR of other prospects is a better guide to how the QBASE system judged him: as one of the top quarterback prospects of the last two decades.

Speaking of Manning, QBASE does a good job of separating out the successes and failures of No. 1 overall picks. The most notable failure among them is Alex Smith, whose -763 passing DYAR in Years 3-5 is lower than all other No. 1 overall picks in the sample except for JaMarcus Russell. Still, QBASE essentially predicted Smith to become a serviceable starter, which is basically what he became after getting off his early-career offensive coordinator carousel. Michael Vick's projection may be surprisingly low, but his actual passing DYAR was even lower in Years 3 through 5. Accounting for his rushing ability would push Vick's projection, and that of other running threats, higher.

For its successes in capturing past quarterbacks, QBASE also has some notable misses. Included among the quarterbacks with projections over 1,000 DYAR are notable failures John Beck and Christian Ponder. Beck was one of three quarterbacks in the data to be 25 or older by the end of his draft year. The other two (Jim Druckenmiller and Brandon Weeden) were also NFL failures, as was a quarterback who just missed being selected in the top 100 picks, Chris Weinke. So we could add a variable for age to get a lower prediction for Beck. But with such a small sample size of older quarterbacks, the effect of the age variable cannot be estimated with precision. We preferred to keep the model simple and leave that variable out. We will consider including it in future versions of QBASE if the data warrants that.

On the low end, QBASE is far too negative on Matt Ryan and Ryan Tannehill. While QBASE beats the scouts in cases such as Russell Wilson and David Carr, Ryan and Tannehill appear to be two notable cases where the scouts saw something that QBASE missed. In the historical table, Ryan falls right in the middle of a long run of quarterbacks like Joey Harrington and Blaine Gabbert who were just as terrible as their adjusted stats and experience predicted. But only Peyton Manning's DYAR (in Years 3-5) eclipses Matt Ryan's among all the top 100-drafted quarterbacks since 1995. We would have expected about one of the bottom 50 quarterbacks on the list to be elite, and Ryan appears to be that one surprising outlier.

(Brian Griese is also an outlier, but this seems to be more about an inconsistent player whose best seasons happened to fall right in the period being projected by QBASE. Griese averaged 408 passing DYAR per season between 1999 and 2005, but his two best years -- and only two of the three seasons where he had positive DVOA -- happened to fall in 2000 and 2002, his third and fifth seasons.)

Tannehill's projection comes out so low because he had just one season as starter at Texas A&M. QBASE hits him pretty hard for that. As with the age variable for Beck, we have opted for simplicity and kept the basic experience variable as the number of years with more than 150 attempts. We will certainly keep an eye on it, though, and consider revising to limit the penalty that QBASE currently imposes on one-and-dones like Tannehill.

Projecting most previous quarterbacks accurately is, of course, not surprising. The model is designed to do exactly that. As with the previous quarterback projection systems, the test of QBASE will be how well its projections are calibrated to future outcomes. QBASE will be validated not if Jameis Winston fails and Marcus Mariota succeeds, but if quarterbacks projected to fail 60 percent of the time indeed fail about that often. So let’s leave behind predicting the past and see QBASE’s first predictions for future NFL quarterbacks.

Projections for the 2015 Draft Class

Using the historical data, QBASE generates projections for both the average performance and the range of possible outcomes for each quarterback in the 2015 draft class. QBASE is appropriately cautious with its predictions. Even when QBASE thinks the odds are stacked against a quarterback, it recognizes that there is some chance that he will defy those expectations.

Winston faces long odds of becoming a successful NFL quarterback. QBASE gives Winston a 61.3 percent chance of being a bust (less than 500 DYAR in Years 3 through 5) and just a 12.8 percent chance of being at least an upper-tier quarterback. His projection here is higher than it would be if the stats did not correct for his tough schedule of opposing defenses. According to our numbers, Florida State only faced the tenth-toughest schedule overall in 2014, but it faced the nation's toughest set of defenses. Still, Winston's projection puts him just third in the 2015 draft class, well behind both Marcus Mariota and Brett Hundley.

If Tampa Bay picks him, QBASE will give Winston the third-lowest projection among the 13 No. 1 overall quarterbacks since 1996. David Carr and Michael Vick are the only two top selections who would rank lower. And nobody with a QBASE projection in Winston's neighborhood has been worth the top pick.

QBASE finds fault with Winston for the same reasons it dislikes Couch and Carr. All three quarterbacks started for only two college seasons. Also, all three had good-not-great stats in their last college season. QBASE wants to see high levels of performance across all passing statistics and Winston has the same weakness as Couch: an adjusted yards per attempt that is not as good as his completion percentage. Winston’s raw yards per attempt ranked 20th in Division I last year. After an upwards adjustment for his tough schedule and downward adjustments for his interceptions and quality of teammates, Winston still ranks 20th in our adjusted measure of yards per attempt.

Note that the QBASE projection does not account for the hard-to-quantify potential concerns surrounding Winston's off-field issues or his weight. Any adjustment for those issues could push Winston's bust potential even higher. Winston simply does not look good like a first pick should.

Mariota has the highest projection of any player since 2012. Since 1995, only six quarterbacks (Philip Rivers, Carson Palmer, Donovan McNabb, Russell Wilson, Robert Griffin III, and Peyton Manning) had better projections. QBASE sees Mariota as a three-year starter who posted huge numbers without any weak points. Adjusting for opposition and teammates, Mariota's adjusted yards per attempt relative to other Division I quarterbacks in his last college season trails only Wilson and Griffin. Also, QBASE likes that Mariota's completion percentage is high relative to his peers.

But are Mariota's numbers a product of the talent that surrounded him and the system in which he played? Mariota's projection accounts for Oregon having two tackles and a center projected to go in the early rounds of the 2015 and 2016 drafts, but makes no adjustments for Oregon's unusual pace of play. Questions about Mariota's ability to adapt to a more standard NFL offense do lend a note of caution to his projection. At the same time, the model also ignores Mariota's potential off-field strengths. And the two other quarterbacks with top-ten projections who got the most questions about their college production translating to the NFL -- Wilson and Aaron Rodgers -- both turned out well, although the concerns with Wilson and Rodgers were different from those with Mariota.

Simply put, there has not been a quarterback in the last three drafts with Mariota's chances of being an upper-tier to elite-level quarterback. He is far from a sure thing, but quarterback-hungry teams should not let Mariota slip by.

The final adjustment to include 2014 team passing efficiency knocks down Hundley's projection by 144 DYAR. His projection now penalizes him for his worrying tendency to take sacks. While Hundley is now about 10 percent more likely to be a bust than an upper-tier or elite quarterback, he still has upside for a team grabbing him in the second round. His 26 percent chance of being upper-tier or elite is about twice as big as Winston's. Hundley's revised projection puts him on roughly equal footing with Teddy Bridgewater.

While not as highly ranked as his completion percentage, Hundley's adjusted yards per attempt is more impressive than it seems at first glance. Hundley faced the third-toughest set of opposing defenses in Division I last year. (As noted earlier, Florida State had the toughest schedule of any offense; Alabama was No. 2.) He also had fewer future early-round offensive teammates than either Mariota or Winston.

Petty projects to be substantially worse than replacement level, in large part because QBASE questions the opposition that he faced in 2014. Petty accumulated his college stats against the 70th-toughest slate of opposing defenses. His 6.1 percent chance of developing into an upper-tier quarterback makes Petty unworthy of a third-round selection.

Grayson projects poorly for some of the same reasons as Petty. He faced college football's 73rd-toughest set of defenses last year. If Grayson gets picked late in the third round, he will have the ninth-lowest projection of any top 100 quarterback in the last 20 years. Mel Kiper recently had Grayson going in the middle of the second round to the Bills. Taking Grayson anywhere near that high would be about as bad as paying Charles Clay like he was Shannon Sharpe circa 1994.

Mannion, like most middle-round quarterbacks, is a likely bust. His bust potential is even higher now given the final adjustment to account for Oregon State's 80th-ranked passing offense in 2014. But Mannion has the highest chance of NFL success outside the top three prospects. With an 8.7 percent chance of being an upper-tier quarterback, Mannion is the quarterback in the next group most worthy of a mid-round flier.

QBASE does not use the change from the previous year to the last college season. I wanted to distill the model to as few numbers as possible. That one wouldn't make the cut, but I also don't love the idea that someone could have a lower prediction by improving their performance (in the next-to-last year by making the change in the last season lower). To me, I think that is a sign that kind of variable is good for predicting the past by chance and not all that useful for the future. Any improvement in performance in any year should not decrease a projection.

So Luck isn't penalized for having a negative change in performance his last year. His projection would just be even higher if his final year numbers were even better.

The parameter estimates are the same for each projection, so not out of sample. I did split the sample to test the model. The parameter estimates are randomly drawn in each simulation to account for uncertainty about the true values.

Wow - of the young QB's still developing, it REALLY seems to have whiffed on Ryan Tannenhill. By the end of his 5th year, he's almost definitely going to be VERY off from his projection...

And its belief in Christian Ponder and John Beck is pretty amusing. Any reason Ponder doesn't get the red "underperformance" designation? Is it because he hasn't reached the 5th year yet? Overall, the system seems a little stronger at picking who will be bad, than who will be good - of course, a QB is much more likely to flame out than to succeed...

In fact, I'm not sure how the "over/underperformance" tag is being applied - just glancing at it, it seems to have been notably off on McNabb, Brian Brohm, Byron Leftwich, Kellen Clemens and Kevin Kolb, as far as over-projecting is concerned. Matt Schuab is the only under-projection I see that doesn't get highlighted - was the worry about too much red?

We should probably wait before making decisions on Tannehill, but he does look like a big miss for QBASE. I think there are cases where we'd want to put less weight on QBASE. Beck would be one case because of his age and you could argue Tannehill, too, given his unusual path to playing QB. But that may be post-hoc.

And I agree on system seeming to work best at predicting high pick busts. In fact, that's how I would use it if I was a GM: to avoid the QBs like Carr and Couch that just didn't make much sense to pick at the top of the draft.

I think it makes perfect sense that the system mistakenly punished Tannehill for only starting one year, as he was a WR earlier in college. This reminds me of an model for QB success Chase Stuart published in his site where he said if Tannehill became a successful QB after the way his first two seasons went statistically he would have taken a very unusual path to it. I guess he's just been an unusual player all around.

Tannehill started for 1.5 seasons, beating out Jerrod Johnson in the 7th of 13 games in 2010. It was performance based, rather than injury, so maybe it needs to be counted as 2 seasons as starter if the model can't adjust for partial seasons.

What happens if you use 1.5 seasons as starter, or 2 seasons & prorate the 2010 data for a full season if partial years aren't allowed?

If you gave Tannehill credit for two seasons, his projection would be +16 DYAR. So still low, but not nearly so low. Either way, Tannehill's unusual college situation is maybe a good example of when the scouting info should get more weight and the stats less.

If only we could hook up these college qbs to small ekg and eeg monitors during their college careers, then we could make an adjustment for the guys who, when holding the ball in the pocket, have heartbeats like a hummingbird's, with brain wave activity that is confusable with the football's.

I think by including experience as a variable you may be convolving two concepts. Are you including it because more college experience makes a better NFL QB, or because more college experience gives us more confidence that good college performance wasn't a random fluke?

If it's the latter, then I don't think it should be combined in a regression with variables to project expected performance, but rather should point towards a separate measure of confidence in the progression. In other words, the concept should be thought of in terms predicting a true sample mean (the QB's actual "goodness", on average) and a confidence in our ability to project/measure that based on college performance. This could help with the Tannehill problem.

Imagine you're measuring two things. You get just 5 measurements of the first thing, A, with values of 85, 90, 91, 92, and 97. You get over 100 measurements of the second thing, B, with values as low as 81 and as high as 95, but with an average of 88. Is A better than B? On average, from what we know, A is projected to be better, but we have much lower confidence in that measurement, whereas we're pretty sure B will average around 88. In other words, we can say, perhaps, that B's average will be 88 +/- 1 (with some level of confidence), whereas A will be 91 +/-10 with the same confidence. If you're drafting, should you draft A or B? Well, if the measure of a "good QB" is "at least 80", then A is the obvious choice since both will be good and A has more upside. But if you want a QB who is at least an 87, then probably there is a higher probability, given what we know, that B is a safer pick.

Similarly, I wonder if you could formulate the model to project a player's pro performance just based on whatever college sample size you have, and then frame the confidence in the projection according to how much data that projection is based on. You might then see, maybe, that Tannehill had a lot of upside but was a risky pick, whereas, someone like Ryan was a solid choice but with a lower ceiling...

Games started is included because, going back to Dave Lewin's original research, it correlates well with pro success. The same is true of completion percentage, and (I assume) the other factors that have been added.

My assumption is games started is significant largely because it increases the amount of data for the other two factors; if you're a one-year guy like Sanchez, there's simply not enough data to project that your completion % and YPA are meaningful. There's simply far more possible long-term variation of performance with that limited of a subset of data, so the model adjusts accordingly. The more years of consistent data you have, the less probable variation you have in future performance, and the model accordingly scales up.

I don't see "he started a lot of games" as being significant in any way, but "he started a lot of games and we therefore have more data to base projections on" strikes me as something really useful.

1) The player beat out his competition. Except in rare instances, an NFL caliber player should be a starter by his second year. Being unable to unseat a non-NFL caliber player is a worrisome sign.

2) Actual experience. Doing things usually makes you better at said things. There is some thinking that college offers a learning curve too. If you took a HS QB and him on an NFL team, he's going to fail no matter how talented he is, but a couple years at a mid level of difficulty lets him adjust and be prepared. See also all the NFL Europe QBs that eventually had success.

3) What you said. More film means more chances of scouts correctly assessing you.

I think the point is that level of confidence is included in the percentiles, in a matter of speaking. Many players will look reasonable in college and then completely overmatched in the pros. The smaller sample's far lower confidence projects as a much greater likelihood of being a bust.

It's also conceivable, given the consistently powerful correlation between length of playing time in college and probability of pro success, that having played before is a separate factor in success, apart from sample confidence. Players generally have to make a very significant transition into the pros, and increasingly, they get very little time to make that transition. Perhaps a player like Russell Wilson is more well equipped for it based on extended playing time. Perhaps the playing time makes the players inherently more versatile and able to adapt.

Great points. I spent a while working on exactly these ideas. Basically a Bayesian updating framework where you start with a prior of average ability and get more opportunities to update when you have more years of college experience.

So experience was not going to enter the model separately as you say. It is the least satisfying part of the model to me, to be honest. Do I think Andrew Luck would have been any worse an NFL quarterback if he came out as a sophomore? No, not really.

The model currently does better than just having years of experience in there. The adjustments for quality of play end up meaning that the experience variable is actually part experience and part performance. So Luck gets some extra credit for being really good for his three seasons as starter.

And I did look to see if the variance of the prediction should depend on years as the starter. I expected fewer years to lead to higher variance as you suggest. But I didn't actually find evidence for that in the end. Surprising. I'll keep looking at this.

I would think that age would be a significant variable. There is a big difference in upside/potential of someone who graduates at the age of 21 and someone who was held back for athletic reasons or was born early in the school year and red shirts in college and graduates at 24.

Have you done (or would anyone please do) a validation study on the use of mock drafts in place of draft value for upcoming prospects? It seems to be an important assumption, and one would like to know how that variable substitution affects accuracy. Should be relatively simple since past mocks should still be available to compare to actual draft status.

I also think RG3 needs to get out of DC, but think that if any place tailor-made for him it's Philly as long as Chip Kelly is there. He'd also look good in San Diego with McCoy, who modified his offense to make Tebow look reasonable.

From the article: "Right now, we limit the model to top 100 picks since doing so produces the best model for predicting quarterbacks who get NFL playing time."

I think part of that is once you get beyond that point, there are often very good reasons the QB wasn't drafted, e.g. weak arm, system QB, not accurate, off-the-field issues, etc. Brennan at #186 would be well outside that range.

I realize this has nothing to do with QBASE; but I just wanted to say that Brennan looked calm, poised and in control in every pre-season game I saw him in. Also, he's not a little guy like I thought. He's 6'3"!

I can't say I like the idea behind the change from games played to seasons played. You've gone to a coarser measure. I understand that you want to capture periods of playing well, but couldn't even this be done on a per game basis?

I always wanted the QB projection system to go to an even finer measure of experience with passes attempted.

"I always wanted the QB projection system to go to an even finer measure of experience with passes attempted."

In some ways, wouldn't that also be introducing more variables in addition to more data?

Pass attempts are pretty reliant on the type of offense being run, where seasons/games started are pretty consistent from team to team.

It's possible that playing in an offense that throws a lot should be a positive for QB development, but I think you'd have to check on that first. I think it's possible that starting games correlates better than pass attempts to NFL success.

It's possible, but if as has often been suggested the experience variables are mainly important because they tell us that scouts have had more tape to wok with and are therefore more likely to be right, then it seems like passes attempted would be the best option.

I was bit surprised that you didn't include the actual draft positions of players from the year prior to the draft selection. Was that simply because you wouldn't have any data for that for the 1 year starters? A sophomore QB playing with junior and senior WR/lineman/TE/RB who gets drafted highly played with an assumed quality player. RG III played with Danny Watkins who was taken in the first round in the 2011 draft, but that doesn't factor in as a quality teammate for him. If you are going forward one year, using mock drafts when doing the projections, why not go back one year as well and use actual data like the historical projections had?

Discussing him in the other thread, but him being projected to go in the 2nd by a lot of folks, QBASE liking him and him getting to be developed as a back-up in Green Bay makes me feel like he's got the most hope of any QB take in this draft. (I think Winston is a surefire bust from several different angle and that Mariota is not impressive at all...)