In a perfect world, all baseball fans and self-proclaimed analysts would have access to the proprietary scouting reports on college players their favorite teams were considering selecting in the June MLB Draft. And in this world, each of us would be given the patience necessary to wait the appropriate five-or-so years after a draft to judge the accuracy of different scouts’ pre-draft player reports. But, as evidenced by the fact that Scarlett Johansson is not sitting on your lap right now, this is not a perfect world. So, we’re forced to do the best we can with what we’ve got.

Over the past few years, two developments have brought us to a point where we feel it’s possible to begin to reliably analyze the accuracy of publicly available college scouting reports.

First, is the amount of play-by-play detail now available on college players. We have spent the last two years collecting batted ball data on over 30,000 college players from more than 400 different schools. The level of granularity now largely mirrors that of play-by-play data made publicly available on major league players and teams.

Second, is the growing number of sources providing the types of reports necessary for evaluation. Baseball America, Baseball Prospectus’ Kevin Goldstein, and ESPN/Scouts Inc’s Keith Law (among others) feed a growing band of MLB Draft-watching fans hundreds of thousands of words of scouting content; some straight from their sources in the scouting community, others directly from their own observations while watching players perform. So, there is now a much larger well of reports from which we can draw.

Scouting, of course, is a balance of art and science. Without the benefit of being years removed, it’s difficult to double-check the opinions of a scout’s projections for a player’s future performance. But there are a number of observations included in each write-up, and that’s what we intend to begin analyzing. Because observations, gone unchecked, soon become Conventional Wisdom.

We’ll start by looking at 2007 pre-draft scouting reports on a handful of players who were eligible for last year’s draft, who were selected but chose to return to school for another season.

Player: Kyle Russell

…He has a quick bat and lefthanded power to all fields, and he also offers solid athleticism, speed and arm strength. Yet a lot of scouts aren’t sold on his stroke and approach. They say it’s a grooved swing with too much uppercut, and pitchers can get him out by working up in the zone or coming inside. They also wonder how he’ll handle quality lefthanders.

It’s hard to find a better assessment of a player than “lefthanded power to all fields” for Russell. In 2007, we have batted-ball direction for about half of his 200+ at-bats; 57 of them went to the outfield: 18 to left, 20 to center and 19 to right. When infield outs are included, his balance tilts more to pull, but still, it’s 42 to right (including first base and second base), 26 up the middle and 29 to left. In 82 at-bats so far this season, the bias is a bit more to right field, but still about as even as you can ask for from a young slugger.

We can get even more specific than that. Between last season and this season, we have batted-ball direction for 26 of Russell’s home runs. Of those, 11 went to left, 12 to right, and three to center. Doubles and triples were equally well distributed.

How Russell has handled “quality” left-handers at the college level is a discussion for a different day, but we can take a look at his platoon splits in general. Last year, he posted a respectable 871 OPS against lefties. That doesn’t mean his platoon splits are as even as his spray chart, though: Compare that to the monster 1449 he assembled against right-handers. It appears that he’s far more comfortable against one side than the other.

Player: Mitch Harris

…He throws a hard breaking ball that has some late tilt, but he also has a cutter, and the two pitches tend to morph into one, lacking true shape and depth. He has some feel for a changeup. He does not hold runners on base well…

Holding runners on seems like the kind of skill that must be developed as a pitcher works his way through the ranks, so we have little doubt that something Harris does gives rise to this comment. However, the numbers simply don’t bear it out.

In 2007, Navy batteries allowed 74 steals against 28 caught stealings and 7 pickoffs. That’s not bad for college ball. Harris, however, was in a different league altogether. On his watch, seven would-be basestealers were successful, while six were caught and four more were picked off. That’s a 41 percent success rate against him, while the rest of the Navy staff allowed a 73 percent success rate.

So far this year, we don’t have much to go on: one steal and one pickoff over about 50 batters faced. If anything, that’s something of a good sign. Plenty of batters have reached against him, and they are hardly running wild. Time will tell how Harris fares with professional speedsters, but at the NCAA level, he seems to be quite good at controlling the running game.

Player: Josh Satow

…Satow, just 5-foot-9, 155 pounds, ranked second in the Pacific-10 Conference in ERA (2.23) throwing in the mid-80s with his fastball and baffling hitters with a plus-plus changeup with fade and sink. He keeps everything down, having allowed just three homers in 117 innings…

Keeping home run numbers down is impressive, and that Satow did so (in the warm Arizona air, no less) is a matter of historical record. However, if he is in fact keeping everything down, Pac-10 hitters are doing a heck of a job of getting under the ball. Our database isn’t 100% complete for Satow in 2007 and we only have ground ball/fly ball data for outs, but in the 200 or so batted balls for which we do have information, he shows a ground ball-fly ball ratio of about 0.87.

As Jeff touched on in an article earlier this season, the average Division 1 pitcher gets 61% of his outs from groundballs. To get the proportions on the same scale, Satow’s 2007 numbers indicate that 46% of his outs came from the ground ball. That’s quite low—perhaps even a bit confounding—for someone with enough sink on his changeup to get a scout to take notice.

So far this season, we have data on about 30 innings of work from Satow, and the sink appears (statistically, anyway) to be even less effective. In 2008, less than 40 percent of his outs have come from ground balls. He’s only allowed two home runs, so his success in that department is holding firm, but he has a long way to go before he can use his changeup to earn a reputation as a groundball pitcher.

Player: Dominic de la Osa

At first glance, this holds up. Taking all batted balls into account, de la Osa, a right-handed batter, is spraying the ball all over the place, going to the opposite field nearly as often as he pulls it. However, it would appear that his home run power is almost exclusively to pull. Here’s a look at his batted-ball distribution for last season and this season, compared to his two-year total for extra-base hits:

LF CF RF
2007 45 31 36
2008 20 11 20
XBH 39 10 7

His circuit clouts present an even more dramatic split: Of the 25 in the last two years for which we have directional data, 21 (including all five this year) went to left. There’s no doubting that de la Osa has “plenty of power,” but it would appear that it’s highly concentrated to pull.

Early Returns

One thing that surprised us when sifting through published scouting reports is just how few of them contain statistically testable propositions. There’s nothing wrong with that: In an era when numbers guys can track everything, the scout’s job isn’t to guess what the numbers will say. Instead, the focus is on projection, and as we mentioned at the outset, that’s something we’re not able to evaluate for another few years.

Of the four claims we tested in this article, only one of them really held up. We have no idea whether that’s indicative of a larger pattern, or whether we happened to pick out a few outliers. We do know, however, that with more and more data, the scout’s role is a bit more circumscribed. Perhaps the observer who saw Dominic de la Osa witnessed an impressive batting practice display and could see him spraying doubles into both gaps in pro ball. If de la Osa’s swing is of the sort that will generate power to all fields…well, we can’t argue with that. But for the first time, we are in a position to evaluate many of those claims about how the players are performing now.

In the weeks running up to this year’s MLB Draft, we’ll turn to sifting through scouting reports available for a new batch of college players and test scouts’ observations against the play-by-play data we continue to compile.

Have you run across a scouting report of a college player that could be tested with play-by-play data? If so, we’d love to see it. Drop us a line and let us know.

Kent Bonham, a co-founder of CollegeSplits.com, lives in Washington D.C. He can be reached here.