/m/sabermetrics

Reader Comments and Retorts

Statements posted here are those of our readers and do not represent the BaseballThinkFactory. Names are provided by the poster and are not verified. We ask that posters follow our submission policy. Please report any inappropriate comments.

If teams aren't incorporating scout evaluations into their statistical models then ... well, their statisticians are asking the wrong questions and/or don't know what they're doing.

Out here we don't really have the luxury although somebody out there should be adding in Sickels (or somebody's) evaluations where available.

The quality and utility of scout evaluations (relative to the numbers) are statistically testable.

The issue has not a lot to do with which sort of information is superior but which sort of information is available. The numbers for 18-year-olds are not widely available and pretty near meaningless except maybe in fairly extreme cases (very small sample sizes, unknown quality of competition, bodies still growing). For high school, you probably have little choice but to rely on scouting reports. I'm not aware of any serious saber types claiming high school draft decisions should be based on numbers, much less signing a 16-year-old Dominican.

The numbers of a potential college draftee presumably have some, at least short-term, predictive value because we have (or should) a reasonable idea of how that context compares to, say, low A.

Once you've got a reasonably large set of numbers, I doubt a scout evaluation does much good other than possibly providing some useful information as to who might be better/worse than the numbers look. But you usually don't have a reasonable set of numbers until they're ML ready (or not).

I'll add however that it's distinctly possible that observations of defense beat measures of defense for the minors.

I'll add however that it's distinctly possible that observations of defense beat measures of defense for the minors.

I absolutely don't doubt this, even in regards to the majors. Defensive metrics at the major league level do not take into account positioning in the slightest, it's impossible for them to be accurate as a rating tool on quality of an individual level of defense in comparison to a well designed scouting level of defense. Having said that, that doesn't mean scouting is currently better, it just means that a properly designed scouting report on defense should always beat the numbers.

distinctly possible that observations of defense beat measures of defense for the minors

Not least because the sample size of game performances in the minors is tiny. A guy plays 40 games in one level and 40 in the next as his bat carries him upwards, and who knows what he's really capable of in the field, not to mention he's nervous as heck and trying to learn what a bunch of different coaches want from him.

It occurs to me (along the lines of fanboy's remark, though I'm much more sanguine about defensive metrics than he is) that the best ML metrics are a form of higher, quantified scouting. They attempt to organize observations into a coherent framework for comparison. That's the ideal, I reckon.

Defensive metrics at the major league level do not take into account positioning in the slightest, it's impossible for them to be accurate as a rating tool on quality of an individual level of defense in comparison to a well designed scouting level of defense.

But this well-designed scouting would presumably note the positioning of the player prior to the pitch, thereby allowing for that variable to be accounted for in the defensive metrics.

Scouts measure stuff whether they put a number on it or just write that he has "good makeup." Even the subjective stuff can be coded even if it's only coded as a set of 0/1 variables. Those measures then go right into the model. There is obvious measurement error involved in the coding but then there's obvious measurement error involved in somebody reading a scout's report and interpreting what they wrote.

They started putting those 0-80 ratings on reports because managers wanted a numerical scale by which to summarize the scout's report and compare players. The hard part has already been done, plug those into the model and see if they tell you anything above the numbers and at what point you should start believing the numbers more than the scouts' rating of the potential.