Thursday, March 11, 2010

Doesn't it make financial sense for teams to hire more sabermetricians?

Suppose you have two hitters, each with the same home run rate. The first guy hit mostly long home runs, but the second guy hit quite a few that just barely cleared the fence. For next year, you’d expect that the first guy would hit more HR than the second guy, right?

The study is ongoing following commenters’ suggestions … I’m suggesting a regression to smooth out the categories, so that we can have a single result to predict the amount by which a player will drop.

----

Anyway, Greg's study got me thinking about another of Tango's posts today, one where he links to Sal Baxamusa's summary of a panel at year's MIT Sloan Sports Analytics Conference. One of the panelists was Kevin Kelley, the high-school football coach who became a celebrity when, after studying the issue, he decided never to have his team punt on fourth down. Baxamusa writes,

"Kelley, barely managing to get a word in edgewise, said, "It's not just the method in which it's said, it's who says it."

"Well, fellow statheads, that's just it, isn't it? We can bang the WAR drum all day. We can refine our PITCHf/x studies until we find the one pitch that even Pujols can't hit. We can play all the fancy analytical tricks we want. But when it comes to using these analytics, teams have to do more than just hire a few quants to sit under the stadium and code all day. ...

"John Dewan, of Baseball Info Solutions and pioneer of fielding statistics, said that his organization meets with lots of baseball teams, not just the sabermetric-friendly ones. He was candid in his assessment that some teams that he speaks with don't understand how to effectively use the defensive data that's available. John Abbamondi, the assistant GM of the St. Louis Cardinals, gave an example of a colleague who wanted platoon splits for relief pitchers over a one-week period—data with such small sample sizes so as to be rendered irrelevant as a predictive tool."

The two-part summary seems to be:

-- a lot of teams don't get it, and-- even when the teams get it, sabermetricians are low status and not listened to much.

Which I don't get. I mean, suppose a team had hired Greg Rybarczyk, and, instead of revealing to the whole world what he found, he told the team. Now, when a team is shopping for a free agent, Greg's study should get them a little more accurate estimate of a player's HR potential.

It might not be a lot: even if you find that a 40 HR hitter with lots of "just enough" home runs should drop to (say) 32, you probably suspected that already, because he probably had an unusually good season. Also, many of the JE home runs would, under other circumstances, have been doubles, not outs, so the drop isn't as significant as it looks.

But still. Suppose your estimate was 1 HR better than before. One HR is about 1.5 runs, which is about 1/7 of a win, which is at least $500,000. One good sabermetrician like Greg, who you could probably hire for, at a guess, $70,000 including benefits, would give you, with this one study, information that had the potential to save you $500,000.

So why don't they hire them and at least listen to them? A few possibilities:

1. Just a prejudice against them and their low status. They're young smart guys with no baseball experience, and don't fit into the culture.

2. Even though this one study has the *potential* to save $500,000, it probably won't. There might be one or two guys who fit the extreme-JE profile, and the team may not be signing those two guys this year. Also, there's only a small chance that they'd be outbid by exactly $500K, and the information would make a difference.

3. It's hard for the GM to tell if the stud sabermetrician's results are valid or not. Peer review, before and after publication, makes sure the results make sense. The GM isn't in any position to do that himself (and, indeed, nobody is as good as the community).

4. In the past few years, so much intelligence is out there for free, on the various websites, that the team has enough trouble keeping current with that stuff, never mind creating new knowledge. If Greg comes up with this study on a public site, the team has to run in place just to keep from *losing* the potential $500K to teams that know about it.

5. The team doesn't know who to hire. For every good sabermetrician, there are ten mediocrities that won't help you much.

6. Even if you find a few runs, they're runs you earn *on average*. If you find a guy who's expected to hit 2 more HR, he might wind up having a bad year, and your good move looks like it was a bad move. So, in that light, it's hard for teams to actually see and believe that your study saved them $500K.

To me, none of these reasons seem to be enough. For #5, for instance, you could just ask Tom Tango. I'd hire anyone Tango recommended ... although, I guess I needed to know in the first place that Tango was someone to trust.

Anyway ... for, say, $200,000 a year, you could hire three intelligent, inquisitive sabermetricians, and, among all three, they'd only have to give you ONE RUN A YEAR in extra intelligence to make the hiring worthwhile. Is there something wrong with my logic? Why doesn't every team have two or three Gregs working away?

9 Comments:

It seems that 6 is the big one... As Sal mentioned about Showalter, 2 games was all it took for him to "prove" that the "numbers" were wrong... Unless the stats folks come up with enough little things that some of them work out to make an difference...

I think that 2/3 are it. Considering the error bars in many of our measurements are at least a couple of runs here or there, then *even if the expectation value is -3 runs (or whatever)* the error overwhelms the measurement.

This has two effects. The first effect, a real effect, is that it is hard to use this information to come out ahead in the longterm since you only get to execute the decision one or two times. Unless something is a clear win, it's too easy for a GM to hear from a scout who can argue against the small effect of -3 runs.

The second is that it's too easy for a sabermetrician to get burned on the prediction and lose credibility. On the other hand, amateur scouts can have a hit rate of >1%, but finding one Pujols generates a *huge* net return. I don't know if it's fair, but I think that's the reality.

$70K seems very low, especially including benefits. You could probably hire a recent college grad for that ($40K salary + benefits, office space, etc.=$70K?), but I doubt someone like Greg would feel that was a fair wage for him. (His personal situation may mean he accepts it just to work for a team, though- but in that case he may be happier giving the info away.)

Fair enough. It seems like the team would have to understand the idea that, even with a lot of random variation, an edge of 3 runs is indeed worth a couple of hundred thousand dollars, if you can take advantage of it.

It's like betting ... if I'm betting $1000 a game on baseball, and you can give me an estimate that's 0.1% better ... well, it'll take a LOT of bets until I can confirm that you're right by observing the results. But, if you can convince me by other means, I'm going to earn an extra $10 on the bets where your info made a difference, and that's worth quite a bit.

But teams have to think of making moves as trying to get a percentage edge, rather than a move being 100% "right" or "wrong".

I would say mostly #4. To take it further, in the long run there is a zero-sum-game issue to address. If everyone hired 3 "quality" sabremetricians, then no one has an advantage. Now the owners are paying $200k just to tread water. Sort of how most energy companies now have 5+ meteorologists on staff. Perhaps the owners are disinclined to start this arms race, particularly if your reason #4 is true and the low and middle hanging fruit is there for free.

Thanks, this post was really useful. I've thought a lot about how quantitative statistical analysis can be applied at the management level across all sports.

Another big question is - how many decisions are made per week/month/year where this kind of information is actually useful? How many trades/signings does a team do per season, and in how many of those situations would statistical analysis beyond simple fundamentals actually make a difference?

The advanced stats, as in field, are only of value to the consumer. Given the consumers may typically hold the lowest level of statistical "comfort", SABR stats requires a tremendous leap of faith to embrace consistently.

Give the perception of the messenger as well as the message itself being viewed with varying degrees of suspicion <-> contempt by the consumer, I suspect we may have reached a plateau of acceptance; especially with tools such as WAR.

Perhaps, that plateau is best addressed by better quality data vs more statisticians.