Tuesday, June 10, 2008

Draft Review - About Process

Now that we've had a few days to recoup (and since we've secured a few dramatic W's), I thought it would be a good time to do a quick review of our draft. However, first I'd like to write about the distinction between process and outcome, because this distinction permeates everything we do in baseball ops, never more so than in the draft.

Geoff Young from Ducksnorts wrote a comment after the draft saying that the tough part of the draft was that we won't know the outcome for five years or so. I couldn't agree more, so what I'm going to focus on is the process.

Many years ago I was playing blackjack in Las Vegas on a Saturday night in a packed casino. I was sitting at third base, and the player who was at first base was playing horribly. He was definitely taking advantage of the free drinks, and it seemed as though every twenty minutes he was dipping into his pocket for more cash.

On one particular hand the player was dealt 17 with his first two cards. The dealer was set to deal the next set of cards and passed right over the player until he stopped her, saying: "Dealer, I want a hit!" She paused, almost feeling sorry for him, and said, "Sir, are you sure?" He said yes, and the dealer dealt the card. Sure enough, it was a four.

The place went crazy, high fives all around, everybody hootin' and hollerin', and you know what the dealer said? The dealer looked at the player, and with total sincerity, said: "Nice hit."

I thought, "Nice hit? Maybe it was a nice hit for the casino, but it was a terrible hit for the player! The decision isn't justified just because it worked."

Well, I spent the rest of that weekend wandering around the casino, largely because I had lost all of my money playing blackjack, thinking about all of these different games and how they work. The fact of the matter is that all casino games have a winning process - the odds are stacked in the favor of the house. That doesn't mean they win every single hand or every roll of the dice, but they do win more often than not. Don't misunderstand me - the casino is absolutely concerned about outcomes. However, their approach to securing a good outcome is a laser like focus on process...right down to the ruthless pit boss.

We can view baseball through the same lens. Baseball is certainly an outcome-driven business, as we get charged with a W or an L 162 times a year (or 163 times every once in a while). Furthermore, we know we cannot possibly win every single time. In fact, winning just 60% of the time is a great season, a percentage that far exceeds house odds in most games. Like a casino, it appears as though baseball is all about outcomes, but just think about all of the processes that are in play during the course of just one game or even just one at-bat.

In having this discussion years ago with Michael Mauboussin, who wrote "More Than You Know" (a great book - a link to Michael's strategy papers appears on my blogroll), he showed me a very simple matrix by Russo and Schoemaker in "Winning Decisions" that explains this concept:

We all want to be in the upper left box - deserved success resulting from a good process. This is generally where the casino lives. I'd like to think that this is where the Oakland A's and San Diego Padres have been during the regular seasons. The box in the upper right, however, is the tough reality we all face in industries that are dominated by uncertainty. A good process can lead to a bad outcome in the real world. In fact, it happens all the time. This is what happened to the casino when a player hit on 17 and won. I'd like to think this is what happened to the A's and Padres during the post-seasons. :-)

As tough as a good process/bad outcome combination is, nothing compares to the bottom left: bad process/good outcome. This is the wolf in sheep's clothing that allows for one-time success but almost always cripples any chance of sustained success - the player hitting on 17 and getting a four. Here's the rub: it's incredibly difficult to look in the mirror after a victory, any victory, and admit that you were lucky. If you fail to make that admission, however, the bad process will continue and the good outcome that occurred once will elude you in the future. Quite frankly, this is one of the things that makes Billy Beane as good as he is. He is quick to notice good luck embedded in a good outcome, and he refuses to pat himself on the back for it.

At the Padres, we want to win every game we play at every level and we want to be right on every single player decision we make. We know it's not going to happen, because there is too much uncertainty... too much we cannot control. That said, we can control the process.

Championship teams will occasionally have a bad process and a good outcome. Championship organizations, however, reside exclusively in the upper half of the matrix. Some years it may be on the right-hand side, most years should be on the left. The upper left is where the Atlanta Braves lived for 14 years - possibly the most under appreciated accomplishment by a professional sports organization in our lifetimes. In short, we want to be a Championship organization that results in many Championship teams.

I'll touch on our draft in greater detail in the next day or so, but I will say that we are proud of our process and it was carried out with discipline. Will it lead to a good outcome? We don't know for sure, but we have confidence in the group of picks that were made. I do know, however, that our process gets better every single year, and we expect it to be better again next year.

20 comments:

The casino is glad the guy hit on 17 and won, because that player will stick around and play longer, bolstered with confidence by their great decision, and will then hit on 17 several more times and lose nearly every time.

How long does it take before you figure whether your process actually works, or you've actually been sitting in the bottom category the entire time thinking you were in the top category? The Braves were highly successful for 14 years, but they only won 1 championship, and towards the end of their run they weren't able to sell out playoff games.

Great post. The first place I ever read about the process/outcome distinction was when I read Harvey Dorfman's book, as a lousy high school outfielder 18 years ago. Luckily for me, the approach works just as well in academics, business, and the law.

My question: Does Dorfman deserve any of the credit for Oakland's institutional approach?

This is something that is an absolute truth not just in baseball or casino's but everywhere in life. I feel that Bad Process/Good Outcome is the downfall of the majority of baseball players at a young age (myself included). So many players do not worry about proper mechanics in either hitting or pitching at a young age because they would rather just go with what works now. I can recall countless young hitters that ignore hitches in their swing because in batting practice they can hit a ball a few feet further. Then come game time they end up swinging through pitches.

I think that's one of the thing I admired most about (uh oh, controversy alert) Barry Bonds. The guy didn't focus too much on getting certain results, he just focused on making sure his process was right. I've seen him visibly upset after swings that resulted in doubles and pleased with himself on foul tips. The key is knowing that if you stick with the good process long enough, it will end more often then not with good results, which is exactly why Casino's make billions.

Question for you about drafting mindset. I know teams say they don't draft for need, and I know that means they're not drafting for today's major league needs as so much can change. And teams often say they're just drafting the best player on the board. However, you do have a couple short season rosters that you have to fill. What if it worked out that out of your 40-something picks, you ended up with no catchers? Or not enough pitching? Or 15 shortstops? Does there come a point later in the draft where you're looking at your rookie league team roster and think, "we don't have a catcher" and then draft some kid in the 42nd round because he's a catcher? And maybe otherwise, that kid would never have gotten drafted? How much are the later rounds used to really fill out the short season rosters?

PS. my poster name (what's he thinking), has nothing to do with this blog, it was the name of my own.

Agree, about focusing on process over outcome, its what every successful business does. ...It's under the same life umbrella that should have you focus on what you can control, rather then things that are out of your control... I also want to take this opportunity to thank you for including us, your blog readers, in at least a small way, in what you go through during the draft. I found the experience very satisfying, almost as if I was involved in the draft. I hope that this draft bear fruits at the ML level. I might boast one day, when my memory starts to go, to have been a part of it...

At this point we have agreements in place with close to 20 of our draftees. However, these are not official signings until docs are signed and physicals passed. I won't be leaking these, but I will report on them when they become official.

There is definitely a point at which we must focus on positions guys play as the vast majority of players will be playing on one of two rosters - Eugene or Peoria. We can't sign 12 outfielders, for instance or only 6 pitchers. We have a number of player in the system already who are also destined for one of those two rosters. That number (in-house players) changes every year, so when we start drafting "for need" changes every year as well. Nevertheless, we're still taking players that we believe are the best available in the particular positions.

nathan had a great question, so though it's all the way at the top, I'd like to respond to it.

How do we know when a process works? The fact is that we are constantly stress-testing our processes, which helps to refine our thinking. There are times when it makes sense to make a small change to a current process, and there are times when it makes more sense to scrap the current one and start anew. Typically we re-evaulate virtually everything on an annual basis as we gather new knowledge (often in the form of new data). The cycle of the baseball calendar makes an annual review worthwhile.

Slater's right about Bonds, which hopefully won't be controversial in a forum like this; he's an excellent example of taking ferocious talent and wedding it to a rigorous process. Dude wasn't a machine because of magic pills; he was a machine because he made you throw his pitch.

Maybe I'm jumping the gun a bit here, but I'd like to hear more about what makes a process good or bad.

Aside from a smattering of teams in MLB that seem to be either consistently successful or unsucessful, most teams seem to drift through a fairly even number of years of success and failure. I imagine this complicates matters when it comes to determining if your process is working or not.

Most people know that you should double down on eleven and stand when the dealer's showing six, but, obviously, running a baseball team is not such a straightforward proposition.

Based on your post (which I agree with), we know that the outcome doesn't always tell us whether the process used was good or bad. So, what's the best way to determine if you're using a good or bad process?

I assume it'd be the aggregate outcome of several years, but that seems like it's a long time to stick with what may be a bad process before knowing that it's a bad process. So, is there another way of evaluating the process without relying solely on the result?

I can't get into all the details, as I'd be entering a proprietary area. However, I will say that the evaluation of a process can depend on the type of process. Also, if one creates a process that they believe is worthy of implementation, there ought to be some evidence of its efficacy. The process can then be judged against that evidence.

For an example, let's go back to the casino. A casino has the odds of what should happen in a particular game, and they can track what actually happens in that game. If there's a discrepancy, they will try to figure out why (or drag the offender into a back room with no windows).

You are correct that baseball isn't quite as straight forward - baseball is a complex system that creates a lot of noise. That being said, there are signals buried in that noise that can inform us about our various processes.

Lastly, one of the things we ask ourselves when implementing a process is whether or not it's measurable. If the answer is no, we'll often try to make some changes in the implementation in order to get a "yes" answer.

I've been both the guy on first who has had perhaps one too many and hit for the fun of it, naturally to the dismay of the subsequent players. My only rationale for doing so as I had to explain to a particularly upset 3rd base was that I came to the Casino (fist time) looking to have fun for the night and lose all the money I walked in with. If I was to win anything at all great, but I had no delusions of "winning" at gambling. Anyway I realized that my fun was not shared by anyone else so I just went a sat at the Roulette table and had a blast for hours betting on whatever whimsy I had. (I did in fact walkout $50 richer)

Excuse the rambling. My question is how is it that MLB teams at this point can have such fundamentally different processes? Certainly there will always be more than one way to skin a cat but what/who drives the process? I would think all teams realize their players need to cross home plate more than they allow the other team to at least 50% of the games they play to be considered somewhat successful. How they accomplish this is up the the teams philosophy i.e power as opposed to contact hitters or an emphasis on veterans as opposed to younger players to suggest a few. So when you draft the best players available operating under a certain philosophy and a few years down the road do the ones who are major league ready alter that philosophy because of their particular skill set?

As a Mariner fan I always smiled at the mention of "pitching and defense" during the Sweet Lou days. And now our AAAA Mariners claim to be about pitching and contact (hilarious I know). But the majority of players in the farm system drafted under one philosophy are going to be there to implement that of the new regime.

Ok I see my problem here. I'm trying to take a great post about processes and critical thought and make sense of M's pitiful F.O.

Fantastic post. I love the fact that you make the time to explain your thoughts on how these type of things work. Most of us fans don't have the privilege of working in baseball and your thoughts are quite insightful.

This might be the first time I could refer a sports blogger to Alfred North Whitehead's Process And Reality! You and Whitehead are on the same page.

Rather than Bad Breaks, the upper-right box might better be called 'Breaks of the Game', since you WILL eventually have hit a four on 17--it's not perjorative...

The NYGiants, to me, were a team who had just started to exercise better process and got rewarded by some breaks at the same time...certainly Reese's draft, Spagnuolo's hire and coordinator, and Coughlin's 180 on communication were all positive changes that were reflected in the result....