Everyone's Perfect

Evaluating Risk

the archives are now free.

All Baseball Prospectus Premium and Fantasy articles more than a year old are now free as a thank you to the entire Internet for making our work possible.

Not a subscriber? Get exclusive content like this delivered hot to your inbox every weekday. Click here for more information on Baseball Prospectus subscriptions or use the buttons to the right to subscribe and get instant access to the best baseball content on the web.

How do we arrive at an estimation of what a ballplayer is worth? It’s easy to say that we gather all the historical evidence, project our best estimate of what’s to come, and then sort. But obviously, it’s more complicated than that.

Time for a blind taste test. Here are two pitchers and PECOTA’s 2012 projection for each:

These two are pretty close in projections, and yet, they could hardly be further apart in perceived worth. In fantasy leagues, Pitcher A is a top 20 pitcher heading into this season with an average draft position in the early seventh round. In “real life,” he’d easily command at least $10-15 million per year on the open market and, as a trade commodity, would return a bevy of young prospects. Meanwhile, Pitcher B is barely perceived as a top 75 pitcher in fantasy leagues, goes undrafted in many of them, and in real life, can’t get anything more than a one-year, $4.5 million contract in a pitcher-starved market.

Does the valuation make more sense now that you know the identities? Latos is 24 years old and seemingly has a bright future ahead of him. Bedard will be 33 years old next month and, as everyone knows, is as big an injury risk as they come.

Ah, there’s the word: “risk.” As much as projections play a part in the determination of a player’s value, there are qualitative aspects that help us determine what a player is worth. And high up there in the evaluation process are our fairly abstract notions of risk.

Then again, not all risks are alike. There’s the risk that comes from injuries, yes. There’s also the risk of managerial decisions, influencing playing time opportunities and lineup potential. There’s the risk of slumps and mental weakness. There’s also the risk of things like alcoholism relapses and steroids suspensions.

Are we good at properly evaluating which risks pose greater dangers than other risks?

Perhaps not. Surveys have long shown that humans are often a poor judge of this sort of thing, thinking, for example, that tornadoes are a more likely way to die than asthma, even though the latter causes 20 times as many deaths. Look at our public policy priorities: we spend billions of dollars to guard against aviation terrorism even though it’s more probable that an individual will die in a car crash on the way to the airport and that we might be able to save more lives by investing in safer highways.

Our emotions often interfere with our judgment, and what influences that? Many psychologists point to recent memories as a leading reason for letting emotions interfere with our assessment of risk. If you think about it, this may well make us prone to overestimate the risks of a perpetually-injured player like Erik Bedard given the amount of frustration that often goes hand-in-hand with that owning such a player.

What else can we say on the topic of risk?

For starters, although we tend to think of risk in binary terms—either a player is risky or he is not—risk can come in varying degrees. Even a single player can present different forms of risk to different teams in different leagues.

Take Nelson Cruz. He’s a player who has a long track record of getting nicked up and missing games here and there, and we should expect nothing different in 2012. And yet, in those leagues where teams are allowed to adjust their lineups daily and even pick up free agents at will, Cruz probably presents less risk than in leagues where teams are allowed to make roster moves only once per week.

Indeed, when examining risk, there are many external factors that will sharply impact a player’s risk quotient. Among them: roster flexibility, the number of roster spots per team, the quality and quantity of free agent replacement options, the allowance of DL spots, and various other rules and league structures.

We might say that it was foolish and too risky for the Philadelphia Phillies to give Ryan Howard a $125 million, five-year extension that takes him up to his age-37 season, but it would not be nearly as risky for a fantasy team to make a short-term investment in Howard’s immediate upside given the deep talent at first base. At what he’ll cost, if something goes wrong with Howard, fantasy owners can cut bait and pick up a decent first base replacement. The Phillies wouldn’t have that luxury and will be paying a huge contract no matter what. In other words, risk is all relative.

Further, we should not forget about the various risk management options we have at our disposal. In real life, we drive cars even though we know the risks of a car accident. That’s why we wear seat belts and purchase auto insurance. Likewise, it’s indeed possible to adjust one’s self to the risky players out there. Planning to make future use of the waiver wire, should something go wrong, is one obvious route that needs to be considered when drafting. There are other possible techniques too, such as investing in players who can play multiple positions and fill in for a gimpy star on occasion. Yes, even fantasy baseball has insurance plans.

Lastly, risk goes hand-in-hand with price. The bigger the investment, the more production you’re going to expect because of the other, high-value options you’ll be turning down. Ideally, this means that when we’re drafting, we’re looking for high-upside/low-risk players first and foremost. In reality, though, there will always be some risk involved when making a big investment. The question might be how we both judge those risks and compensate for the things that will inevitably go wrong.

Let’s turn our attention back to Mat Latos and Erik Bedard.

To this point, the fantasy community has collectively judged Latos to be the less risky proposition… but is that really true?

Consider an alternative framing of the issue: Latos is a player with just two good years under his belt. Bedard has had five quality years. Latos has experienced success with just one ballclub in his life. Bedard has pitched well for three different ballclubs already. Latos is moving from perhaps the best pitchers’ park in the majors to one that is friendly for hitters. Bedard is moving from a league that favors hitters to a league, division, and home ballpark that’s receptive to pitchers.

Lastly, as far as injuries goes, there’s no question that Bedard has annoyed his owners in recent years with an inability to pitch a full season. On the other hand, Latos is no stranger to injuries these past couple of seasons, and the workload for this young arm has increased over the past two seasons. He’ll now be pitching for Dusty Baker, a manager who is notorious for overworking young pitchers.

The point is not to make the case that Bedard is primed for a better season than Latos, merely that our construction of the risk issue is often funny and fickle. Both players have similar projections. One is going in the seventh round, the other near the end of a draft. I’ll leave it to you to decide who really represents a greater risk right now.

Eriq Gardner is an author of Baseball Prospectus. Click here to see Eriq's other articles.
You can contact Eriq by clicking here

In the case of Bedard, you also have the risk of not playing due to lineup competition. Latos has a stronger recent pedigree and the Reds traded the farm club to get him. Bedard was a flier on the free agent market.
So how does the PIT pitching staff line up? They have four starters from last season: James McDonald, Charlie Morton, Jeff Karstens, Kevin Correia. Along with Bedard, they just traded for AJ Burnett with two years on his contract. That's 6 starters. And PIT's SPs weren't that bad last season; here's there ERAs: 3.38, 3.83, 4.21, and 4.79. It's not a given that Bedard, despite looking like a lock for the rotation, is even in it. You do have to like that their top quality minor league pitching talent however is probably too raw to take starts away this season.

So, the Depth Charts haven't been re-run with AJ Burnett in the situation yet. Who loses their starting job? That's one of the risky parts of drafting Bedard this season.

This gets close to what I used to do for a living, and I will say that risk is not that abstract. Computational models for determining risk based on event-tree analysis exist, and while they can be time-consuming to set up, they're conceptually simple and do a reasonable job -- if the input data are good. The problem is, they're an extreme Garbage In, Garbage Out kind of thing. Miss the probabilities of the risk-inducing thing happening (injury, loss of effectiveness, getting in a manager's doghouse, etc.) and the resulting assessment is completely bogus. I take the liberty of claiming we know a lot more about quantifying risk than we do about the chances of Dusty Baker or Ozzie Guillen getting down on a guy.

I'm sure the risk assessment community has a good handle on how to handle risk. However, I don't think that's the point of the article. The point is that human beings (not using event-tree analysis) tend to under or over-calculate risk.

Nice article. Bedard is definitely on my radar this year (and I'm concerned about Latos)However, last year was the first time in 3 years that he was healthy enough to throw more than 100 innings, and in 7 seasons he has only managed to pitch over 150 innings twice. So it's not like he just had 1 or 2 years with health issues.

I am quite intrigued by this idea and might have to do some research on "event-tree analysis", but it makes a lot of sense to me that they would have a huge GIGO problem. And I very much agree with your last sentence.

PECOTA quantifies risk. The PECOTA spreadsheet includes 4 probabilities: Breakout, Improve, Collapse, and Attrition. Beadrd's probabilities are 17, 55, 16, and 17, respectively. Latos's probs are 30, 62, 6, and 6, respectively. In other words, Latos is more likely to improve and more likely to significantly improve over his projection and less likely to bomb than Bedard. That's why he is valued more.

The real question is: Why don't the existing valuation algorithms take this risk into account. Auction values (e.g., the SGP method) are based on the mean projection. But a player's actual value to your fantasy team is based on his ability to increase the odds of winning your league. A player's risk profile (as quantified by PECOTA) is a key part of that. That's why I largely ignore the mean projection and focus on the PECOTA probabilities when ranking my players.

It's fun to point out how ('other') people aren't as bright ('as I am').

If I don't have asthma, I personally am at zero risk of dying from it. And if I haven't smoked or have a congenital condition or indulged in some other obvious risk factor, I'm highly unlikely to develop asthma. That's the problem with so many of these cognitive studies showing how 'nonrational' we all are. Often they badly misframe the issue. Often they're 'hothouse artifacts', relying on experimental results which don't and won't translate into real-world behavior.

Player performance risk. There are too many inputs. Despite all the efforts to anticipate injury risk, including Will Carroll here, I have yet to be convinced that anybody can provide meaningful numbers.

You can measure risk against player projections for fantasy purposes because you have draft pick or dollars spent as a benchmark. But you are measuring subjective (yes they are subjective as much as we might think that they aren't, even PECOTA) projections so you're still just taking guesses about one half of the equation.

The concept of risk is a place that's rampant for our biases. It's where we allow perception and narrative to intrude on our evaluation. Most people create a mini-narrative for each player that includes: stats, perceived talent, perceived risk, playing time evaluation. Out of this narrative arises the perception of player. While the qualitative components of that narrative might be important, because they are qualitative they are a place where we need to be most on the lookout for congnitive illusions influencing perception. Overconfidence in our ability to assess risk is a perfect example of that.

My only quibble here is that you call projections "subjective", and I don't think that's correct. Subjective implies (at least to me) that it's more of an opinion, and might have wildly different ways of projecting different players, whereas my understanding of the main projection systems is that they are "objective" in the sense that they have a set of algorithms and all players will be projected using the same set of rules. I think your point, however, is that they are not guarantees, and that any particular projection might turn out to be very wrong, which is of course true.

I only say that they are subjective (and this is particualarly true for projecting playing time) because the choice of which inputs varies from projection system to projection system. The subjectivity is in the design of the algo.