Expanded Horizons

Is It Next Year Yet?

the archives are now free.

All Baseball Prospectus Premium and Fantasy articles more than a year old are now free as a thank you to the entire Internet for making our work possible.

Not a subscriber? Get exclusive content like this delivered hot to your inbox every weekday. Click here for more information on Baseball Prospectus subscriptions or use the buttons to the right to subscribe and get instant access to the best baseball content on the web.

When you think about Baltimore cuisine, you probably think about seafood, and more particularly crab cakes. The Chesapeake offers a rich harvest of the ocean blue, but it isn’t all high quality. For example, a common cheap eat in Baltimore is known as lake trout, which has three surprising characteristics. One, it isn’t actually lake trout (although that is a real fish). Two, it isn’t actually from a lake (it’s from the Chesapeake). Three, it isn’t actually trout (it’s Atlantic whiting). For the uninitiated, the surprise of being served two filets of whiting and some white bread might be rather severe. That being said, if you don’t have much money and you’re not really used to better, get yourself some lake trout. Trouble is, lake trout isn’t very healthy, since most of the taste comes from the deep fat fryer, and a steady diet of it would get monotonous—it is, after all, a mostly bland white fish. Every once in a while, you’re going to want some steamed crabs.

The Orioles are bad again this year. When is the last time you couldn’t have said that about the team? That is, when was it that not only were the Orioles good (or at least not bad), but that they had created reasonable expectations that they would be good? Here are the Orioles’ records for each season of the last decade:

The last time the Orioles had a winning record was 1997, when they were 98-64. Do you know how long ago 1997 was, in baseball terms? Put it this way: the only contributor to that team who is still playing major-league baseball is Arthur Rhodes, a fact that is revealing particularly because it demands some serious respect for what Rhodes is currently doing for the Reds at age 40. The last time you couldn’t really say “the Orioles are bad again this year” was during the Clinton administration.

It has not really been a decade of soul-crushing, monochromatic gruel (you know, the kind intentionally low on protein to keep you docile); that’s the diet Pirates fans have been on. Rather, it has been an awful lot of lake trout. There have been moments where Orioles baseball kind of hit the spot—at least while it lasted. There was 2005, when the team darted out to a 42-28 record on the strength of its .285/.342/.481 batting line—oh, Chris Gomez, how we never tired of your BABIP shenanigans. The year before that, the team actually outscored its opponents 842-830 and finished in third place, despite the fact that Sidney Ponson yielded an astonishing 136 runs.

That it was third place in 2004 with 78 wins demonstrates the real problem. Lake trout is nobody’s first choice of seafood. If you had a little more money, or were in some other town, you’d probably opt for something else. But that’s because other seafood can be absolutely delicious. The hegemony of teams atop the American League East is a ceiling impervious to the Orioles' charms. That they stood by while the Rays darted past them puts a particularly bitter spin on the state of affairs, but it doesn’t change the basic calculus. Since the ascent of the Red Sox and Yankees in the late '90s, the Orioles have been forced to eat fried Atlantic whiting from corner stores that insist on calling it something other than what it is.

But Hey, At Least They Get Some White Bread, Too

It was supposed to begin to turn around. Sure, it took a little wishcasting and some magic beans, but it wasn’t too hard to see a 2012 Orioles roster that featured front-line starters Brian Matusz, Chris Tillman, and Jake Arrieta to complement a middle of the order featuring hitters Matt Wieters, Adam Jones, Nick Markakis, and Josh Bell. Pretty soon, the Orioles were going to buy themselves a nice condo on the Inner Harbor and it would be bye-bye lake trout, hello crab imperial.

That plan has not worked out. Before their current four-game win streak, the Orioles had a winning percentage below .300. According to our postseason odds, the Orioles wouldn’t make the playoffs this year even if they had 100,000 shots at doing so. Even as it stands, they are on pace for a 50-112 record, which would be by far their worst season since the truly miserable 1939 St. Louis Browns went 43-111. Matusz has an ERA close to 5.00 and a SIERA that isn’t that much better—thanks, bullpen! Tillman, after finally getting called up, was torched in four starts and has a 5.91 SIERA. Arrieta? He's at 5.69. The hitters aren’t faring much better. Wieters (and his .233 TAv) has as of yet been neither deadly nor accurate. It took Jones a June on fire to reach a .255 TAv, and he still isn’t OBPing over .300. Bell is hitting a disappointing .263/.307/.438 with a 24 percent strikeout rate in 296 Triple-A Norfolk plate appearances. Only Markakis and his .295 TAv have stood above, and even his power ceiling is much lower than it once was: his .427 slugging percentage is the lowest of his five major league seasons.

Of course, a great deal of it has to do with their rough schedule. They’ve gone 8-25 against teams from the AL East, but their 15-27 record against all other opponents is considerably better. They’ve also gotten a little unlucky this year, and reading from left to right on the BP Adjusted Standings report is like washing away a little bit of awful from the 2010 Orioles. Their third-order record is 29-46, which translates to a .387 winning percentage. The Postseason Odds report projects the Orioles to win 57 games, which is some small consolation to the dignity of George McQuinn and the '39 Browns.

Things get dramatically better (on the order of six wins) when we do this because the Orioles have faced better hitters than any other team and have faced better pitchers than any other team. “Yeah, yeah,” you’re thinking, “we all know the AL East is good. But is it really that good?” I’m glad you ask, because I like to use numbers. The hitters the Orioles’ have faced have a combined .264 TAv, which is the highest mark in baseball. The pitchers they have faced have allowed a combined .254 TAv against, which is the lowest in baseball.

Doesn’t it just strike you as unfortunate that the same team that has been miserable for a decade would face not only the best hitting so far but also the best pitching? It makes sense, since they are the only bad team in the AL East, meaning they are the only team that doesn’t get to play themselves. And the AL East has played a disproportionate number of interleague games against the also-strong NL East. But, really, lake trout again?

Eh, I kind of disagree. PECOTA definitely had its fair share of misses this year to be sure (Wieters, Sizemore, McLouth, etc.), but it also had more than it's fair share of hits that no one else saw coming. Colby Lewis, anyone?

The problem isn't PECOTA, the problem is the inherent unpredictability of player projections. Every system, no matter how accurate, is going to have quite a few big misses.

Actually, if you look at any of the articles comparing the projection systems (on other sites, Silver used to do them himself, when PECOTA was the most accurate) you would see that PECOTA is now one of the least accurate systems.

PECOTA not only is less accurate compared to other systems now, it even has a lower accuracy than previous years. Considering that this change accompanied the loss of Silver from BP, this seems like it is NOT a coincidence.

Feel free to disagree, but you could look into it instead of just following along like the rest of the unquestioning masses while I look for better content elsewhere.

Other readers have rated this comment below the viewing threshold. Click here to view anyway.

"Feel free to disagree, but you could look into it instead of just following along like the rest of the unquestioning masses"

Obviously, you decided not to look it into it and just assumed that your uneducated opinion is correct. Typical idiot behavior and another sad decline, as this used to be a community of thinking people.

The claim that PECOTA was not significantly more accurate than the '08 numbers is based off the Standing Gains Points analysis. What I want to emphasize here is that looking at SGP looks at two elements of forecasting simultaneously - projecting a player's talent and projecting a player's usage.

If you look at the evaluation for OPS listed there, you find that PECOTA was, in fact, more predictive than the '08 totals. Converting the RMSEs listed to a more intuitive scale, using some numbers provided by Jared in that Book Blog thread you list:

What this means is that, for instance, 68 percent of CHONE forecasts tested were within .079 points of the observed. What you tend to see is that there is little differentiation between forecasting systems at projecting OPS. They're all better at predicting results than single-season stats, but there's not much room for differentiation.

[I do want to register a methodological complaint here, while I'm at it. The results you see are RMSE between projected OPS and observed OPS. The average OPS of players in sample was 0.769; the average OPS projected by PECOTA was 0.763, compared to 0.770 for ZiPS and CHONE. But we don't care about a player's OPS forecast in the abstract, but his OPS relative to other players. If you correct the forecasts to have the same average OPS as the observed, you should see the spread of forecast accuracy tighten up further.]

So, if PECOTA is predicting the rate of performance roughly as well as the other forecasting systems, why did it show up as doing so poorly in the SGP analysis? Because PECOTA did a terrible job "projecting" playing time. Of course, the values in the weighted mean are a projection of how much a player could play at all levels, including the minors.

If you're looking for projections of playing time, we provide those in the form of the depth chart values, available through the Player Forecast Manager. (The PFM values are also available for download in CSV format.) But the Steamer evaluation you linked did not use the depth chart playing time, it used the weighted mean playing time.

So what that result is really telling us is that PECOTA is substantially worse at projecting playing time than other systems, if you ignore our playing time forecasts altogether. I don't know - is that a meaningful result? I don't think it is. (If you'll note, the Steamer forecasts DID use the PECOTAdepth chart results, and those PA forecasts grade out as among the best, substantially better than CHONE or ZiPS.)

Other readers have rated this comment below the viewing threshold. Click here to view anyway.

I don't feel obligated to tone down anything. My initial comment was just being amused at the playful joke that Tommy Bennett made about Matt Wieters failing to come anywhere near his forecast, which everyone has agreed was terrible in retrospect.

My comment was voted below the viewing threshold for criticizing BP despite the fact it originated from a BP author, which continues a trend of other subscribers automatically supporting the BP brand without thinking. I find that incredibly depressing, and I'm not going to play nice just because people want to assume that everything is great because they'd rather not look into it themselves.

While your take on PECOTA doesn't show as drastic of a difference between the systems, it still shows that PECOTA performed the worst out of the systems. Some criticism of PECOTA is warranted and hopefully that would lead to improvement.

Overall, I'd say that the criticisms of PECOTA, the Baseball Prospectus brand, and other comments is more than warranted.

The original objector was simply saying that while PECOTA has had a miss or two, it has also had a few hits that are worth pointing out, and he rightly makes the point that all forecasts have errors. You're the one that ratcheted up the vitriol by lumping him in with the "unquestioning masses."

Actually, the first commenter failed to recognize that the projections in question were 2009, not 2010. His only counterpoint was Colby Lewis, which is a victory for Clay Davenport's work with adjusted league difficulties. I'd say that's a victory for Clay rather than PECOTA.*

He assumed that PECOTA was just fine regardless of how many misses or by how much, instead of questioning it, so I'd say he's clearly part of the "unquestioning masses."

*Clay has taken over PECOTA from what I understand, so it might seem odd that I'm basically criticizing him and praising him at the same time. I guess that's true but I just think he does some things better than others.

All of the examples I used were based on the 2010 projections, simply because they would be more familiar within the current context than than the hits and misses of a year ago. If anything, my mistake was assuming someone of your vast intellectual prowess could recognize that.

In any event, I'm done with this one. Frankly, you are just wasting your time here and the time of everyone else who actually enjoys reading the site.

Thank you, that was exactly what I was saying. I never said PECOTA was the end-all, be-all of baseball knowledge, and as I mentioned then I'm more than aware of the inherent unpredictability at play here.

And frankly, again, I'll reiterate my point about if it's that bad here, why even read? Even if the subscription itself is free, it's still a massive waste of your time on something that is apparently worthless; time which you could otherwise be using to read ever-more profound baseball analysis.

One final point: if indeed you are receiving the subscription for free and not paying like the rest of us, that was something done by one of the authors here as a good-faith gesture in an attempt to win you back over. If you still don't like the final product, fair enough, don't read the site. However, if someone made a good-faith gesture to you to give you six months of a free subscription, the least you could do would be not publicly bash the authors and their work product on the site itself. That's not you having objections to the work product as much as you just having the need to be an ass in a public forum.

Would you honestly write this post if you were not an employee of BP? I should hope not. (Some other Colin Wyers apparently wrote a more objective critique here: http://www.hardballtimes.com/main/article/the-death-of-superman/).

There was a time not too terribly long ago when the motto of PECOTA was more ambitious than what sounds a whole lot like: "not that much worse than the average projection system." Any impartial observer reviewing the data would quickly realize that PECOTA was a relative disaster in 2009; and yes, I am talking about rate stats, not counting stats. I am also talking about standings predictions: http://vegaswatch.net/2009/11/evaluating-april-mlb-predictions-2005.html.

The additional problem with the "not much worse than the others" argument is that the others are far simpler systems and are freely available. I.e., the systems' authors would readily admit there is not a ton of scientific rigor behind the algorithms used. These systems should be used by PECOTA as the benchmarks to beat -- rather than shooting only to beat using last year's stats.

Nate Silver used to pride himself on beating other projections. I sense none of this drive to outperform in this post or any other by a BP author with regard to PECOTA over the past year. After the pre-season debacle this season, one would think there would be an in-depth article (or ten) on the steps BP was taking to improve PECOTA. You know, cool research articles on possible ways to improve forecast accuracy -- the former bread 'n' butter of BP. One would think BP would be putting its top minds on the project *now* -- not next Jan or Feb.

As a final note, how can you praise the PECOTAdepth chart playing time forecasts, when among the four systems designed to project playing time, it finished third?

"You know, cool research articles on possible ways to improve forecast accuracy -- the former bread 'n' butter of BP. One would think BP would be putting its top minds on the project *now* -- not next Jan or Feb."

Um, I thought that was what Eric and Matt spent a series of articles on... namely, introducing/explaining/using SIERA. Nor have those been the only ones. And they did it last January/February as I recall...

The main beef people have about PECOTA is it is proprietary and a bit black box, so it's hard to critique it or know why it does/doesn't work.

Then again, I don't subscribe to BP solely, or even primarily, for PECOTA.

Do you remember what the relative differences were between the better and the worse projection systems? Something like a perfect or two, taken as a whole.

PECOTA ain't perfect. That's why you should use several of them. But if you're not using PECOTA, you're missing out. It informs its projections with player comps using things like body size, injury history, etc. That certainly has value.

No, it doesn't always nail down Ichiro or Javy Vasquez. But other systems have similar shortcomings as well. Multiple tools are almost always better.

Actually, I haven't paid for BP in over 2 years. I got a 2 year subscription while Sheehan and Silver were still writing. It since ran out and I shot an email to Christina Kahrl to thank her for her contributions, but saying that I wasn't interested anymore overall. I was given a free 6 months to give them the opportunity to win me back.

I promised to give them that opportunity, which is why I'm still reading these articles.

Agreed, the website in my mind is all about the written content. My personal preference was Silver and Sheehan, but it's fine by me if people stick around if they like the current cast of authors.

My real complaint about PECOTA is more related to the annual, which is no longer as reliable a reference guide because the player comments no longer go out on a limb and the projections are less accurate.

I actually thought that the decline in PECOTA was somewhat public knowledge. It seems like many people still believe it's working at the high levels it posted years ago. There's a reason why all of the favorable quotes on the cover of the annual are 5 or more years old.

It's so tough being an O's fan these days. On one hand, you think you have reason for optimism. They finally put a semi-comatose baseball guy in charge. They drafted three quick-risers in Matusz, Wieters, Machado. They pulled off an all-time trade in the Bedard deal. They avoided handing out a crippling contract (though they tried, with Carlos Lee). And yet none of it matters.

I will say this... watching the futile rebuilding efforts in Baltimore makes me appreciate even more the freakishly awesome job Tampa has done. And, in turn, that makes me profoundly depressed. The fact that a team has to absolutely nail it on every front--mdrafting, development, free agency, trades, management--just to compete with the Yankees and Red Sox... I may just stop cheering for a team and start rooting for realignment.

I really think fans need to start pushing for the balanced schedule, not realignment. Much easier sell than full blown realignment. Taking the O's from 18 to 11 or 12 games each against the Red Sox, Yankees, and Rays would do wonders for their outlook.

I think it's becoming clearer that the Orioles have been doing some obvious things right, but have a serious problem with the foundations of the organization. They have drafted better, they've paid bonuses to guys like Wieters, they've avoided silly contracts, they've brought in young talent like Jones and Bell and Pie.

But deep down there are some fundamental problems. The scouting is fatally flawed - players don't seem to adjust their approaches to the opponent, they pretty clearly recommend signing players who're done (Atkins, Lugo). The coaching and development system has long had a problem with the jump from the minors to the majors. And their international "efforts" barely exist.

I go back to a Bill James' quote that MLB teams do tons of little things to improve all the time, and that he can see from the inside that the Red Sox are clearly better in numerous ways than even 5 years ago. The O's seem to be getting better 5% a year, while the competition moves up 10%, 15%, 20% a year.

It hasn't worked out because there aren't too many teams who've ever been in contention in a strong division two years after a 55- or 60-win season that left them with big holes at 3, 4, 5 positions and little left in the minors. The plan MacPhail outlined involved being around .500 in 2010 and moving towards contention in 2011. Now 2012 seems really optimistic. MacPhail's plan involved the young core stepping forward starting in 2010, not uniformly moving in reverse.