Carlos Marmol blew a save last night so it naturally sparked discussion about whether or not you should sign relievers to longterm contracts. Carlos Marmol is a very good relief pitcher and likely will be worth the contract he signed this past offseason. However, we know relief pitchers work so few innings that their sample size in a single season isn’t nearly large enough to really know what you’re going to get. We have an expectation, but consider for a moment that a starter with 200 innings will more than likely be within 1 run per 9 innings of what we expected entering the season. Most elite relievers work 65 to 70 innings. In Marmol’s 3 seasons, he’ll barely have pitched the equivalent of one full season as a starter. So the range in a relievers expected production is huge. As such, it’s always been my opinion that there are very few elite relievers worth signing to longterm contracts.

But just how unreliable are they from season to season? We’re only halfway through 2011, but for what I want to do that’s good enough. I want to make clear that no conclusions can be reached based on what I’m doing. I’d have to use many back to back year pairings and they’d have to be full seasons, but I’m not really interested in the exact amoung of variance an elite reliever has. Somebody else can or already has done that work.

I didn’t want to use just one pitching statistic so I chose to use Run Average (runs per 9 innings), FIP and Base Runs per 9 innings pitched. Run Average tells us exactly what happened. It tells us how many runs were scored against the pitcher without concern for luck or defense. BaseRuns per 9 innings (BsR9) removes the sequencing. There’s little to no skill difference between pitchers when it comes to pitching with runners on base, but in small samples some players will give up more hits with men on 2nd and 3rd than another pitcher. BsR9 corrects this. FIP tells us what the pitcher controlled by looking only at the events he’s in complete control of: walk (and hit by pitch), strikeouts and home runs. FIP ignores sequencing (when the events occur) and batted balls. Before we call this a weakness of FIP, keep in mind that OBP also ignores batted balls and sequencing. FIP also ignores defense. xFIP, which I’m not using, removes luck from home runs per fly balls since pitchers are at about 10% over large samples. There’s very little skill in allowing home runs as a pitcher.

Each statistic is important when evaluating a pitcher. In the real world, luck matters. The defense behind you matters. How you pitch with runners on base matters. Over large samples those tend to correct themselves, but in a single season we’re often interested in how the pitcher has done and that stuff matters. If we’re trying to figure out the pitcher’s value absent the defense and batted balls, FIP is great. If we’re trying to figure out the pitcher’s value absent sequencing, BsR9 is great. They all tell us something different like AVG, OBP and SLG.

I took a look at the top relief pitchers last season using a 50 inning minimum. I figure that allows for a little bit of DL time. It also includes the occasional start. Only 6 of those pitchers had more than 6 innings pitched as a starter. None of the top pitchers had any as a starting pitcher, which is what we’d expect.

Hong Chih Kuo allowed the fewest runs per 9 innings at 1.20. Joaguin Benoit was second at 1.49 followed by Joakim Soria (1.78), Billy Wagner (1.82), Mike Adam (1.89) and Brian Wilson (1.93). Those were the only relievers who allowed fewer than 2 runs per 9 innings.

Kuo’s 1.29 BsR9 was best in baseball last season followed by Mariano Rivera (1.83). Those were the only two below 2. Rafael Soriano and Carlos Marmol were at 2.01 while Benoit was 2.03.

I calculated FIP myself and I noticed the numbers do vary a bit from what Fangraphs has though I’m unsure why. It could be intentional walks and hit by pitch, but I’m not sure. It doesn’t matter anyway. Kuo was also the best in FIP at 1.92 and only he and Carlos Marmol (1.97) had an FIP below 2. Thornton was right at 2.00 followed by Heath Bell (2.03), Wagner (2.08) and Axford (2.09).

Relievers allowed 4.38 runs per 9 innings last season. The top 10 relievers last season allowed a 1.86 RA. Calculating ERA+ in a way that tells us the percentage better or worse than average, that’s a 158 ERA+. I used this formula: ERA+ = 100 + 100 * ((ERA – pitcherERA) / ERA) where ERA is the average of all pitchers. In this case, I used relievers. Those 10 pitchers were 58% better than the league average reliever last season.

Only one pitcher in that group has not pitched this season: Billy Wagner. The first thing I noticed was that the 9 pitchers were on pace to pitch about the same number of innings as they did in 2010. Then I noticed they were considerably worse. Those 9 pitchers have a 3.32 RA, which is 21% better than the league. Still very good, which we also expected, but not nearly as good as the previous season.

I don’t want to bore with you with each stat like I did with RA, so I’ll just say what the best 10 in BsR9 and FIP were and then look at how they have done this season. The top 10 in BsR9 was 2.13. This year they’re at 3.63. The top 10 in FIP posted a combined FIP of 2.11 last season and this year those same pitchers are at 2.85.

These same groups were awesome last season and they’ve been very good this year, but those 150ish ERA+/FIP+/BsR9+ numbers were down to around 120 to 130.It’s not too surprising that FIP had the smallest difference. It’s measuring only the pitcher’s contribution while the others include some things out of the pitcher’s control.

if you’re wondering where Marmol ranked last season, he was 24th in RA, 4th in BsR9 and 2nd in FIP. As expected, he’s regressed some this season and ranks 29th in RA, 57th in BsR9 and 35th among the 113 relievers who have thrown 30 or more innings this season. As I said, Marmol is still a good reliever and he was worth his contract at the time. I’d rather the Cubs have not signed him. He’s not one of the elite closers in baseball and the only season he has been was 2010. The Cubs didn’t pay him to be as good as he was a year ago so at least they didn’t offer him a ridiculous contract.