Which Teams Use Their Bullpens Optimally?

Earlier this week, Casey Boguslaw posted an excellent article over at Baseball Essential regarding Lineup Optimization. The premise of his argument revolved around comparing a team’s wRC+ and their run production per game. In theory, a team with a low wRC+ but high R/G implies that the lineup has been optimized, i.e. they are squeezing every run out of which they are capable out of the lineup. Conversely, a team with a high wRC+ but low run production is suboptimal, and not scoring as much as they should.

Let’s apply this concept to bullpen use. Each team has a certain number of relievers they are able to use in different situations. Similar to the lineup, different points in the game are more or less crucial. This is tracked by the leverage index (pLi). In a few words, a game situation of average leverage has a pLi of 1, with more intense game situations greater than 1, while less intense situations are less than 1. For a bullpen to be optimized, as the leverage increases incrementally, better and better relievers must be used, i.e. the relationship is roughly linear.

Separating Good Relievers from Bad Ones

It can be tough to gauge reliever effectiveness, as they enter the game in wildly different game situations. Another obstacle is the assignment of earned runs. If a reliever enters the game with the bases loaded and lets in a few runs, they will not be charged with the runs and a different pitcher will be. In order to optimize reliever usage then, we must use a different system of assigning blame and awarding praise.

To do this, we will use Run Expectancy Wins (REW). REW tracks the number of wins a pitcher adds/subtracts based on the number of runs they give up in the 24 different Base-Out situations (e.g. 1 out, men on 1st and 2nd, etc.). By using REW, we can track the context-dependent runs for which a reliever is responsible.

Notice that REW is very different from the other advanced pitcher statistics. Most of those statistics look to assess a player’s performance regardless of context, in order to determine their “true” ability. The reason that we use REW for relievers in this situation lies in the fact that we are looking specifically for their context dependent stats to determine if they are the correct reliever to use.

How Teams Deploy Their Bullpen

To gauge a team’s bullpen deployment, we will evaluate the strength of the linear relationship amongst their team’s relievers. That is, we will fit a best-fit line across the different relievers for their pLi vs. REW graph. A team with optimal reliever usage will have an r of 1 (as leverage increases, reliever REW increases linearly), while a team with perfectly suboptimal usage will have an r of -1 (as leverage increases, REW decreases linearly). A team that haphazardly or inefficiently uses their bullpen will have an r of 0 (there is no relationship between leverage and REW).

To ensure that only a team’s main relievers are counted for, we’ll limit the IP count to at least 10. The data used, through 6/22, can be found here on Fangraphs. Below are the rankings for each of the 30 MLB teams, separated by league.


Bullpen Use Optimization for each MLB team during the 2016 season as of 6/22

It is no surprise to see teams with multiple flame-throwing relievers, like Kansas City, Tampa Bay, and New York top the list. Quite a few of the teams toward the top of each list are also known for their extensive analytics departments, like Los Angeles, Oakland, and Pittsburgh. Similarly, some of the teams toward the bottom of the lists are known for their more “traditional” approach to the game, like the Nationals and manager Dusty Baker.

It is important to note at this point that the fact that a team optimally uses its bullpen does not imply an inherent quality to the bullpen. Rather, it merely indicates that the team is using the pieces it has appropriately. A good example of this is the Reds’ bullpen, which has far and away been the worst in either league, with a 6.21 ERA. A large divide occurs between the difference in bullpen ability an bullpen usage.

The Best Individual 2016 Relievers Based on Game Context

There is also worth in evaluating relievers on an individual basis. Below is a graph that characterizes each reliever that fits the above criterion. Relievers generally fall into one of five categories. First is the general “relief” category, in which relievers appear in variable leverage situations with variable results. Then, there are relievers that are “Stoppers”, i.e. relievers that enter high leverage situations and deliver appropriately high performances. Third, there are relievers that enter high leverage situations and disappoint, referred to as “Letdowns”. There are low ability relievers that enter low leverage situations and measure to “expectations” and as such are referred to as “Meltdowns”. Lastly, there are relievers that enter low leverage situations and perform even better than expectations, and are referred to as “Firemen”.

For purposes of definition, we define “low leverage” as pLi of less than 0.85, and “high leverage” as pLI of greater than 1. Further, exceptional great performance is defined as a REW of greater than 0.6 wins and a poor performance as less than 0 wins.

To simplify, “Stoppers” are the best relievers in the game, those that are thrust into difficult situations, and perform as expected. “Firemen” are relievers that can likely be given more difficult assignments, as they outperform low expectations. “Letdowns” are those relievers that are used in important situations and don’t measure up to the task. Lastly, “Meltdowns” are the relievers of lesser ability that are used in lower leverage situations, like blowout victories or losses.


As exemplified in the graph above, reputation does not necessarily correspond with performance. For example, Jonathan Papelbon, the all-time leader in saves for both the Red Sox and the Phillies, qualifies as a “Letdown”. Sometimes, a reliever’s reputation fits their ability, like Wade Davis of the Royals. His success in high leverage situations is part of the reason Kansas City is able to use their bullpen so optimally. Chris Devenski of the Astros is one of the best relievers in low leverage situations, and should probably be trusted with more responsibility out of Houston’s bullpen. Lastly, pitchers like Luis Perdomo of the Padres and Evan Marshall of the Diamondbacks have shown to be guys that should only be brought in during lopsided games.

Interpreting Bullpen Results

Teams with deep bullpens like the Royals, Yankees, and Orioles can stretch their success even further by using their resources in the most optimal manner. By correlating game context and reliever ability, they can conserve inning counts for each reliever and prevent fatigue and meaningless usage. The metric presented above attempts to quantify this usage, with the caveats of small sample sizes and the lack of quality interpretation.

Similarly, teams may be able to identify relievers for trade deadline acquisition or role “promotion” by identifying Stoppers and Firemen. Stoppers obviously come with a heftier price tag because their success is more apparent, but Firemen might be able to have the same success at a lesser cost. A great example of this was the 2015 Pittsburgh Pirates acquisition of Joe Blanton, who was very successful as a Firemen for them down the stretch. It is no surprise that Blanton appears on the graph above in the same section in 2016.

The categorizations again come with the caveat of very small sample sizes, and year to year variability. Year-to-year consistency would indicate the best uses for each reliever however, and could help teams identify extension and free agent candidates. The best relievers may be those that are not even recognized as such.

Readers, what do you think? What teams do you feel use their bullpens the best? What factors do you use to make your decision? Which players do you think teams will look to acquire at the trade deadline to help? Leave your thoughts in the comments below or on Twitter @SaberBallBlog. Don’t forget to subscribe to SaberBallBlog by clicking the green “Follow” button in the menu, and follow on Twitter for all of the latest updates on the MLB!

Share This Post:

12 responses to “Which Teams Use Their Bullpens Optimally?”

  1. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  2. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  3. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  4. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  5. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  6. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  7. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  8. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  9. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  10. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  11. […] SaberBallBlog breaks down which teams are the best at using their bullpens. […]

  12. […] the first inning or two, the order largely gets jumbled, and different players come to the plate at different points in the game with different leverages. With this in mind, which players have been the best this year at leading off an inning for their […]

%d bloggers like this: