Effectively Wild: A FanGraphs Baseball Podcast - Effectively Wild Episode 373: Why the Manager Challenge System Might Be Broken

Episode Date: January 28, 2014

Ben and Sam talk to Dan Brooks about why the expanded replay review system should motivate managers to challenge early and often.

Until I see you around, until we clear the accounts, leave it there. Leave it to us, we are the challengers of the unknown. Oh la, oh la, oh la, oh la is our good friend Dan Brooks. You know Dan Brooks as the developer of brooksbaseball.net. If not the best site for baseball information on the internet, certainly the second best site. And out of respect for my company, I won't even name, I'll just let you guess who I mean for the first one. But Dan also wrote a wonderful essay in the BP Annual this year, which we're not here to talk about, but I'm noting it. It's wonderful, and I really enjoyed it. And we're here because he brought a topic. That's

mainly why we're here. So Dan, how are you? I'm pretty good, man. How are you? That's mainly why we're here. So, Dan, how are you? I'm pretty good, man. How are you? Good. Okay. And basically, I realized that I'm on this podcast for one reason and one reason only, which is to provide you all with a topic because you're now sick of picking them.

Yeah, pretty much. now sick of picking them. Yeah, pretty much. Well, you know, you actually, um, you, um, you inspire a lot of the things we talk about, um, sort of behind the scenes because you are often asking us provocative questions and forcing us to think them through. And so I, I, I actually figured that, um, uh, you might just give me a topic. You might just tell me one if I asked you, and then I thought, no, we can go further. And particularly, I wanted you to come on so that you could plug Saber Seminar. So we'll do that either now or later or whenever.

This is all just a, like, soon I'm going to be providing you the topics, then I'm going to be on the show, and then soon you're going to be off the show and it's all just an exit strategy for you basically yeah well this is this is I hey look I appreciate that this chair I sit in

no man can sit in it forever if the powers that be decide that I've got to stop sitting in it like Jay Leno I will go out with class. Jay Leno went out with the least class possible. He's been kicked out and re-kicked out like seven times. Yeah. I will go out with much more class than that.

You will not hear a single word from me in protest. So what do you want to talk about? All right. So I'd like to talk about something that I actually wrote with Russell. I don't know exactly when it's going to run on BP, but it's about challenges and the replay review system. So here's a simple question, right? So imagine you are a manager and you know the rules of the new challenge system in MLB, which are essentially that up until the seventh inning, managers get a challenge, a red flag, a pink flag, I don't know exactly what color it's going to be.

They get to throw this flag to initiate a replay review of any play, basically they want. And, you know, there are a series of restrictions you know they can't do certain things they can't argue balls and strikes um but you know is there actually a flag are we just calling it a flag or is there actually a flag i have no idea i mean there's an actual flag in football. In football, there's a flag. Right. Okay. There's no actual baseball flag. It's a metaphorical, figurative flag. I feel like they're going to get a flag.

You don't think they're going to get a flag? I hope they have their own thing. I hope it's, I don't know, a rosin bag or something that they throw. Wait. Why do they need something? or something that they throw? Wait, why do they mean something? Yeah, well, they have to relay the challenge to the umpires somehow. Presumably they could just come out of the dugout or say something. Well, they need indisputable visual evidence of the fact that they are challenging the play,

which is the flag being thrown onto the field. i don't know we know i know okay what they should throw they should throw the water cooler out onto the field like they should dump the water cooler out like in protest and then they should take a base that should be the sign that they've that they've protested they've thrown a water cooler out and then stolen a base and taken it with them every single time. I think they should just throw their smallest player out onto the field, whoever that is. I thought Touvet would be constantly at the ready.

So, yeah, so this was not my topic. And the Tigers never have a challenge. This was not my topic, but this would be a fantastic topic. You know, what should managers throw onto the field and request a challenge? We'll come up with something. If you want to just do this one. We'll think about it.

Or people can email us if they have a good flag alternative. Okay, so anyway. So there's something that they throw on the field, or not, maybe metaphorically throw on the field. And so the question is, is where would you set your threshold for wanting to challenge a play, right? So there are different possibilities, you know, like maybe you don't want to challenge plays until somebody hits a two-run home run and, you know, there's a screwy call made on the field, and you want to save your challenge or something like that.

Because those are the calls that we tend to remember. But there are also other possibilities, like maybe you want to challenge any play, even if it's sort of close, and you think maybe you have any chance. And so that's the question, right? It's where would you set your threshold? And we should, should i guess remind people how many what the what the current challenge system is so what was it as

finalized uh in the latest press release was it just one one challenge unless it's upheld in which case you get another one for the first six innings is it and? And then after the sixth, it's just umpire's discretion? Yeah, I mean, so like there is something in the rules, which is sort of weird, that says something about managers being able to challenge after the sixth inning. But umpires can initiate a challenge after the sixth inning. And so, I mean, we just assume that from the seventh inning on,

it's going to be handled by the people on the field. And if you throw your whatever, your smallest player on the field or whatever, they're just going to kick them back in the dugout and be like, look, this is our job now. Uh, but do you think that the default, I mean,

in the, uh, the way the NFL has it essentially is that on the plays that matter most, you know, scoring plays late in the game or, or I don't actually know that answer this but i know that like there is some threshold where the play is considered important enough that the that it's just an automatic review like in the last two minutes or something like that

it's just automatic so do you think that that seventh inning on when it's essentially umpires discretion does the default be to just review everything or does the default be that this is when Joe West gets his revenge you know it's probably going to be it's it's probably going to be um pretty well handled I mean it's a good question but but but you know hopefully they're they're they're good enough that when something comes up in the seventh inning or the eighth inning or the ninth inning they're not just like you like a few guys and they actually try to do their jobs the best as possible. And I think that's sort of what umpires want to do

is they want to do their jobs as best as possible. And if there's some technological tool they can use in the ninth inning to make sure they got a call right, I don't think they're going to hesitate to do that. I hope. All right. So is the question been asked? Is the question on the floor? are going to hesitate to do that. I hope. All right.

So is the question been asked? Is the question on the floor? Yeah, the question's on the floor. The question is, where do you put the pressure? So, like, how often do you throw the challenge flag? So there you go. So that's the question. You're a manager.

The question is, how often do you throw the challenge flag? And can an umpire use his discretion in the first six innings, too, if the manager is out of flags or water coolers? Can they still instigate a review if the call is in question? Or are non-flagged reviews completely prohibited? We don't know this. I'm throwing it out there for you both to go, I don't know, actually. I'm going to go ahead and assume, actually,

that no, the umpire can't get a call reviewed. All right. Because that's how it is in football. Once you're out, you're out. So I imagine that they're not going to be like, well, you have challenges, but you have sort of other challenges if you go out and argue with the umpires. That would be lame, right? All right, so let's first, before we talk about the winning the game element of this,

let's acknowledge, I mean, the managers are human beings who want to be loved and want to look good at the end of this. And so there is some element of what is more likely to get them ripped that might come into play. go against him because he didn't throw a flag that he could have? Or is it worse for it to go against him because he wasted his flag early and no longer has it to throw? And I feel like, to me, there's something about being kind of having expended your resources having gone broke having squandered your fortune i guess so to speak there's something uh like it's the opposite of the american rags to riches story where you you go from having a resource to being broke that nobody respects and so i feel like wasting your flag early is potentially worse for the manager's rep.

And in addition to that, if you're debating whether to throw a flag early in the game, you know that you're not going to have a completely egregious call where it's like the 1% of decisions that are most obvious and most detrimental to you if you don't get it right. Whereas if you've thrown your flag, you don't know what could happen in the final four innings, and you might get that call. You might get the grand slam that would certainly be overturned and that would just completely kill you. So the potential stakes of not having a flag when you need it are much greater than the potential stakes of not throwing it on a borderline call. So I would guess that just from that perspective, the manager's incentives skew toward holding on to the flag, to hoarding flags.

If they could, they might even hoard them until the end of the series, if they can swing that. Just at the end of the year, they've just got like 200 flags, and they're just throwing them all. Oh, yeah. No, no. I totally agree with you. I think so.

So that's one aspect of it, right? I think that if they could, managers would hold onto these flags because you're going to get toasted in the media if there's some critical play that goes against you and you've screwed up by using your flag on some out-safe-at-first-base call in the second inning, whatever. And in the bottom of the sixth, there's a grand slam that could be overturned. You're not going to hear the end of this for the rest of the year.

Home runs, I think, as I recall reading, are not subject to challenge. Home run replays will work under the same system that they have worked under to this point. So a manager can request that an umpire review a home run call, but it's up to the crew chief to make that decision. You can't use an official challenge on a home run, as I understand it. And they traditionally, over the last couple of years, since they've been doing replays on home runs, it has always been essentially that when

the manager asks they do it there's not a lot of fighting and so the umpires have been very good about you know just saying well you know there's no harm in this so we might hope we might be optimistic the man the umpires will use every opportunity they can to uh you know to have as many replays as possible once it's all right so so We can throw out the home run then. It's the bottom of the sixth inning. Bases are loaded. Miguel Cabrera is up at bat. He hits this scalded line drive to left center field. Mike Trout goes back on the warning track, slides, and makes what appears to be a great catch. But on the TV,

it's clear that he traps the ball. No, I'm sorry dan dan i have to interrupt you no he did make the catch it's it's mike trout he did make the catch you've made a mistake dan all right okay sorry we could take whatever you know um uh johnny gomes bumbles around in left field for a while. He did not make the catch. He did not make the catch. You know, in fact, in that case... Use a less black and white example here. In that case, it's actually the replay is a gift.

All right, look, so the point is, if he can come up with some great, you know, runs going play where it's critical that the manager has a challenge flag. And if he doesn't have one, he's going to get roasted in the media. Yes. But, okay, so never mind that part. Okay, so you've got that part. So never mind that part.

Assume that umpires or managers were not real people and that there was no sports radio and um you know skip bayless or whatever had suddenly uh gone off the airwaves um you know so so then what would you do um i feel like i haven't heard enough from ben and so i'm gonna i'm gonna stop talking in a second but uh no no yeah just go ahead, Ben. Well, I've already read Dan's and Russell's article. So my opinion is very much colored by their analysis in that article. So I'm not sure what I would have said before reading that. But do you want to run through the analysis that you did or summarize what you and Russell concluded?

Because I would probably just be parroting that at this point. Is there – can you give like a multiple choice quiz to me and see if I get it right? Because like it's just a very complicated question. I mean, how you're basically asking how certain do I need to be that the call is going to be overturned and how, what is the kind of win expectancy have to be for me to, to, to want to use it? So basically like if it were a, if it were like a 0.3 win expectancy shift and I were 80% certain that the call was right, then the throwing of the flag would be, you know, a 0.06 win expectancy. So that's pretty big. I mean, like that in that case, that would be like about, you know, like half of a first inning

solo home run is the equivalent of that um yeah first inning double basically and so is that what you're asking me what what uh you know what kind of pro-rated winning uh win percentage uh shift do i need to have before i throw the flag yeah there's two there's two variables there's two variables here you're asking me right and so it's hard to give an answer that addresses both yeah so so let's talk about runs and not win expectancy because we're really dealing with the first six innings. You know what I mean? So there's not huge shifts either way, I feel like,

if we talk about runs versus win expectancy. But whatever, we can talk about either. But so the question is, on some marginal call, let's say safe versus out at second base on a steal, would you say that top of the third inning there's nobody out? I don't know what the probabilities are exactly, but let's say, would you want to be more than 50% right? Would you want to be more than 50% sure? that you know would you want would you want to be more than 50 sure so uh base runner top of the third runner on first nobody out attempts to steal uh and uh it's 50 50 that i get my call

um would you want to be more or less sure i would uh ignoring for a minute that the count on the batter might in fact might be relevant as well, I would say that I would throw the flag on that one. That seems like a pretty big shift to me. Okay, so would you want to be more or lesser than 40%? I would throw the flag at 40% that I'm right. 30%? Well, see, now your tone and your voice just changed.

No, no, no. I was just asking, where is the line? You keep confirming some random percent. So what would the line be? I think 40% is about my line. Around 40%, I start to feel less comfortable if I'm two-thirds sure that the call actually did go against me so here's the problem we have the math in the article

I mean I could run through it but nobody can actually keep track of numbers when listening to podcasts in a car so the moral of the story is that basically challengeable plays are so infrequent yeah espn right so i immediately regret it i immediately regretted my decision so so summarize that what that espn study found if you can or give the

general idea it's basically that challengeable plays are infrequent. And of those challengeable plays, the umpire gets it right a fair portion of the time. And even further subsetting the problem is the fact that, say, on some random half of calls, it's either going to go in your favor or against you, right? So you wouldn't want to throw the challenge flag if it helped the other team. You know what I mean? So it becomes this sort of like ever-shrinking probability problem, where the base rates on having a useful play to challenge have gotten so small

that basically if it's in front of you, that's sort of all the important parts are, is there a play that I could challenge that I could maybe win? And at that point, you might as well throw the flag, right? Because the chance that all of those things will line up again are actually pretty small. And so the problem is that the rule as it's sort of currently conceived, especially because there's no cost to making frivolous challenges, basically forces managers into this position where they have to make, I mean, if they want to be doing the right thing in terms of gaining runs for their team, they have to make frivolous challenges all the time, right? It's like you should be about 20% certain on the first play you see, you should just throw the flag, right? Because it's just unlikely that that situation

is going to occur again. So the numbers from that outside the lines thing were 1.35 calls during the average game that they found were close enough to justify using replay. And since we're only talking about the first six innings, it's like two thirds of that 1.35, which is 0.9. And then, as you mentioned, you would only challenge when the call goes against you, which over the long run should be half of the time. So you're talking about 0.45 plays, less than half a play on average that would be replay eligible, that would possibly be overturned in six innings. And most of those wouldn't be overturned, is what I think I heard Dan say. But on the other hand, a half a play per game is more than I thought. I mean, it puts in perspective the original rules, which were, what, two in the first six innings and one in the last three, right?

So that's a crazy amount. But a half a play a game is not nothing. You can conceivably feel like you are using your flags, that you are making use of them without throwing frivolous flags over the course of a season. Um, is there any way that a flat, do you, uh, Dan, I know that at one point, uh, we talked about whether, uh, we would start seeing like flag shenanigans, like, uh, I forget what it was. I remember giving guys more time to warm up that's yeah giving guys more time to warm up exactly so but i can't really think of any significant ways that we'll see flag shenanigans i think you might see uh the thing that you might see is the race to get the

next play in before the manager has a chance to throw a flag on the opposing team. So you might, like, if you, if a guy hits a, you know, if a guy hits a ball and it's caught and you're worried that it's going to get challenged, and there's a runner on first already, then you might conceivably have the pitcher just run to the mound and then, like, throw the ball away to get a play in the books, something like that. But as far as actually throwing flags frivolously to try to get a play in the books, something like that. But as far as actually throwing flags frivolously to try to get some advantage, it's hard to see it. So they should at least be on topic, probably.

Well, the race to get the challenge flag out would be actually more interesting, I feel like, with the water cooler, you know? You have somebody like John Farrell running down the dugout, grabbing the water cooler, you know, you have like somebody, John Farrell, like running down the dugout, grabbing a water cooler, throwing it out there. Would you have gotten back there in time? No, but the point is, so yeah, I don't think we're going to see that. I don't think we're going to see any flag-framing shenanigans. But you do have this weird situation where, and I think this is sort of the point you were making in the first part,

where it's like, you know, okay, so the media is going to hate you if you do this, right? So if you're always challenging the first play that's possible to challenge, the media is going to hate you because over the course of the season you're probably going to miss one or two big plays, even though you're going to have gained your team a lot of runs by, you know, putting extra guys on second base or putting extra guys on first base or whatever, right? You know, the umpire theory.

You're also just going to be wasting people's time, or that would be the perception, that you're making the games longer and slower and more boring. that you were making the games longer and slower and more boring. Well, and the other thing, you've instructed us to ignore that these are people and that they exist in a Skip Bayless world. But, I mean, all the evidence you need to see, or all the evidence you need to anticipate that managers are going to be too stingy with their flags and hold on to them too much, at least in this rationalist perspective that you're offering, is the way that they treat their backup catchers, right?

I mean, one in 500 games, maybe you might have your catcher leave a game for injury in like the ninth inning or later, and yet there are managers who will never, ever use their backup catcher for fear of running out of catchers, right? I think that's true. But, you know, the thing is, is that there's like this weird, crazy scenario in which you can imagine like the catcher getting injured and the backup catcher getting injured. And then you have like Derek Cheater catching for an inning and you're like, oh, God, what have I done?

catching for an inning and you're like, oh God, what have I done? This is even a more costless scenario, if that makes any sense. There's no reason to hold the flag, right? The worst possible thing you could do is at the end of six innings have this flag in your back pocket never having been thrown because essentially you've gotten you've wasted a chance for there to be some you know chance outcome on a replay review for your guy really getting safe at second base or your guy really got been safe at first right yeah right i think that's right i don't know that i that is right so yeah that's right. I don't know that that is right. So yeah, that's right. But I don't anticipate seeing it that way necessarily in action.

I would imagine that it might take many, many, many years before baseball's wider culture appreciates that. I imagine that Joe Madden will be just a flag-throwing fool, and it will be presented as sort of like this quirk, this Belichickian quirk, instead of just like a thing that everybody should do. It will be seen as radical the first couple years. I don't even know if Joe Madden will do it, right? Because – so not only will the media hate you, the umpires are going to hate you, right? That's true. that's true. You know, the umpires are going to be pissed. You know, like, oh, goddammit, here comes Joe Madden again.

He's a stupid flag. You know, he was clearly out, but this jerk thinks he was safe. You know, the players are probably all going to hate you. You know what I mean? It's like, you know, how annoying would it be if every time you were thrown out at second base, your manager came out of the dugout and was like, no, no, no, no, we need to get that looked at on video. And, you know, I think that actually Joe Torre and Tony La Russa and everybody else on this committee is going to hate Joe Maddon, right?

Because he'll be doing something that doesn't seem quite right, right? Like this was supposed to be the failsafe for, you know, an umpire making a horrific mistake. And instead, Joe Madden is using it as a way to compile hundreds of a run over the course of 162 games. You know what I mean? And so it sort of perverts this challenge system into this, you know, really tiny accumulator. So long as you throw the flag often enough and, you know, with any amount of certainty, you're gaining a little bit of an advantage. And maybe that advantage over the course of the season is worth, you know, five runs or

six runs. But hey, that's $4 million or whatever. So, you know, that runs or six runs, but hey, that's $4 million or whatever. So, you know, that's a pretty good deal. So you think that if a manager were to pursue the optimal strategy here, that it would sort of ruin the expended replay system that that it would be a Destabilizing thing that it would it would be applied in a way that was not intended and we would all get sick of it and annoyed but because It's unlikely that anyone is going to Look at the numbers and throw the flag as often as he should

look at the numbers and throw the flag as often as he should, maybe we will bite that bullet and not have a problem because people will be behaving in a counterproductive way? Yeah, you know, I think that this system is sort of strange in that, I don't know, I tend to think of rules in games like if you make rules for your games, you shouldn't know, I tend to think of rules in games like, if you make rules for your games, you shouldn't make it so optimal behavior is universally detested. You know what I mean? Like, it should not be that the best way to play the game is everyone's least favorite way to watch the game. That doesn't sound like a great idea. If we were to come up with a new way to idea if we were to come up with a new way to, if we were to come up with a new rule set, I don't think we would want that. You know, and I

don't think we would want to design that in. And so, you know, the other thing that's sort of odd about it, and you guys have brought this up before, but it's this strange punishment system where it's like, if the umpires are screwing things up, we it's like, if the umpires are screwing things up, we should just fix the things the umpires are screwing up. And so it would make a lot of sense probably to just take it out of the manager's hands entirely and just let the umpires handle it, you know? I didn't see that one as close as I should have. Maybe let's just look at the video. And, you know, I mean, that's what they're going to be doing from the seventh inning on. I would think that we should trust them for the first test skimmings, right?

Sure. I would have liked to see that. That was our initial reaction when the challenge system was announced. But since that ship seems to have sailed, at least for the time being, you did have a suggestion or some suggestions for how this could be mitigated somehow in the short term? Yeah, you know, I think the easiest suggestion is just to add a cost for frivolous challenges, right? So if it costs you an out when you're wrong, then the threshold will move from being really low to really high, right? So you won't want to be wrong, and so you'll only screw up or only challenge when the umpires really screw up, which I think was the point of correcting this system in the first place, right?

And, you know, I mean, it's sort of like a classic signal detection problem in that, you know, if the threshold for action is really, really, really low, you know, if the threshold for action is really, really, really low, then, you know, you sort of inspire this sort of weird strategy of just throwing challenge flags all the time. But as soon as you add that cost, then, you know, you really change the decision criteria. And so it really changes the way that managers should behave optimally and probably puts it closer in accordance with, you know, the way you intended the rule to be played out anyway. I would suggest that the cost for an unsuccessful challenge should be you lose your backup catcher.

How would you designate that person? You're not allowed to replace your cat. You're only allowed to use the cat during the game from that point on. But what if he gets horribly... Well, I guess you don't have to worry about home plate collisions anymore. That's right.

You're just designing all the fun out of the game, aren't you? That's the price you have to pay. You have to play with a mangled, concussed catcher for the rest of the game. You know, I feel like the one person who would hate that more than anyone is the home plate umpire. Can you imagine a mangled, concussed guy has to catch the 100-mile-an-hour fastball? He doesn't want that.

No. like niggles can cuss guy has to catch the 100 mile an hour fastball like he doesn't want that no well i'm sorry but it's the only way to make this happen well uh mlb has has said that this is sort of a work in progress system that they'll sort of see how it goes this season and possibly make changes after that uh So maybe if Joe Maddon does become a constant compulsive flag thrower, that's something they'll take into account. Who is the smallest player on the race? Is Sam Fold still around? He's a free agent, right? Oh, that's a good guess. Yeah, he's not signed yet, but

well, they could re-sign him for that express purpose of being thrown. That would be an interesting wrinkle, wouldn't it? You'd have a race to the shortest guy in the majors just so he could be like the guy who gets thrown onto the field.

You guys are being, I think you guys are being unrealistic. Yeah, maybe. Just a bit. Well, it's an interesting wrinkle to consider. You can read Dan's

and Russell's article at BP probably today at some point, Tuesday. If not, certainly Wednesday. I will add a link in the Facebook group and in a comment on the blog post at BP so you can be sure to see it. Would you also like to bring up the Sabre seminar now? Yeah, that's a good point. So we just announced the dates for Sabre seminar, Sabre Metrics, Science, Sports, Scouting, and the Science of Baseball.

I always screw that up. It's going to be August 16th and 17th at Boston University in Boston, Mass. It's the weekend that the Astros are at Fenway. So if you've ever wanted to visit Fenway and see the Astros, this is your chance. But, you know, we bring in a bunch of people, you know, talk about baseball, share their experiences. You know, last year we had John Farrell and Brian Bannister and Tom Tippett and Keith Wohler and other great people like Ben

and, you know, Harry Kalidis and others talked about baseball. It was great. Yeah, and it's for a good cause. It is for a good cause. It's all for the Jimmy Fund. Sorry. These unplanned things, I'm terrible at them. So yes, we have raised many thousands of dollars to the Jimmy Fund. 100% of every ticket sold goes to the Jimmy Fund.

And so it's been fantastic the last couple of years. We've had great audience feedback. We had 13 or 14 MLB teams there last year. It was a lot of fun. Yes, it was. I went last year, my first time last year, and I would recommend that if you enjoy this show, you would probably enjoy attending that event

because there are a lot of interesting speakers over the course of a couple days in a lot of different disciplines uh so you will probably enjoy attending uh and i guess that's that's all we need to say now um so you can follow dan on twitter at brooks baseballball you can go to his site Brooks Baseball at brooksbaseball.net for tons of interesting pitch effects info that Sam and I use in our articles all the time

and thanks for joining us Dan no problem alright so you can send us emails at podcast at baseballperspectives.com we will be back with another show tomorrow you're on because you promised to bring a topic that's true i did bring a topic and because when i came up with the idea

of literally holding a guest hostage unless they brought a topic I looked at my gmail to see who was online and you were Wow such an honor I'd like to thank my parents for you know bringing me into the world you should thank Ben's parents you should thank Ben's parents for bringing him into the world yeah that's the only reason for bringing him into the world

yeah I wouldn't be here if it weren't for Ben's mom well that's you know we're just going to end that there