The Athletic Football Show: A show about the NFL - The state of NFL analytics: It's more than going for it on 4th down

Episode Date: July 20, 2023

Analytics departments across the NFL have moved way beyond, "So, should we go for it on fourth down this time?" But where exactly are they, and where are they headed? Robert Mays and Sam Schwartzstein, analytics expert for Amazon's NFL broadcasts, dive into those questions on this episode of The Athletic Football Show.Follow Robert on Twitter: @robertmaysFollow Sam on Twitter: @schwartzsteinsSubscribe to The Athletic Football Show...AppleSpotifyYouTubeThis episode is sponsored by BetterHelp. Give online therapy a try at betterhelp.com/MAYS and get on your way to being your best self.The Football 100, the definitive ranking of the NFL’s best 100 players of all time, goes on sale this fall. Pre-order it here. Hosted on Acast. See acast.com/privacy for more information.

This is the athletic football show. Welcome to the athletic football show. I'm Robert Mays, cool show for you guys today. When I was thinking about some of the discussions I wanted to have before we really got into the meat of the season, before training camp started and your life just gets taken away from you in a good way. And you have a little bit more perspective. Five years ago, it just felt like we were having so many discussions about the state of analytics in the NFL and what was possible. And I think that's for a few different reasons. The tracking data became available in 2019.

or so, and it became a huge point of discussion, how teams were going to use this, what was going to be doable within these buildings, how science fictiony was all of this going to eventually get. And that's also when we started seeing, quote-unquote, analytics people becoming the heads of certain personnel department. Sashi Brown took over the Browns. Obviously, we have a couple other examples of that since then. But over the last four or five years, it feels like the state of analytics in the NFL has become more and more of a black box. and it's become less of a discussion point. A lot of our conversations around analytics, quote, unquote, in professional football are now about fourth down decision making. They're about stringing up coaches in the public square when they do or do not go forward on fourth down.

And before we got into this season, I kind of wanted to take a step back and just do a state of analytics discussion in the NFL. What are teams doing? What are they prioritizing? What's possible? What's not possible? What are we overrating? What are we underrating? I try to talk to people within teams as kind of a precursor to this.

I had conversations with people that work in these sorts of departments with about a quarter of the league just to get a little bit of perspective. We're not going to quote people from those conversations as I think that's the safest way to do this because this is still very cloak and dagger. But just to get a lay of the land about what is going on in some of these departments around the league. And someone else who does a ton of work in that exact same space is Sam Schwarstein, who is the analytics expert for Thursday Night Football and Prime Vision. on Amazon. Sam, really appreciate you joining us for this conversation today. You do this all the time. So I thought it would be helpful for you to kind of translate a lot of this stuff into English for me and everyone else to understand. Yeah, this is exciting to be back on. And it's really scratching the edge of we're really playing football now. You know, sometimes the people might

think outlooks people are doing it in the spreadsheets. But a lot of analytics people are on the field right now watching practice. And we're getting into the swing of football season. All right. Let's start this discussion. kind of a bird's eye view on a very basic level. I just want to start with how these departments are kind of defined, structured, organized around the league. And I think it's worth beginning with just laying out how different these or these kind of groups and these departments can look around the NFL.

So what sort of differences are we looking at from team to team within the league? where you house your department, you'll see that team having success with analytics in that space. And what I mean by that is there are four main categories that analytics helps the team out. There's two within game plan and two within your roster management. So there's pre-game analytics, there's in-game analytics, then there's player acquisition, and then player retention or sports science and performance. All which we will get into. Right.

And so there's different spots that a team is housing that department. And so if you want to find out how your team best utilize analytics, just look at the team front office website. Once you see that that's where your team's analytics team is housed, that's where they'll have the most impact. Some teams have them spread across all four categories we talked about. But most often than not, you're going to see that wherever that team has their team, their analytics team, housed, that's what I have the most success. So that's going to be in the coaching side of it, on the personnel side of it. Some people have it on the business side or those guys came from the business side.

So the silos of where those things exist are different from building to building. And I think maybe even beyond that, the most important thing to keep in mind here, as we're talking about, the word is so, it's so dirty and it's so vague. And so I hate using it, but there's really no other way to do it because that's the parlance that we use. But the analytics department from team to team looks very very. vastly different, okay? For the Cincinnati Bengals, the analytics department is one or maybe two people, depending on how you're defining it, because some teams define the analytics department that includes kind of software people and people that build the in-house capabilities that they have.

But Sam Francis, who works through the Bengals, is really their only research and development guy. He's one guy. And then you look at, and that's not news to anybody. He's been featured on broadcast. You can look at the Bengals front office chart. I'm not blowing up anybody's spot. But you look at some of the other teams that have heavily in. invested in this. And it's hard to keep track of how many people are housed within their

analytics department. And those are the usual suspects. Those are the teams that you would anticipate, the Browns, the Eagles, the Ravens. I mean, these are teams with 10, 12, 14, 15 members in this department. So what teams are trying to do and what they are capable of is different just in terms of how many people they're unleashing on these tasks. And even if you take a step back, oftentimes they're housed under the technology team and the technology team is actually someone who came up through the video space because it's really not that long

where we were on analog tape to where we are now I sent my high school highlight tape out in 2007 on VHS by time I graduated college there were no DVDs it was a link and so this graduation of technology if you talk to a lot of different teams there's a guys who got their start in film rooms but when the word analytics was coming up,

the coach or the GM said, we'll just give it to the nerdiest guy on our staff. And that's the video guy. So that's where a lot of teams have a video guy as that person who's kind of rose at the ranks of analytics as necessity because this technology has changed so rapidly and so fast. So looking at the different sizes of these departments, one of the things I know you wanted to talk about

is just because your team has 15 guys working on these problems. It doesn't necessarily make your analytics department any better. And that was one of the interesting insights that I came away with from some of those conversations over the last couple weeks is that there are some analytics departments that have four or five guys, which is pretty typical around the NFL, four or five people. And they think that their work is actually more focused and more practical than some of these teams that are off chasing these far off realities and tracing all the tracking data and try to create all these new models. So why in your mind is bigger and not necessarily equal better when we're having this discussion? because this business is still a people business. When I was building out the rules of the XFL, I was using analytics to create all the rules. But the end of the day, I had to convince Vince McMahon and eight coaches that we were doing the right thing.

If I was going to talk about the expected points added by having different starting field position locations, that was not going to gel well with them. It was about identifying how could I help them better do their jobs or what is their biggest fears. So if you have a smaller team like our good friend Sam Francis and you're able to develop a relationship with those coaches, you then will have more insight and more exposure to what they need, which will then get more actionable results. Your job right now in step one is not to create the best models or to create the best features. It's to help your coaches accomplish their jobs. And so when you have a smaller team, you can develop a personal relationship versus being locked in an office on something. away from everyone else. It's actually best if you are embedded within the coaching staff or the personnel staff

or the health and safety staff to have the most results. I had a really good conversation with someone who is tasked with building one of these departments. When this person was looking at these departments, it wasn't necessarily who the smartest people would be or who could build the craziest models or who had the highest understanding of statistics or computer science back or any of this stuff. It was kind of who could hang.

And that's so funny that that that's a project. in this world that we think is so nerdy and out there. And these people are just kind of so far removed from that priority. And it's because if you can't get these ideas across, if you can't make the people tasked with the decision making trust you with these ideas, if they don't know who you are, if you can't inhabit their spaces, then the work that you're doing ceases to be important. And that disconnect, we've seen that with how teams have kind of reshuffled their staffs. My understanding is that a lot of these teams that have moved on from certain people in the

analytics department kind of reshuffled who they wanted in charge of those places, it's because there wasn't a level of trust. It was like, ah, the nerds over there don't really know what they're doing. If you can't speak the same language, then none of this is going to matter. So I think that is one of the biggest takeaways I had from everyone that I talked to is that this is 80%, 70% based on communication, 20, 30% based on ideas. And even that number may be a little bit high. Right. Football analytics is not Brad Pitt. saying can you get on base? Right?

It's a very different thing where you can just chase a number right away. It's more nuanced and especially because there's more coaches. You were doing one in baseball. I was referencing money ball in that Brad Pitt remark. In baseball there was a pitching coach,

a manager, and a hitting coach. They are between 16 and 24 coaches on the staff that you need to convince that your work is helping provide them a better chance to win, especially if you're teaching them something that they've never learned, right? Going forward on fourth down, being aggressive, not risky, calculated. That's where I differ, is being able to be calculated in a decision-making where you can put

yourself in a perceived risky spot. That is a nuance that takes time versus just saying big number better than small number. That's never going to work in football. You're going to need people that will, you can develop. of trust with by helping understand the game, helping make their lives easier than ultimately having new outputs and new things happening on the field, which is what all analytics departments want. Yeah, you have to be speaking the same language. And I think that's where a lot of, we can get into this, some of the player tracking data has kind of helped people in analytics department speak the

same language as scouts, because you're no longer talking about this slot receiver creates this EPA per route when he's lined up in this place. You're talking about acceleration. You're talking about speed, you're talking about traits that actually are understood by the scouting community. So the more that you can align the terminology, the tone, the way that you're talking about all this stuff, the easier it is to have the people that are in the decision-making spots embrace these ideas. And I think trying to cross that bridge as best as possible is maybe the biggest priority, the biggest challenge within these buildings. And it's what they're working on all of the time. So I wanted to talk about also, as we're considering the different sizes,

of these departments and the investment the teams are making in terms of how many employees they have chasing after these problems. There's also third party vendors that do a lot of this work for teams. I think every team in the NFL probably has a PFF ultimate subscription at this point. I believe all 32 of them do. The amount of data they get from PFF and the amount of legwork that PFF has done for them, not in terms of grades, but in terms of just situational statistics and just things that they would never be able to do otherwise is huge. There are other than companies like zealous and even more of them that are kind of popping up that are trying to do a lot of this legwork. So if you have a team that's a small market team, maybe throws a little bit less

money around, rather than having 10 in-house employees, you've got two, but maybe you employ some more third-party vendors to get a lot of this data. So from people that you've talked to, how does the third-party vendor process kind of impact the way that teams are working in the data that they have access to? Yeah. So when I built the XFL department, I looked originally at PFF to replace GA or QC work charting plays. I just say, okay, I'm not as concerned about the grades, but give me a way to replace two guys per team. But what made it fascinating is by utilizing a third party that would already chart all the data,

all the games for me, but then provide me insights that I didn't even have access to. So it provided me two extra analytics sources where I was focused more so on changing the rules and gameplay. I had some interesting insights that were being gathered from a third party that weren't looking at the game exactly how I was. So an actual result, I was looking at our 1-2.3-point conversion probability that I built at the XFL. So when teams should go for 1-2 or 3? And they found out a PFF was able to deliver data to me saying from the 3-point play, you say it starts at the 10 and it actually starts at the 8-yard line because of how many penalties took place. on the three-point conversions,

so it's actually more advantageous than going for two. I was able to then deliver that information to my coaches. Didn't go from the engineers at PFF, but from me to be able to communicate, hey, there's an opportunity to go for here, and here are the best plays that we've seen work so far from the 10-yard line to get actual results to where I saw more teams going for 3 and running QB draw, which was the best play from the 10-yard line.

So we are able to then take that. A lot of teams are doing that same thing, But the drawback of third parties is do I trust this group to do it the right way? So I've gotten in many arguments before. I call it green right. A lot of people call it I form. I write. I come from the West Coast world.

So everything was colors and formations. So some people don't trust, oh, I don't want to give PFF or a third party all of my information that I think is like black box information. so people get nervous about delivering that information to them. But you can offset a lot of the grunt work, you can offset a lot of the computational work, whatever you think to a third party that can help elevate you to that same level that a team might have 12 in-house computer vision engineers,

but there could be another product like slants or another computer vision product that gets you 80% of the way there. And then you can do what teams do best, which is game planning for their own team. my understanding is that a lot of the data that are accumulated from third parties that's not going to give you an advantage in any way, but it creates maximum efficiency. And there are aspects to a lot of the PFF data or some of these other services that have gotten a lot better. Coverage specifically came up in conversations where right now, compared to maybe five years ago, how teams are, how these services are bucketing certain coverages is just more accurate.

Like if you're looking at plays that are charted as cover three or charted as cover two, the, detail and the accuracy of things like that have gotten so much better that now you can rely on them in a way that you never really could before. So this is more about inputs and more about the data that you can kind of gather than it is about using PFF to create any sort of edge over the teams that you're playing against. Yeah. And even like next gen stats, which is doing it all from charting data, next gen stats has been fantastic resource to be able to identify trends that are coming up in different aspects of it. But also there's a ton of football fans at these departments. You know, they're excited to work on it.

So all you got to do is ask. Now, if you're worried about your trade secrets, that's fine. You can be fun if you're worried about that, but you play every game each week. And so if you want to ask those engineers to put together a resource for you, you can use them in that way. But mostly it is, yes, input so that your QCs are not doing film cutups, right? People don't understand film cutups. You actually just have to cut up the film. And that's why it's called a cutup.

So the less time they have to do that is they can have an auto push from PFF. saved reports, then they can actually look at how do we help the football team from an ideation standpoint, which right now no one else can do but humans. So getting away from kind of these extreme polls on both sides, the teams with 15 guys, the teams with one guy in terms of how they divvy this up and the investment they put into it. I want to talk about how most teams kind of attack these problems. And my understanding, based on conversations and just looking at org charts and other bullshit, is that most of these teams have maybe four or five people that are in these departments.

And the way that it's divvied up as far as my understanding goes is very similar to what you laid out. There's really like four to five kind of areas where the analytics department is doing work. One is game planning. Two is game management. Three, which is two different things. Game planning is what we're talking about here. You're doing the QC work. You're figuring out how often teams are doing this, a tendency based.

Game management is what we talk about all the time in this space. Four down decision making. But it goes way beyond that, which we can discuss. The third is probably player acquisition. and resource allocation. You know, you're talking about college scouting here. You're talking about pro scouting, but also salary cap management, how much you're willing to pay a player,

player retention, things of that nature. And then the fourth is kind of sports science and performance. So you're looking at how the tracking data affects or gives you insights into how fast the player is moving. You can get that about based on free agents. So those are the four kind of houses that people kind of communicated to me. I want to talk about the first one, first and foremost. We've talked about it a little bit.

That's the game planning. And that's this idea of replicating the work of QCs and kind of helping to maximize efficiency in the process. This is a huge part of what these organizations and departments do within these teams. And I think it's important to point out that this is work that's been done forever. It's just been done by an army of quality control coaches. And now it can be done by a ton of automated services and a ton of advanced statistics. One guy with a team essentially explained it to me is it's like having 10 quality

control coaches on steroids. So this is in line with what football preparation and football analysis has been forever. It's just a little bit more efficient now. Yeah, absolutely. And right, the first analytics sport was football. I had played tendencies in seventh grade football, like what formational tendencies. Texas football is a little bit different than other states. But, you know, I had that stuff. And this is just automating that process. And that's a large thing I used it for with utilizing service like PFF and Next Gen Stats. A big aspect of it is freeing up your people to then do creative things. Yes.

So you're helping a coach not have to spend time with crayons drawing cards for blitz pickup period, right? You're able to spend time with that coach and really understand what his problems are. So your first job is to go in and this is what a lot of teams do, coach, what's your biggest problem, then use analytics to automate that process. And then you get to interview that coach and say, what's something you really wish you knew? And that's when the game planning gets to the next level, when we're not just repeating those same things. And I think what's important to understand is every coach starts with film. And so if you're an analytics department, you're not getting through to your coach that you're a partner you're working with, your job is to understand, how do I make their film watching process better?

Yes. Do I automate those cutups? Do I automate their ability to then get that film, those animations on the screen? I've talked to many teams about how do we make prime vision in our video service that we're using? Because they want to make sure that they can help their coaches better watch the game. And so that's a huge aspect of where an analytics department will see the most success is making their coach the best film watching junkie. Because that's what these guys are trained professionals of doing is understand the film watching process.

and that game planning statistics and Tennessee basics, can you integrate it best with video. Yeah, if you, you can do anything. The level of granularity that you can do with these services is insane. You know, I can look at how many times that they run this blitz on third and five or fewer in this sort of front with whatever, whatever detail that you want. So you're really just streamlining the process. It's always tied to tape when we're talking about the sort of game planning and game

preparations, the game, the game planning stuff and the game preparation stuff. because most of the services that teams have, you can, every single thing you're pulling up in terms of, all right, these are all the times that they've done this. Within the software that teams have, you can literally click on every single play. So you're just streamlining the process for, this is what they do in all these situations,

and here you can watch it in 20 minutes. This would have taken you three days 20 years ago. And then another aspect of this game planning process that I found so interesting talking with them is a lot of times teams will have these guys be, there. The rule breakers or their rules analysts as well.

So when you're trying to understand, you're saying, okay, hey, give this game plan for this specific team, but you're also these analytics. People are looking at analytics on officiating. On where are spots on the field that we might lose coach to player communication? What cameras does this crew? Are we the A Fox game, the B Fox game?

We're on Thursday Night Football. Where did these cameras have the best visualizations for our review process? That's the same amount of data that's going in into this game planning process that was never there before. Because now we've automated the process to give coaches the information they want. Now the coaches get to ask for more and more and more. And that's where this stuff becomes even crazier. When teams told me, they're like, do you have your camera angles that you can send over?

I was like, whoa, this is too much. It's hilarious. And because certain types of penalties may get called or more reviewable based on camera. angles. And so that kind of takes me to the second bucket that I wanted to talk about. And that's this game management process that happens within these departments. And the aspect of this we all know about, we all talk about ad nauseum is the fourth down decision making. If you talk to people within these teams, obviously there's a lot of leeway and there's a lot of error within individual fort down decisions. But most of that stuff and the models that go into it, if it's not solved,

it's much closer to solve than all the other things that we're going to talk about. That's kind of salted away. The areas of this that we don't talk about enough and actually are more interesting are the stuff that you're talking about. Camera angles, the way things are officiated, but also game situations. So a lot of people within these analytics departments are tasked with watching the last three to five minutes of halves over the entire league, over each and every week. and you're trying to figure out if a situation arises that we don't have a plan for, what are we going to do? And you're going through all of these in-game scenarios with your coach every single week.

There is a person within these departments of pretty much every single team. That's one of their jobs, where they're sitting down with the head coach every single week and be like, all right, with three minutes left in the Broncos game, this weird thing happened. Like there was a rule that dealt with the play clock, and if this ever happened to us, what would we end up doing? So it's way beyond the forward down stuff. It's trying to figure out any scenario that could come up in a game, whether it's clock rate, it rule related, any of that stuff,

and having an answer for it in advance. Right. And again, to go back to film, coaches watch cutups. So they don't see what happens between the play often. It's why every fan at home is so much better at fourth down decisions and end game scenarios is because we watch so many on Redsend. We'll watch every end game scenario in real time. They don't.

They watch it by clips. And teams have gotten so much better at saying, we're going to watch our entire end game scenario. We're going to watch the whole four-minute process, even with commercial breaks in, to get better at these end-game scenarios. Because the way they watched film was seven seconds at a time. And then we're rewinding the seven seconds.

So it's a huge impact that these teams have made by saying, we're going to rewatch exactly said. We're going to do a – and the coaches now have to be players. And that was so new when I was at the NFL coaches did not like being players. They like graduating to the coach where they get to tell somebody. But no, when you put them on the spot as an analytics person saying, what would you do in this scenario? How would you approach it?

Here's what we know. Here's this rule that got changed this many years ago that people haven't adapted to. Here's an officiating part of the game that they've stopped officiating the same world. There's less point of emphasis here. There's different things you can now do that you have to then put that coach on the spot. And the coaches that buy in and work with their analytics down, the best there, they see huge dividends over time being able to adapt to these situational football. One example that came up and when I was talking about this with someone is the Eagles.

And the Eagles are just very intentional about how they think about all this stuff. And that's not an accident. We've talked about this in their personnel department. They turn over every rock when they're trying to find an advantage. And the Eagles look at timeout specifically kind of as resources to be used within the game. The Eagles will do everything they can to try to get a team to jump off sides, whatever, and then have these timeouts where if I can get it from two yards to one yard, then I can run that quarterback sneak that we use and we can go for two.

Even if it doesn't feel like it's necessary at that point in the game, because those timeouts are resources in that moment of the game. And so thinking about everything you have, challenges, timeouts, as little tiny edges that you can create, that is something that your analytics department can consistently do, and that goes way beyond should we go forward on fourth down in this exact scenario. It's funny you bring up that piece of it. I call it resource management.

Like video games. People forget this is a game sometimes. And you have things and resources that you have to use, that you don't get to use anymore. And so timeouts to me, I'm of the belief that I never call a perfect timeout at the end of the game. I just have to use them. I cannot go into the end of the game

and knowing I didn't use the timeout in those scenarios, right? Because you don't know what next play is going to come. You can't predict whether you're going to land inbounds or not in bounds and things like that. But there's so many different aspects that teams approach it. There's a lot of teams that approach timeouts as a way to say, like you said, we have to use them. There's other ones that say we only want to use them in the perfect scenario. And it's funny that analytics can get you to the same spot on how you're going to use them. You can utilize it to get to a better play, like using it to draw people off sides, different aspects of it.

I find that remembering that this is a game and that there are resources and everyone has their own way to try and game it and maximize it. It's kind of what makes it fun is we can do different things within the same game framework. So getting to the third bucket here, just player acquisition and resource allocation. On the college scouting side of things, what are some of the most interesting applications or insights you've gotten from talking to people with teams about how they're trying to use data and the information they have to kind of better maximize. their college scouting process. So they don't get GPS data as a resource from gameplay. There is no game tracking data that they can pull from other than pulling from video. So then they have to then either do they want to pull from TV copy video or do they want

to pull from the All 22 to be able to then do video computer vision tracking, which is not a one-to-one reflection. You try and make it a one-to-one reflection of what you have from. the LPS tracking data they have in the NFL during the game or their practice GPS data or LPS data, right? And the difference between GPS and LPS is they get used interchangeably a lot, but in-game LPS data is location positioning system versus the global positioning system. So it's much better data you can pull from the in-game. It's down to, you know, centimeters in-game versus you can be off if you've ever been using your GPS on your phone before you can be switching back and forth. they don't have that.

The best teams that are using analytics from college is they're using, they have either a computer vision resource that's able to pull it from game film or all 22 or they're able to then ask those coaches, can we get your practice, your practice GPS data? Because what used to be, oh, everyone wants the 4-4 guy. Everyone's not looking for the 20 mile per hour running. Because now we can do the thing of, oh, Jerry Rice ran a 4-6 at the combine, but he had game speed. you get that practice GPS data and you really know game speed from these college players or you're pulling it from player tracking on the computer vision aspect.

You can try and pull what a player's true game speed would be versus, hey, either they had, if they didn't go to a bowl game, they'd have four months to train for the combine. If they did go to a bowl game, they only have three weeks. So all those Georgia and Alabama guys are at disadvantaged because they're always playing on a national championship. So there's different aspects of how you can now get a true value of that player. and see how they're performing by using either asking for that tracking data from the team or computer vision. When we see this, and I don't mean to take shots here, but when we see this on television broadcasts,

a lot of the time it is top speed. Like, top speed is what we get when it's player tracking data in the GPS. That, it's much more nuanced when the teams are looking at it from the college side of it. And it's, the top speed is not the most important thing when you're thinking about what a receiver can do in an NFL field or what a receiver can do in a football field. but the data that they now have, based on some of this computer vision stuff, they can capture acceleration. They can capture how receivers change their tempos and speeds. They can capture change of direction numbers now in ways they never could before. So speed is a huge component of this, but you can get a ton of insights on the ways that guys move. And more importantly, you can get a ton

of insights on the ways that guys move compared to everybody else. What this allows you to do with these sorts of numbers allow you to do is it allows you to look at every player in the country immediately. You can scout every single guy immediately and not have to watch someone or have someone at the game to watch this player or even watching tape. So you can get a sense of, well, this guy from this small school just has these crazy change of direction numbers. Like, are we even paying attention to this person? So it just allows you to kind of unearth some of these little nuggets about players that maybe you never would have gotten otherwise that kind of send you back to the tape in ways that you wouldn't have 10, 20 years ago before this data was available.

You bring up a great point. Eric Galco ran personnel for me at the XFL back in 2020. And I picked him because he really focused on analytics and having a reflective process between analytics and tape. Tape would then, if you looked a guy on tape, you had to drive back to the analytics. And if you had a mismatching grade on those analytics versus the tape, then you had to have the team-wide discussion.

Otherwise, analytics can unearth certain people for you to them look at and see if that player, why is that player's film grade so bad? And a lot of times it was he's out of position. Or he can try this. Or he's able to maximize these parts of the game, but he can't process the game fast enough. And so there's things that you can now use of a cross-check to say this whole personnel process is a flawed system, but you can now refine and have at least more backing in how you're making the decision, you can process through both analytics and film combining the two together. If you're thinking about models for evaluating college players, and I think within position,

it's much easier than it is a cross position at this point. I think one of the holy grail things is getting drilling down to like an actual number for each individual player. And there are teams that are working on that where it's like, okay, we have baseball war now. And PFF has even been public about they're chasing that. They have a PFF war that they use. teams are also working on that. But I think within position groups modeling for pre-draft evaluations and modeling prospects,

that's something that is happening. I mean, you're using a ton of different inputs for each individual position being like, all right, how are we going to evaluate X offensive tackle versus Y offensive tackle? And we're looking at every single snap. We're looking at every single rep. There are qualitative evaluations that are becoming quantitative and kind of the same way the PFF are doing. But you're also looking at the level of competition that they're planning.

what kind of prospect they are, what sort of offense are they playing in? So I think there are just so many different inputs that teams are using right now, but they're trying to get to a place where you can use a lot of these pre-draft models to properly compare players within positions. Yeah, I am very partial to the NGS draft score developed by Mike Band and team at NextGen Stance and their ability to blend threshold-based analytics on all players. So Anthony Richardson and his athletic score versus production score and then his blended score. In their athletic score, if he runs, they've identified that a 4-5 is a threshold.

Anything below a 4-5 no longer puts you in, or it's the same calculated advantage that you're gained. Right. So a 4-4-4-29, 4-21, at quarterback position, it doesn't matter. We've seen from the data. And they let the data and the model tell them what are each thresholds needed to be met from a certain athleticism by position. You're not just saying you're not looking at a 4-4 in vacuum or a 429. We're saying we're letting the data and the gameplay tell us which ones have converted

best to success on the field. And so I love what they've done there. Then combining it with there is a production score. How well did you do? How good was your college team? What was your team's record? How well did you? And then on top of that, they've included human intervention, which I was like, why would

you ever do that. But they don't understand, okay, there is some aspect to understanding like Ben Robinson with grinding the mucks of understanding where should players get drafted. That's, there are a perceived value in the draft still has a benefit by you're now saying the hive mind of analytics is able to now see how good a player is. And so there are three separate inputs looking at there. There are combine athleticism looking at how well they played in college. And then what is the general feel of people around the league on the value of this player, all three of those numbers come together to give you a projection on what this player might be.

And you now have three different aspects that are the different nature of analytics that are combining together. And that's why I think the NGS draft score does a great job putting all of it in one tight package. And you can go down even further than that. When you're looking at the production, it's hard to compare production across conference, across different divisions of college football. So you have teams that are literally trying to figure out and model the best way to figure out

the level of competition of players playing against at any given time. I mean, that's how granular a lot of this stuff is getting. It's not perfect. It's far from perfect. And it's just one more tool. But these are the things that teams are actually working on right now when it comes to projecting college players. My takeaway, based on some of these discussions, is that a lot of the tracking data and a

lot of the data period was much more efficient and effective at helping with professional decisions and professional personnel moves because it's all controlled. We know what the competition is. There's so many different inputs. So one thing that came up consistently is that if you're looking at tracking data specifically in applications of tracking data, it's very useful when you're looking at where a cliff might exist for a veteran player. If you're looking at a veteran pass rush or a veteran receiver, DeAndre Hopkins, for example, you can see what his acceleration and speed numbers look like now compared to what they looked like when he was one of the elite players in the league a couple of years ago.

So those insights can go so far beyond what we could previously do that I think it's going to lead teams to make smarter and more efficient decisions in this area that they couldn't do before. It's not great for players, but I do think it's going to help teams kind of be just more mindful when it comes to these sorts of decisions for players later in their careers. Absolutely. your play caller series was awesome identifying how coaches will put players in their best position versus trying to fit them into their own system.

And now you can see, okay, D'Andre Hopkins still has, his hand size has not changed, right? But maybe his ability to get open has. Okay, maybe he's more of a red zone threat now for us. We're not going to ask him to do the same things. So a coach armed with that information about what makes this guy still special, what makes him want a potential Hall of Famer, let's use that part of his game that has not gone away, but avoid some of the pitfalls that might come.

If you ask a guy to be the same guy he's been his whole career, you are going to have a bad time. And I think that the data will now help inform coaches that they might have been able to perceive themselves, but now help make them an informed, confident decision or how to utilize their players. And I think that also you mentioned getting open and those kinds of things, teams have insight into that as well.

I mean, teams have their own models for how open is a player based on the sort of coverage that's being played, how open is a player based on what he... So just getting it out a little bit, because I was very curious about this when I was talking to people. Some of the separation data, I feel like, is very noisy and messy when we're looking at it, that stuff that's publicly available.

So I was like, all right, well, depending on the coverage, the separation is going to be different. So how do you guys control for that? And essentially, it's how open is this player based on the situation that we're putting him in? So when it's man coverage, how much more open is he compared to what we could expect with another player.

When it's zone, how much more open is he? And man is more efficient than zone and kind of figuring out what those separation metrics would look like. So these are things that teams are working on and they have ways to kind of make them more accurate. And that's more on the advanced end. Those are the teams that are really throwing resources into this. But that's something that teams can do when you're trying to make free agent decisions,

when you're trying to figure out if you should trade for a player, there are teams that have models where we can say, even if this guy's production isn't very good, he's open all the time. And they can figure that out with a good deal of confidence. without watching any tape at all. It'll bring them back to the tape, but again, it kind of streamlines the process. One of the things that I've been passionate about for a while is how do you separate the result of the play from the process of the play and what the player was asked to do? So last year, we released a feature on Prime Vision, which was called Open Receiver.

It'll have a new name coming up in the next year, but it was identifying mid-play if a player was open. And we had a lot of different places of it. We know that quarterbacks can throw tight window throws and we have a guy open. Maybe throwing a Jamar Chase with a guy drip over him is better than your fourth string-wide receiver. But we now have an in-process metric to identify was this player open based on design or his ability to get himself open? And it's a new metric that's going through the process versus almost everything that we've done right now in the play is result-oriented. So analytics has always been result-oriented. We are now working into getting more process-driven to see not just with the player the quarterback through it and the separation there, but did we get this guy open throughout the play?

Could he have run his route closer to the line of scrimmage? For the line-scrimmage, did he run into that coverage, which is now something that you have to watch for in the film room to say, okay, you know, this guy, like fans might not see this sometimes, but this guy runs a 15-yard bender, and he rolled it to 17. you might think the quarterback threw it into a spot to get the player hurt because he got closer to the player. But that wide receiver actually ran himself closer. He ran the wrong route there. But open receiver, different metrics that we're putting process base will now be able to set, show exactly our players are in the right position to be open in different aspects of the in-play metrics, not just end-to-play. So I wanted to get into that. We just spent a lot of time talking about what is theoretically.

possible now in some of the applications of this data. But I also want to talk about what teams are chasing. What are kind of the things on the horizon the teams are trying to do? And that idea of how can we kind of extricate into components of the play and decide what is more important? How can we look at a play with a quarterback and receiver and figure out who's more responsible for the success of this play? How can we look at offensive line play and say, okay, if we isolate this guy from the

rest of everything that's going on around him, how can we figure out exactly the role he has in past protection. I think that is something that a lot of teams are chasing because the NFL and football in general, it's really, really hard to do that. There are 22 players and there are endless kind of bullet points of context that you have to think about. So trying to develop models or ways to isolate the impact of individual players and individual positions is something that I think teams have a hard time doing now, but they're trying to devise ways to be better about that. Absolutely. I think the biggest one for me on the defensive line is everyone knows the best stat right now is not sacks, but pressure. How much pressure are you putting in a quarterback? But a defensive end who breaks contained and cuts inside to break his responsibility and cause pressure that way is very different than a guy going up the middle. You talked about with Mitchell Schwartz, how you generate pressure from inside, how different that is from outside. But if you're cut inside and you lose your gap, that's an easy. 10-yard scrimble, that's a first down that you might have created.

So identifying the different aspects of pressure and going more so nuanced part of the play. O-line has been a golden goose for analytics forever because it's one of the hardest positions of fine is the most amount of players on the field at a single time is offensive line, but we don't have results-based metrics for it because there's no calculated stats. You can't just look at your box score and identify ways to do it. And that's where the people are chasing is how do we get better at understanding an offensive line? So I was surprised in some of the conversations that I had that there are people within buildings that are actually more confident about their ability to use data to evaluate offensive linemen than they are for some other positions because there is a data point on every single play for an offensive lineman. An offensive lineman has to stop someone on every single passing play.

So you have a qualitative bit of information that can become quantitative on every single play for an offensive lineman, even if there are no counting stats. So for teams that are doing this in a way, we're like, all right, if I can give a plus one or a minus one to a player on every single play, and we can use those qualitative inputs to create data, they feel more confident about their ability to assess offensive alignment in that way because it's happening on every given play compared to, let's say, a safety. Who know, safety, you can go through 20 plays, and you wouldn't know to give a plus or a minus to that player

because they're not really doing anything. They're kind of in a quarter's drop and they're not really involved in a play. So that was one of the things that I was a little bit surprised about is that some of the teams that are getting into this world where they're using qualitative analysis and trying to make it data actually feel better about their ability to assess players closer to the line of scrimmage and closer to the ball where my previous thought was it's much easier to evaluate players that are further away because their play is isolated from the guys around them. That makes 100% sense. The issue that you have, though, is the schematic differences and coaching points for offensive linemen. across the board. Oh, that's what, those are walls that are going to be very hard to get over, no matter what you're doing.

I'll give an example. You know, how do you assess Tyler Linderbaum, who I thought should have won the Heisman when he was playing center at Iowa? And he, because I've never seen a center be able to reach a three technique like he did at the college level, right? He was doing Jason, Kelsey level stuff. Then he gets to Greg Roman and they're running duo and peasant. power gap scheme at the Ravens.

So how do you now evaluate that player? He's got plus minuses, but he's being asked to do very different things. Versus, if you're at the wide receiver position, run on the nine rounds, run on the nine rounds. You know, and it's a very different part of the game, and there's less nuance there. So I'm all for qualitative research getting involved, but I'd love to see some of these teams who feel confident being able to transition players out of the offensive lines from spot to spot.

Maybe it's because I played offensive wine, but that's where I find the nuance matters there. Yeah, you want to give those guys as much credit as possible. It's much harder and much more nuance than the people from the outside could possibly understand. So one of the other things, just the last thing I wanted to mention on this, and what can teams try to accomplish and what kind of is on the horizon, when the tracking data, when I heard about it, that it was going to be available, my first thought is how can you use this to kind of map out structures? Like, how can you use this to think about the distribution?

of players on a given pass play and whether or not that's efficient versus certain coverages. And that is becoming closer to a reality, I think, for some teams, where there's going to be a time maybe in the next four or five years where you can figure out this is, this is how we should distribute receivers against this coverage consistently in order to create the maximum efficiency within our passing offense. It's difficult now because defenses aren't simplistic enough to have that information. Even if you have a team that's playing a ton of cover three, the ways that they do it are different, the ways that they teach it are different.

So that's something I think may eventually be possible, but is not possible right now and that teams are chasing in this moment. Yeah, I think what's going to happen is the need for diversity and understanding that like it's called Goddard's law. You cannot, once you make the statistic, the goal, the statistic gets worse. and so if you're going to look at the data and I would venture to say running four verts every play might be the most efficient thing talk to any DB about it just run four verts

and we know that's not true and so it's being able to say looking at the data and understanding there is going to be some nuance involved to how we can process this information and be able to put it in the right spot to get us those places I think because

then there's the other aspect of distribution of players is I grew up in a system where players ran their routes and there was a lot less option routes. There are, there's the, I learned the run and shoot from June Jones and AJ Smith, and every plays an option route. And there's aspects of that part of the game in distribution. And then probably the most important aspect that I want, that these models need to understand is right now in football, one guy throws the ball and one guy catches the ball in place.

we don't utilize the offload hand, the Kevin Kelly, the rugby style hook and ladder as often. And that's an underutilized rule in the game that I wonder if would that be an aspect of the game? How do we distribute to manage offloads better in the game? There are different aspects of the game that need to be accounted for in the rulebook as we advanced this technology forward. That's going to be one of the biggest things I think you have to convince coaches of. What we're talking about incorporating more of that stuff. It's like, wait, you want to do what?

You want to use a hook and ladder plays in the middle of the game? You're going to have had your, this is one of my favorite things about pistol. If you describe the pistol offense this way, you're going to have the quarterback turn the back to half the field to be able to do a vertical handoff. And then you're going to ask him to run at an open side d end when he doesn't have to run lateral. He just has to run up the field. When you describe it that way, it seems ridiculous. But if Chris Alt hadn't run it years ago in Nevada, it wouldn't be a base offensive play for, half the teams in the NFL.

And so there are some crazy things. Rich Rod invented the zone read off of a blown play by his high school quarterback, right? Like there are some parts of the game that seem crazy until they become all of a sudden mainstays in our game. The last thing I want to talk about is just kind of, you know, misconceptions that maybe the public has about some of this stuff, but also advice that you would have for teams and the way that they should operate.

So on a very simple level, what do you think the public kind of gets most wrong about what's happening within these departments in the NFL? That they're not huge football fans and lovers of football and purveyors of football and their outsiders coming in to change the game. They are people that have a passion for football and their way to get into the building oftentimes was through analytics. That was their superpower. Everyone at every part of the team has a job to do and their jobs to make the team more

informed and provide insights and ways to make the team better. And these are football fans. These are not people who hate football. They don't want to ruin your football. And I think that's a misconception that people think there's no thought process behind a lot of these things. They're just trying to change the game. My biggest issue sometimes is when people say, I don't know why they didn't go forth to them. I don't know why I want to.

And if I hear them say analytics, that's a cop out. Or that analytics has ruined the running back market. Just blame analytics departments? You're not a huge fan of that framing? No. I don't think that other positions are upset about the running back market when the teams when they were making so much money.

So I think that the problem is that these people love football. Every single analytics department, every single person I talk to loves football and they want to find a way to make their team better and to make your team win games. They are just like coaches. They love being around the team. I think it's a misconception that it's complete outsiders coming in

and they're not complete outsiders. And just because you've never put your hand in the dirt, doesn't mean that you can't help inform a team success. I think that there needs to be an understanding about you're asking guys that is a physical game. So I think every analytics person should spend time at practice and see these guys in what they do. Because, again, you can't ask them to call pass play every time

or else they're on their heels. It gets back to what we're talking about, where you need them to be integrated into what you're doing, not only for the ideas to get across, but for their understanding and the way that everything operates to be more nuanced than just better than it would be if they were off in some corner somewhere and some dark office, never actually interfacing with the people

who are doing the grunt work on the field. Right. And they love the game. Don't think that these people are outsiders. I think that's my big issue I see teams have, or fans have, it's like they're coming to ruin the game. That's not the case.

I mean, just the idea that it would be analytics departments that have completely tanked the running back market. I think anyone who's listened to this conversation about how hard it is for the people within these departments to actually get these ideas across to coaches general managers, the people that are actually making these decisions would understand that analytics departments don't wield that much power. They can't change the market for an entire position because most general managers who are the ones ultimately making these choices about how to pay running backs aren't listening to their analytics departments enough for them to have that much influence. I think I want to go back to the part where a lot of these guys need to be historians of the game, and people think that it's just the coaches. The rule changes in the early aughts that made it easier to throw the ball

is the reason why 20 years later the running buck market has tanked. The rule changes about how, so now you can't run the ball. Now the rule changes in 2011 draft to limit salaries for rookies. that has also affected the earning potential because early drafted rookies can't make as much money from their first round pick. So these are aspects of the game that have tainted. It's not just analytics departments identifying that there's a lot of good running backs available in different parts of the market. The one other thing that I want to mention before we get out of here is just this idea of what analytics departments within NFL teams can learn from other sports. And one interesting hire that came up this week is the Cowboys who have completely retooled their analytics.

Department. John Park, who was with the Colts, is now overseeing their group, and I think they're four or five deep now as they've retooled it. They hired Bryant Davis, who is a research analyst for the Cowboys, and he came from the race. So this is just another example of kind of teams looking a little bit outside of their own world for different ways, different strategies, different thought processes that might be able to kind of maximize the efficiency that's happening in some of these departments. Yeah, we are in the infancy. That is what one analytics department had told me right now. So whenever I talk to teams, I ask them, what's next, what's next?

And one was, how can we mimic everything baseball does? Yeah. Right? I think teams would really like the Mets 40-man team and then the extra hedge fund that they have that's doing computation. I don't think they want the results of the Mets right now. But I think everyone wants that size department or if you're in the analytics department to get that many jobs to people. But it's how do we steal from baseball?

That's the most nuance. For me, it was when I was looking at health and safety as a big aspect of how we can utilize analytics to make players of the XFL on the field for longer, Australia was the best at sports science. How do I just do one thing as good as them? How do I do load management as good as Australian rules football? Just to do one thing like them. And so I was looking at different departments and identifying where do other places have

efficiencies, even the financial market for the salary cap. How does financial markets waste certain things in ways? How do we speak that language within our own market on the football field to help us be better because we know that's a more mature market. So we are in the infancy, whether you're pulling from somewhere, diversity of thought is so important in these departments because to think you're ever right and know that you have the secret, that's how you get beat. So be able to pull from different sports, from different businesses. that's how you get better in these analytics departments, and it's going to have impact on teams and how they approach the game moving forward.

All right, one more thing before we get out of here, that we didn't hit that much but want to. Just tell me a little bit more about how these people that are in quote-unquote analytics departments in the league are impacting the way the teams are thinking about resource allocation and kind of the salary cap in general. The salary cap is one of the more fascinating things as we were looking at projects that we're doing on Prime Vision

in different ways that we could help integrate. get more informed on how the salary cap works is how diverse people look at the salary cap. Some teams use their analytics department and see there's three players. There's rookies, there's vet minimums, and then there's super high paid players. And then there's some players that are looking, or some teams that look at players as they are, you have to include what draft pick was used on them, what would their hit be against other players at the same market, what do we see from the college ranks coming in to then get a blended score? Some teams want to make it as simple as possible.

Some people want to make it as complicated as possible. All with the same data set. I think that goes back to what we talked originally of. How do we get the most impact to coaches? How can I get a coach to digest this information or a general manager to digest this information? If they need the more nuanced diversion, if that's how they're going to process it, your team will create that. If your general manager just wants the bare bones, let's create three categories of players. That's another route to go.

but it's interesting to know that within departments, no matter the size of the apartment, how different people can look at how to structure player value from salary cap information. So what sort of, so you mentioned that like how much it would be compared to a different player at his position. Like what are some of the other like bits of granularity that go beyond just guys on extensions? Like what other considerations go into some of these conversations? Draft capital. And then, and then potential draft capital used or lost or letting me,

letting a player walk and the compensatory pick is all involved into how you're evaluating that player's value to your team compared to the rest of the market. It's funny because there's so much thought about, okay, once the ball's kicked off, it doesn't matter. But there are teams and departments that are looking to try and find the marginal wins of how they develop their roster at every possible spot. And it has to do with how they would affect their game plan on their team. what are upcoming things as you're making in-season trades and in-season signings, your upcoming opponent schedule on what you might utilize versus what you used in your previous

schedule has an impact as well as do we have the ability to do void contracts with this player? Does our ownership allow us or even possibly getting a new stadium and having new revenue coming in? Does that change how you're processing, how your team can use cash versus using salary cap? That's all factored in. and now your team has to play within that set of rules. Are you doing Stan Cronky void years, or are you doing your first ever restructure in the history of your franchise and the Bengals?

Right? Both teams in the Super Bowl two years ago. Who was that? Who did they restructure? Mixing. Interesting. Yeah, there's not a lot of that in Cincinnati. But some of these teams, I think you could probably guess. You look at what the Ravens do in terms of how they're making player acquisition decisions

and all of different compensatory pick considerations. The Eagles have a million different ways that they structure contracts. So again, a lot of the teams that are doing this stuff and have a little bit more depth and breadth to the way that they're approaching this, I think that you see a little bit more complexity with how they're making some of these decisions. And it's fascinating because money ball originally was how do we make the team who's not spending compete in baseball, but it's being used in both capacities.

There's a way, how do we make compete when we have less money, but also the wealthier teams, how do we compete and utilize our bankroll as our advantage? against some of the teams that don't have the same bank goal that we do. But usually as in the builders have more cash out than having no wait for salary or season ticket money to come in to be able to pay your bonuses. Yeah, it's a good reminder that not everyone is operating from the same place. And not just because they don't have the resources, but because their owners want to operate in a different way. You know, the Cowboys theoretically have endless pools of resources if they wanted them. But they're a team that I think is probably a little bit behind the times in the way that they've done this.

And they revamped their entire department this offseason because I think they want more returns from this exact world than they've been getting in the past. But you have teams like the Browns, teams like the Eagles, teams that maybe aren't necessarily considered big market teams or teams that are going to throw a lot of money around. And they're throwing a ton of resources at this problem. And John Park is, that's my BIC. He's one of the best in the business. I really understand football. He coached football. He has a history with football.

And I think there's going to be a lot more people that have played football, that have coached football that get into. these analytics departments, I think it's going to pay dividends to how people will start be accepting of some of the information they receive. That's exactly. And it brings this full circle because that's how you get those ideas across. If you can speak the language and you kind of have and that's, you know, maybe a problem just because it cuts off certain types of people in these roles. But if you can speak the language and you have instant credibility when you're having these discussions with general managers, coaches, it helps so much for them to embrace these ideas. And that's the battle. It's not about having the idea. It's not about having the data.

It's can you explain them or convey them in a way that they're going to be actionable for the people who actually make these decisions? Absolutely. Sam, very much appreciate the time, sir. Always good to chat with you. Thank you for the insight. I think it's a fascinating conversation, a fascinating topic, one that we don't consider often enough. I put myself at the front of that line. And that's why I really wanted to dig into this today. So thank you very much for taking the time, my friend. Awesome. Let's play some football. It's football season, baby. Camp starts very, very soon. We have really kind of one more show before we get into camp. Our good buddy, Connor, or from Sports Illustrated, is going to be joining us tomorrow to chat about some of the news that's happened this week, but also the next wave of coaches that we could be looking at in the NFL. Who are the assistants to keep an eye on as we report for camp and as we get rolling here?

But after that, man, it's training camp time. I'll be on the road starting on Sunday, heading to the West Coast first. Be away from home for a good long while as I start visiting with everyone very, very excited about it. For now, though, that is all we have. We will be back tomorrow. Appreciate you guys listening. We'll talk to you soon.

This was the Athletic Football Show.