The Data Stack Show - 202: Predicting the Impact of Competitive Entrants With Synthetic Controls with Evan Wimpey of Elder Research

Episode Date: August 14, 2024

Highlights from this week’s conversation include:Evan's Background and Journey in Data (0:40)Discussion on Synthetic Controls (1:04)Evan's Educational Journey and Marine Corps Experience (2:54)Joining Elder Research (4:38)Synthetic Controls Explained (6:54)Measuring Impact with Synthetic Controls (9:05)Building the Control Group (12:54)Qualitative Context in Data Analysis (14:50)Final Steps with Synthetic Controls (16:29)Client Analytics Maturity (18:56)Outsourcing Decisions in Analytics (21:09)Cohesion Between Analytics Teams (24:18)Validation of Predictive Models (26:37)Confidence in Marketing Predictions (29:01)Setting Expectations for Data Science (36:09)Evan's Background in Data Comedy (39:44)The Journey to Award-Winning Jokes (41:29)Creating New Jokes (46:22)Get the Joke Book and Final Thoughts in the Episode (48:46)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Hi, I'm Eric Dotz. And I'm John Wessel. Welcome to the Data Stack Show. The Data Stack Show is a podcast where we talk about the technical, business, and human challenges involved in data work. Join our casual conversations with innovators and data professionals to learn about new data technologies and how data teams are run at top companies. Welcome back to the show. We are here today with Evan Wimpy from Elder Research. Evan, welcome to the Data Sack Show. We are super excited to chat with you.

Awesome. Yeah. Thanks, guys. Happy to be on. Okay. Give us a quick background. Sure. So yeah, you mentioned Elder Research. It's where I'm a director of analytics. I've been there about five years doing analytics consulting work as a data scientist, and now tell another data scientist what to do. And whenever I get a chance, I'm trying to make them laugh as well, do a little data comedy. Sometimes they're at Elder Research and sometimes for anybody in the data space who could use a little bit of laugh. That's awesome, Evan. So one of the topics we spoke on before we started the show here was synthetic controls,

kind of a data science-y topic. So I'm excited to dig in more on that. Anything else you want to cover on the show? Yeah, I won't be able to stop myself from telling a couple of jokes. But yeah, synthetic controls are great. And we'll dive into some of the technical stuff. Great. Well, let's do it.

Yeah, looking forward to it. Okay, Evan, I think you are the only professional data comedian that I've met. And so, which is a special thing for the show here. We're very excited. But I'm going to give the option. Do you want to tell a couple jokes now? Or do you want to make people wait on the edge of their seat to the end of the show?

I think we want to build their hopes up all the way until the end. Definitely. Yeah, that's my vote. Let them down in one giant crash there. Okay, great. Yeah. So let's start earlier in sort of your data history. So you have two master's degrees.

Is this correct? It is. Yeah. You got to make a decision at some point. Yeah. Well, have you decided? Have you? No, I'm going for another one sometime soon.

Yeah. Triple crown. The audience will learn quickly how little I know. And they'll think, wow, you can get two master's degrees not knowing anything. What's wrong with our education system? What are the degrees in? So first master's is in applied economics, which I think is pretty close to the data and analytics space. So it's from East Carolina.

I got that one a dozen years or so ago, just before I got into the Marine Corps. So got in the Marine Corps, was a finance officer. So don't picture like kicking down doors, but more like sitting behind a computer. Kicking down spreadsheets. Yeah, exactly. Yeah. I did not have an M16. I was issued Excel spreadsheet.

That's it. Basically the same thing. Yeah. Disassemble and clean your weapon, please. Yeah. Data cleaning. Re's it. Basically the same thing. Yeah. Disassemble and clean your weapon, please. Yeah. Data cleaning is important. That was all my bootcamp. So yeah, I would say that I still use that degree some, you know, it's econometrics studying. You learn a lot about linear regression and all the assumptions behind it. You learn a lot about how people make decisions.

But I got out of the Marine Corps, was working in finance for about 10 minutes. And the place, when I was working in finance, I saw opportunities for using analytics. My wife was actually at the time in a master's program in analytics. And so I thought, wow, we could be doing this at the bank. We could be doing this at the bank. And there wasn't a whole lot of appetite for it. I said, you know what? I want to go somewhere where there is appetite for it, but I need to learn a little more. Plus I couldn't stand my wife knowing so much more than me. So I actually followed her to the same master's in analytics program at NC State. And so that was the second master's. And how does you and your wife both having a master's in analytics affect your decision making process together?

That's the big question I have here. That's a great question. When you think about different weights that you give to different input variables, her weight is a one, mine is a zero. Okay. So it's actually pretty easy from a modeling standpoint. Very simple model. Yeah. Very simple model. Parsimonious is what we are. That's great. Okay. So did you go to work for Elder Research after that degree?

I did. Yeah. So Elder Research actually recruits pretty heavily out of that program at NC State. My boss is a graduate. The CEO of the company is a graduate. And so when I was there, those are clarify real quick. Elder research is like an analytics consulting firm. They don't do elder research. They're just founded by a guy named John Elder. If you haven't heard John Elder, as soon as this episode's over, go and read John Elder

and read his Elder and read his books and listen to his talks. He's great. He really motivated me when he came and spoke to our graduating class. So yeah, I was really excited to get a foot in the door with Elder and I've been there for a little over five years since then. And what is, so analytics is super broad. What does Elder, do you have a specialties at Elder or? Yeah, this turns into a long answer. Analytics is super broad. Elder research is also very broad. So when I talk to the consulting friends, like management consulting, I say we're very specific, we're very focused. But when I talk to data and analytics people, we're really generalist within data and analytics. So if

there's a specialty, we have a pretty big footprint with US government, Department of Defense. So that turns into a lot of, if you can believe it, fraud detection, threat detection, network analysis. So I think that's where we probably got the biggest start and the biggest growth 25-ish or so years ago. I don't work with government clients. They have higher standards. So I work with commercial. Is that quality or security risk? That's security.

Look, that was a long time ago, okay? Yeah, so I work with commercial clients, and there it's very broad on the types of problems we tackle. So you think of any big corporation, sometimes small startup, more often bigger fortune thousand companies that have marketing teams that need to know who to market to and how to market to them. They have operations and logistics that are building things, moving them around. So they have a lot of data and they need to make a lot of decisions. And so we sort of help with that. Cool. Well, John, you were really,

I think we should just go for the jugular and bite into the juiciest topic. And I cannot wait for this topic, but I think you're more excited than me. So the data jokes? No, well, we know we're waiting to the end, but of course that's. No, yeah.

Let's, I want to dig into the synthetic controls. We talked about this a little bit before the show, but I was just sharing with Evan. I was on vacation recently, got off on an exit, middle of nowhere. There's a gas station under construction, and then there's one that looks like it's been there for 20 years. Right next to each other? Yeah, right next to each other.

You're making the right turn to get it to you. Yeah, right. Same exits. They're on the same side of the road, and let's say there's less than a quarter mile between them, so pretty much next to each other. So you go, and then the gas station, it's doing great. The cars are full, the pumps are full, people are going in and out.

It's doing great. But you walk into it, it's kind of dirty. You walk in, it's kind of got that this has been around for 20 years like not really kept up the restrooms are gross like that type of thing right and then my driving back out pay more attention to the new one so it's a quick trip qt which is popular in this area and they kind of have this business model it's like really always really fast like you keep it clean they've got like so clean yeah unreal they've got like a kitchen you know they like full food service like whole deal just a kind of a different experience so i'm thinking in my mind from a data background sure that general manager or whoever's owns that that particular location like knows that it's

coming they know that a the other place has the strategic location. It's just slightly closer, right to the interstate exit. Yeah. But bigger, they know like they have that gut feeling of like, man, things are going to get, things are going to get bad for us,

but there's no, they have no way of quantifying that. And let's say that like that general manager is like, man, I wanted $250,000, $500,000 worth of improvements to try to compete. Or maybe we should go ahead and shut this place down instead of letting it bleed out for a year or two. But there's no way to quantify that. I

highly doubt either anybody there is trying to quantify it. So there's the background. We were talking about synthetic controls and I'd love to hear pass it to you. How would you handle that situation or a similar situation? Yeah, I think that's great. And I, John, I sort of share that with you. You just think about things in terms of data and analytics and what it would mean for data and analytics

out in the real world. So I loved it. That's what came to your mind in this dingy old website. We put like the bars over the window. Yeah, you're taking your kid in here. Like my wife is not going to like, if I told her how dirty this was, where I'm taking my kid to the bathroom and you're like, how could

you quantify the potential failure here? Like you walk in and you walk in and you're like, don't touch anything. Yeah, totally. Yeah. Yeah. The young kids don't, don't touch that. Don't touch. Nope.

We got hands and a thresher in the car. Don't use that. Exactly. Yeah. Yeah. Right. Yeah. So I'll even take it one step forward before I set up synthetic controls, but

let's say the quick trip opens and then you've got six months worth of data in that new gas station. And even then it's really hard to quantify what happened because you know, almost certainly there's going to be an impact of the quick trip opening. In this case, it's like, this sounds like a very extreme scenario where, okay, sales are going to plummet because everybody's going to go to the nice lane quick trip. But what if there were already five gas stations there and now they're opening up? It's really hard to measure even after the fact what happened. And so this is where synthetic controls, we've used this with a few clients at Elder Research

and it's been around for several years, but I've only heard of it for about a year ago. I'm pretty slow, but man, it's been super powerful and we've loved it. So I'll try to sort of set it up where in the context of this gas station, where if it's just a single gas station, there's not a lot you can do, but let's say it's a franchise that has a few hundred locations. Well, ideally, when you want to measure the impact of something, you've got a test and a control, but you can't do that in this case. You can't have a test scenario where you say, okay, open the quick trip.

Now go back in time and don't open the quick trip. And then we can measure what the impact was. But what you can do is try to synthesize that control. You're going to have the impact was. But what you can do is try to synthesize that control. You're going to have the test scenario. And if you've got a hundred other, maybe not even a hundred, if you've got 10,000 other gas stations, then you can find some combination of those gas stations that look similar, have similar demographic, have similar sales numbers, same location or whatever, similar location. And you can say, whatever. These stores, A, B, C, and D, they map maybe with different weights, maybe with nonlinearities, but they can

predict pretty well this store E. So they can tell us this is what if we can predict store E based on these other stores. And these other stores are not going to have a quick trip open right next door at the same time. So you're synthesizing a new theoretical store from a combination of other stores that would closely map this one that's about to be impacted. Exactly. Spot on. And then as long as you've got some historical data, you can go and validate those stores to say, hey, does it actually work? You know, you can train it and then validate it on out of sample before the test event. So you can say, Hey, it does a pretty good job. It does a very close job of mapping what this gas station in question is. One, one question quickly. So how intensive

is the process of building the control group actually synthesizing that, right? Especially so you mentioned, you know, if it's 100 stores, and you know, we'll go back to the kicking down spreadsheets thing. If it's 100 stores, and I have them listed, and there's, you know, maybe 50 columns of data in an Excel sheet, right? You can sit down in an afternoon and like, do a human synthesis of like, okay, I can sort of filter down to the list I think that's good. If you have 10,000 stores, what kind of process are you using to try to build the synthesis control? Yeah, that's a great question.

And I think the biggest risk in this is if you've got 10,000 stores, you almost certainly can fit training data perfect on that 10,000 and first stores. You can find some combination that make it perfect. And so you've got to be careful. And this is where I think there's some technical aspects to this, where you've got a holdout set that's still in the past, but there's some human element to this as well, where you're not going to choose a store in New York to map to the store, to this gas station in South Carolina, where the roads are very different.

The competitors are very different. Even if the sales numbers are perfect and you can map the sales exactly, and that's ultimately what you care about. There's some qualitative sort of feature engineering that's going to go into, okay, the market conditions are going to be so differently in this location or this type of store than it is for this store that we care about. But ultimately it turns into basically a feature engineering where you're trying to model this new store and you've got 10,000 features that are all your old stores or all of the other stores. And as you go through this process, sorry, I want to keep going, but this is just so interesting to me, like the source selection. How much time do you spend studying the industry, studying those sorts of contextual aspects, like you said, right? Okay, well, you can write a model to go find the perfect fit but it's not

gonna it's gonna lack the context that would actually make the synthetic control that would imbue it with its deepest levels of meaning right so are you spending a lot of time studying because you haven't i'm assuming you haven't you know managed many gas stations i've been too long so is there a lot. like elder versus a SAS or a product firm that is going to come in and say, Hey, here's a solution that you can plug in and do this versus we're going to come in and we're going to charge you money, but we're going to charge you time too, because we're going to ask a lot of questions. We're going to ask what makes this store this.

We're going to try to talk to store operators. We're certainly going to talk to the people who are forecasting the sales or the impact or the cases. So, you know, a lot of that is, a lot of that you can do almost like in an academic setting where you're doing research to find what you can about managing a gas station. But much more of that is, John, to your point, if you talked to the manager of that gas station, he or she knows that the quick trip is coming and they know what is about to happen.

And they talk to their customers all the time. And the context that they have is what you want for that qualitative context that you need when you're building a tool. And depending on the company and industry, they may even talk to other operators and have a good idea of like, oh, like, yeah, I typically trend with like this other store. Some of them even have an idea with stuff like that. Yeah, exactly. Yeah. Very spot on. Okay. So I derailed us for a minute there. Okay. So we have 10,000 stores. We've dug in from a qualitative standpoint. We have a model that takes that context. We have a good fit.

And so now we have our synthetic control. So where do we go from there? Yeah. So now we're able, now it doesn't necessarily answer the question of what happens when a quick trip opens right next door. If we've never observed that in the past, then it's going to be hard, but it lets us measure that in the future. There's almost, you know, this gas stations, aren't a new industry.

New ones have opened up next to old grungy ones all the time. So probably we've got something close to that has happened in the past. And previously there's no counterfactual. So we can't measure what the impact was. We can say, oh, look, sales kept going up after that. Yeah. But maybe they would have gone up way more if the quick trip didn't. So now we've got this in the past, we can say when a gas station that meets some sort of

objective, you know, some quantitative criteria, like new build in the last six months is, you know, top 10 franchise for gas stations and you know, whatever the criteria is, here's what we have seen as an impact in the past. And that gives you, hopefully with some uncertainty bounds, which this method is pretty good because as long as your model has some prediction interval or confidence window, then you can be pretty confident, hey, it's going to impact sales directionally. And here's some sort of measured interval of what we think the impact will be on sales or number of customers or whatever it is. Yeah. I don't want to get too specific on the gas station example is so Eller's got a pretty big footprint in like consumer packaged goods and the hospitality space where there are a lot of franchises.

There are a lot of products that have competing products. And so that makes a great use case for synthetic controls because out of all the tens of thousands of products that some manufacturer makes and puts on a shelf at Walmart, you can see what happens when a Walmart shuts or when they expand a parking lot or when they put a feature promotion at the end of an aisle. So there's all these events that take place and synthetic controls gives you a way to sort of retroactively measure what the impact of those events were. Yeah. When you start one of these projects, do your clients generally have a really good idea of the question that they want to answer? Or is it more general along the lines of, we want to understand the risk factors for this product line. Is it more general or is it more

specific? Yeah, it sort of spans the gamut. I'm trying to debate if I have a favorite and maybe I like the mix and that's why I like being in the consulting industry. Yeah. But you know, we've got some clients that have good data and they've got a prioritized backlog of analytics projects and they run agile and they work with their IT and they deploy great stuff and they know, hey, we want to measure the impact of new store openings on our stores.

And maybe they've never heard of synthetic controls, but they know exactly what they're trying to measure and they're asking us to do it. That's more rare than... I would suspect that. That's what I was going to say. And how often does that happen? Very small sample size. So, you know, not meaning that this generalizes everywhere.

We work with some pretty analytically mature clients, which would surprise me a little bit because you'd think they're analytically mature. They've got their own data scientists and engineers to do that. But having an outside perspective come in can often be helpful. But oftentimes it's more, hey, the general manager at this gas station has been complaining that this QT is about to put them out of business and we don't know what to do. We don't know what's going to happen. Can you help us figure out what's going to happen?

And then that usually doesn't start with us coming and says, yes, we'll build synthetic controls for you and we'll quantify with uncertain estimates. So no. Okay. Well, what are you trying to do? What are your goals? Very sure. Yeah. And it becomes a much more consultative problem

than a technical problem. I was actually going to ask a question to both of you as consultants. And this is like a... Yeah, this is... I'm genuinely interested. So let's take a franchise, for example. We'll just keep it the gas station thing, right?

But let's say it's a big, you know, national franchise of gas stations. And so, of course, they have a lot of analytical horsepower at the's huge, you know, the firepower is huge, right? And so you would think, you know, okay, wow, we actually do have a ton of analytical horsepower, right? But in every company, there's a point at which an internal resource specializing in something really specific, it means that you're not working on the actual product anymore, right? So now your analytics team is, you know, sort of embarking on something that is helpful to the business, but is, you know, sort of divergent from the core work that's actually, you know, sort of driving it day to day necessarily, right? To answer various questions. How do you think about when to make that decision? And as you think about your clients, right? Like,

what should you keep in house? And what are the situations that you need to outsource? So I think the most interesting thing with large clients, say I've got 500 horsepower and that could be 500 people or some other unit of whatever, but 500 horsepower. And it's all allocated some of it specifically to like, like we've talked before with somebody that was doing forecasting for a call center for benefits for i think it's for amazon oh right very very specific

like in amazon's case so but it's all deployed some of us deployed specifically to marketing or finance or maybe even a subset of hr or whatever and then there's some that may be under more generic corporate roles that do like maybe they do data engineering or maybe they do more generic, they do reporting to roll up to investors or the board, like more generic things. Yeah, executive BI. Yeah, right. But if you had a project that kind of spans between those groups, A, there's going to be gaps in knowledge between the groups

even if they have the right skill sets and b as far as redeploying horsepower like most people do not have flexible resources even if they have the right resources so that that's what it's like well you know it's a big company like why can't they just do it it's like well this person's practically like a they're just focused on this b they need to keep doing their job that they were hired to do and c they don't have the broader context of all these moving pieces so i i think even if you technically have the horsepower redeploying it is really challenging so i don't know if you've had that same experience evan i think that's a great point i was going to mention something else and i will but i'll touch like i i think that's a great point. And it brings an example to mind, which happens all the time, but was really amplified during COVID.

So work with a bunch of like consumer goods, manufacturing groups, and they've got marketing analytics folks, and they've got operations analytics folks. And if you went shopping during the pandemic, you might have noticed the shelves weren't always full of everything. And so marketing analytics folks running specials and promotions on goods that you can't even keep on the shelf is an absolute waste. It's just a marketing waste. And so you've got your operations folks that are trying to supply the right stuff at the right time. And you've got your marketing folks that are trying to push the right products to the right people. And there's not sort of that cohesion in between them that don't market the stuff that we can't

keep on the shelves right now, or start to market the stuff that is taking up too much inventory that was not moving fast enough. And so like, that's a pretty specific example, but where these analytics folks and these analytics folks are doing great, but the outside perspective perspective it becomes easier to connect those dots yeah because they're each optimizing for their thing like i'm trying to sell more in my category like yeah i like that's how you gold me like that's where my incentives are of course i'm going to keep trying that when in actuality like you said you maybe you have not enough inventory in that category and too much in another. And yeah, so that makes a lot of sense.

I would say one, the first thing that I was going to mention that's not as good an answer is I think oftentimes, you know, this comes with, you know, I work. John Elder founded this company about 30 years ago and he's a well-regarded name in the space and a lot of times just like the change management aspect just driving some project can be really helpful to bring in an outside you know an analytics team that is peer-to-peer with a sales team that can't get traction and can't get a sales director or manager to listen and do this. But hey, we brought in these outside experts that specialize in this thing and they can do this. And now a sales director, not always, but may be more inclined to work with that or listen or institute some change. Yeah. Yeah. That makes sense. Yeah. Makes total sense. Yeah. That's always an interesting question. Build versus buy. um changing gears a little bit one of the other topics that i'm really

interested in that we discussed briefly before the show was understanding model performance right so you know it for you know it's easy for people to think about like oh there's a data scientist right so they're just gonna throw this model if you don't know a lot about the space right like you can throw this model of something and wow it does all this crazy stuff and like we get something at the end but there's this huge element of understanding whether what the machine is doing is actually working the way that you want it to and yeah i, we don't even need to get into the question of AI, right? Because it sort of highlights the insanity of what's going on. But how do you deal with that at Elder? And how do you think about that? Yeah, I would say that, you know, you asked earlier what our specialty is,

and I sort of punted and said, well, we're more generalist. I would say we've got, that's where we've built most of our credentials is on validating models, validating findings. And, you know, if anything, it's tough to talk to clients or prospective clients because the flashy stuff sells the, you know, we can forecast this. We can tell you exactly what this is going to be. What we at Elder Research likes to do is come in and say, well, this is how well we can forecast. And this is how much error you should expect.

And these are the places where it's going to generalize well and where it's not going to. And so a lot of the work that we end up doing is actually model validation for work that people have inherent teams that have inherited or have built internally. And, you know, some, sometimes that starts out of, we just want a stamp of approval. Somebody just sanity check this work, make sure it's good. But often that turns into, you know, you know, you're reporting a general performance, but you know, it's performing well in certain subsets and poorly in certain subsets, or you've not validated it well out of sample.

And it's not going to generalize because you've got a leak from the future or, you know, conditions have changed at this point. And, you know, as of this time period, it's not going to generalize well. So I think that is becoming a more important thing is there are more and more tools and vendors that are out there that you can plug in modeling and some of it's super powerful, but it's really tempting to use a lot of really powerful tools and then not know how good they're actually performing. And so we're trying to keep folks grounded in being able to measure how good a model actually performs. They're all introducing predictive models, like on churn, on predicted next order, things like that. And I have yet to see one that provides any sort of metrics around, like, with what level of confidence do we think this thing...

You know, it'll just give it. It just spits it out. Now the persona is marketing people. So from... No disrespect. I'm a former marketer. Yes, because we are easy prey i mean that's no i'm just saying that like the persona is marketing people and they're like they're not they wouldn't know what to do with like percent yeah but the

numbers are like in a sense like i understand yeah benchmarks yeah or like model like yeah like our values like that's not like what are they going to do with that? So from a product's perspective, I understand why that information would not be shown. But as more data teams get involved in this space where you get more and more complex and you end up having a warehouse more at the center of your data stack

and then you're tying in all these marketing tools, I think it's becoming more relevant. And the data teams are asking, oh, so you're sending all these emails or messages around like churn, like where did that come from? Oh, it came from this marketing tool. Okay, great. Like how do we know that's even close to being right? And the answer is like, we don't. And now, I mean, you want to trust your vendors, right? And you want to trust that somebody at one of those marketing companies validated the thing, but you don't really know. And there's at least a few of the tools where some people in the data science community

have kind of come out and be like, hey, we've done this research and these numbers are not very good. Have you ever, did you ever, did you implement any of those in a past life where you like turned something on that was predictive, but you had no idea what was going on? Yeah. Oh, for sure. So we, I mean, especially on the like email was the easiest one, right? Like I, my theory of email is, is it's about touch points. Like you, you get real, you can get really into AB testing everything. And a lot of email is like the more touch points, like the more it like converts. So it doesn't really matter what email says. So says so as a way of like yeah like let's do a campaign around like prediction next purchase so we did that and probably gained some like incremental benefit from it so from that

standpoint it's like great like who really cares yeah but once you like you know get past that like initial stages then you're trying to refine things and we don't want and then we see like maybe we're seeing higher churn because you know it's 2020 and your inbox is even more full of email marketing than it's ever been in your life right right it's like okay it's time like we're seeing this high churn we need to like cut back on these things like how do we like make this a lot more precise and then it's like well what does this model do like is it accurate like i don't know yeah know. Yeah. So yeah. Yeah. I'm going to, I'm going to mention one, one thing, a little serious and one thing, a little silly here, but like even a model that does really well, that makes point predictions is so much more limited in what it can do

than a model that gives some type of prediction interval or window. And we've done, I've done forecasting work where they've already got a model that's in place that doesn't natively have any type of prediction intervals. And if you think like if you're stocking, if you're in charge of a restaurant and making sure there are enough French fries at the restaurant and your restaurant predicts you're going to sell a hundred cases this week and the other restaurant predicts a hundred, well, you treat those the same. But if one says, you know, the 90% confidence interval is between 90 and 110 versus it's between zero and a thousand, well, that really changes how you want to manage your inventory

there. So I think having measuring uncertainty and reporting on uncertainty is not a language. It's like when you start talking about distributional estimates or distributional predictions, like people's eyes glaze over. But if you can frame it the right way, it's like you need to understand that the models doesn't know what it's talking about here, but it knows what it's talking about here.

But I think that's really useful though, because especially in that use case, if I told you in like common English, hey, you need between 95 and 102 cases of fries like that's super easy and i can make a decision right whereas like it's it's not as clear like you need 100 cases of fries with a 89 confidence right like it'd be much better like we can get to like we're almost 100 confident it is in this range and i think even that like phrasing is way more helpful to people i also what we're talking about if we're bashing marketers this is yeah we love we love bashing marketers we do it

we do it and i actually like it even more now that i'm not on the marketing team anymore i moved from marketing to product yeah recently he moved to product. So even more fun. Good promotion there. He can wash his hands. Yeah, this is, I'm hesitant to tell it because it's not even my story. I'm just borrowing it from one of my colleagues at Elder Research,

but working with a client who had segmented their, this has been several years ago, but had segmented their customers into, they call them cubes, but it's like, hey, this, you know, 18 to 24 upper middle-class suburban really likes this product. And this, you know, retired 65 plus widower really likes this type of product. And we're trying to convey like, you've not tested that at all. That could just be by chance by, you know, you break it up small enough, you're going

to find random things and they argued it and they didn't appreciate this. And no, look, the data shows that this is what it is. And they had, this was pre COVID. So it was in person and their, their floor is this black and white, like tiled checkered floor. And so the presenter was very senior. So he could get away with this, picked up the candy bowl that was on the table and just threw them all on the floor.

Okay. And went and looked and said, oh, look, the Snickers all land in the black tiles. And oh, look, the Skittles always land in the white tiles. So we should sell Skittles to the white. That is unreal. I mean, it hits home pretty hard. It's like, well, no, they won't. That's just by chance. He said, they don't. That's just by chance.

You say, yes, exactly. It's just by chance. Yes. Wow. That is an amazing story. Cool. Yeah. It got the point across.

We were not invited back to that client anymore, but I think they're doing better. But you taught somebody learned something that day. Yes. So here's a question for both of you. So one of the things that's really interesting about this is setting our expectations around, I mean, technology generally, right? But if we think about sort of data science as a discipline and this modeling where you're getting into a space that's predictive, right? Which is way different than,

you know, sort of historical look back, you know, analytics, right? This is my time series, and I'm just sort of understanding what happens, right? I mean, actually amazing how often that is done poorly. But you're right. And I think that even reinforces my point, which is, how do you think about setting expectations around technology, right? And I love that you said, okay, at Elder, we say, this is how accurate we can be, right? Which is setting a really different expectation for data science than here, like, I'm going to give you a tidy packaged answer, right? Yeah. I mean, that's a constant struggle for us. And I think probably for everybody in the field, especially in the post-chat GPT world where there is so much noise out there of look what you can do with this new AI tool

and demos look great. And in very specific circumstances, tools can be super powerful. And we, to come in as the voice of skeptics, even you've been in the space for 30 years or been working in the space for 10 years, you come in as a skeptic and it's, okay, well, this guy just doesn't believe, he doesn't think that this is going to work and we're going to jump to it and it's going to work. And so I think it's tough to try to balance that excitement about new capabilities with like grounded in realistic expectations. And it's hard. And as a consultant, when you're trying to sell services out there

the flashy stuff sells yeah so i don't know maybe our strategy is to wait for the flashy stuff to break and then say see we told you so you want us to help fix it yeah i mean actually a great strategy yeah yeah i mean honestly like there's a consultant we worked with that worked with a very specialty kind of niche software and like his whole and he'd been doing like 30 30 plus year years and his whole like thing was like i'm the cleanup guy like i i like it was a very small operation and like the big operations would come in and like and would kind of know the software but not really and he would just like basically be the cleanup guy come behind each one of them fix fix what they screwed up, and did it. It was like a 30-year business model. He did quite

well. Yeah. I mentioned model validation earlier. That's broadly in IT, but in analytics, there's absolutely a niche there for model validation.

I like it, the clean up guy. Yeah. Yeah. Yeah. That's super, inside of a company as well, I'm just thinking about some of the stakeholders that you had, John, even at the executive level. And it seems like a really similar dynamic, right? I mean, you're selling services as a consultant, right?

So, but inside of a company, you have to sell as well. And I don't know, would you say that's like easier or harder? I think it depends. It can be easier in some sense and harder in another. Easier in the sense that like, if you've been there a little while, you have that precursor like track record, right? Of like, if you've been there a little while, you have that precursor like track record, right? Of like if you've been there several years. And that matters.

It can be harder in the sense where if you're getting into something more niche than maybe you haven't, like we got into some very like specific types of forecasting. Like I'm not an expert in this type of forecasting, right? So like that's where it can be harder. But I think either way it's a sales job you know like especially i mean data i think data people are maybe

skewed more this way than a lot of disciplines like they do not want to sell they want to present the data right this is the data yeah evan's nodding yep yep for sure and i mean that's how i feel too like i yeah like i just want to present the data but yeah and the sale becomes clarity and communication and simplification often like without losing the core of it and that's i think the hard part yeah yeah yeah spot on yeah i would agree okay enough with the serious talk yeah it's time oh gosh how did you get into data comedy evan this is fascinating to me i majored in it in college you have four-year degree in data no that's sort of the byproduct of two master's degrees yeah no sorry dad if you, I didn't, I majored in well, business administration.

So probably equally useful. Um, yeah, so I, this is shortly into the pandemic at elder research. We're a Slack company. I don't get paid by them to say this, but if you're on teams, maybe just go on Slack for a little bit and just see how nice it is. Um, Slack's great for all internal communication and we're pretty involved academically. We've got people that do research and write books and somebody was going to a conference on undergraduate statistics education. So basically a bunch of

stats professors and they had a call for papers and I don't have any stats papers to my name, but in the call for papers, they said, hey any stats papers to my name, but they, in the call for papers, they said, Hey, we're also looking for fun stuff. So if you've written a poem, if you've written a song, performed a music. A little talent show. Have a joke. Yeah. Yeah.

So they solicited that. Yeah. And so I don't tell my boss, but you know, I, I spent like 40 billable hours just thinking up statistics. Came up with one submitted. Actually, I came up with three submitted. One of them won one first place. So I was invited to this virtual, but I was invited to this virtual conference, got an award, read the joke. I could like hear people's eye roll on the virtual conference, but I got a $50 prize. So that made me a professional statistics that you have been paid and award-winning.

Yes. And award-winning professional. Yeah, exactly. And so I absolutely wore that out. I told that joke to my colleagues hundreds of times over. My poor wife has heard it so many times and just start to tell it enough and start to think of some more jokes and you become the guy with the stats jokes. And so, you know, we do mention I did several like teaching engagements with clients and say, Hey, before we teach this forecasting class, why don't you open with a few jokes? And so I'd tell a few jokes and started getting a pretty warm reception, started thinking of more jokes. Eventually wrote, I did a little market research. If you want to write a joke book, you have to have a hundred or a thousand in one, but there's no way I was getting to a thousand and one. So had a hundred and one jokes,

published a joke book. And that's a lot of jokes. Yeah, it's a lot. Yeah. The quality drops down a lot there at the end. Right. It's all flow. Well, statistics is like a bell curve. Yeah, yeah, exactly. Yes, yeah. I would, I mean, this is a very far right skewed

or I guess left skewed. Yeah, then, so you have a book and then some conference organizers get a hold, say, you want to tell jokes at this conference and some university programs want you to come along. So now, you know, I still try to tell some jokes for all the research clients, but now I'm not quitting my day job,

but every chance I get, you know, if somebody wants me to come tell jokes for their data or analytics or tech team, then I've been doing standup for folks. Okay, so I'd have to, I want to hear, I want to hear the jokes, but there's a big difference between writing a funny joke and delivering that in front of people, especially live. Can you talk about that a little bit? Cause like that's a big transition.

Yeah, that is a really great point. And I think it mirrors to sort of our earlier conversation where you've got data science people that may be able to build these great technical tools, but they can't sell it. They can't communicate it in a simple way. And so I think you could almost get technical on joke writing, but the joke delivery is very much a human endeavor. It is very much how do you inflect? How do you stay engaged with an audience? I've done a few virtual and it is infinitely harder virtual than it is in person when you can make eye contact and engage with folks. So when I do standup comedy, I don't tell a single joke out of the book unless somebody requests it. They're just, the jokes from the book

are standalone jokes. The standup comedy is much more storytelling around data and analytics and finding the humor in it. So yeah, it's very different disciplines. Yeah, man, that is amazing. I could keep going down that path because that's fascinating to me. Super awesome. We could do a whole nother episode. We could, yeah. Yeah. Data We could do a whole other episode. Yeah. Yeah. Data comedy episode. Stand-up comedy episode.

Yeah. Okay. We're close to the buzzer here. All right. Folks have listened this long. Good night. We have made you wait this long. That was a great call making people wait to the end. Yeah.

I'm sorry for all the angry emails you guys are going to get. Yeah. We'll take it. So the joke that I've told more than any other is the one that made me an award-winning professional. Yes. I was going to ask if you weren't going to share it. I was going to ask her that one specifically. Yeah.

Actually, I'll tell it first, but I got a funny story about it too. Did you hear about the Bayesian who built a model to tell her when she needs to go to the dentist? Well, it said she didn't need to go for six months, but that's probably because she just had a week prior. That's great. That is great. My boss is way funnier than me.

I'm glad he has a successful consulting firm so he doesn't get into data comedy. But every time he hears me tell that joke, which has been way too many times, he counters with, Evan, did they ever tell you if anybody else submitted a joke to that contest? Thanks, John. Wow.

That actually makes it even more funny. That is great. Yeah. funny. That is great. Yeah. Yeah. Oh man. This one, this one, this was, I got the book. Yeah. I'm looking at the back of the book cover. This one is, did you hear about the 12th grade student that failed his machine learning exam so badly that he had to go back to 11th grade as classic case of scholastic grade descent. One of the best things about telling very nerdy jokes is like the built in defense mechanism. If people don't laugh, then I just assume they just don't get, they just don't.

Right. Right. Yeah. You can feel good about yourself because like, oh, they just didn't get it. Do you have a process for writing these or do they like, are they more lightning strikes? Yeah, that's a great question and it's tough. So there are a few in the book that end up being sort of formulaic almost. And I don't like those as much. But most of the time, I've got about a 30-minute commute on days when I go into the office. And I don't do it so much anymore. But when I was writing the joke book and when I'm trying to make new standup material, I just ride in silence, just ride with the no radio on like windows up.

And it feels so bizarre in today's world to not have, you know, some sort of input. Yeah. The tech stack podcast plan in the background, but then just freeing mind. And then realize basically no time in my life where there's no input. And so I started taking walks, driving with no radio on it, doing things where there's sort of the freedom to be bored and let your mind wander around things. Yep.

And there's been a ton of dead end chases, but then I come up with something that I think, oh, okay, that, that could stick. That could make it. Yeah. Yeah. Yeah. could make it. Yeah. Yeah. Yeah. I love it. Awesome.

All right. Well, do you have one more for us before we close it out today? Yeah. I will tell another one. This is objectively worse. It didn't win the prize, but it was submitted to the same joke competition there. So you've got a guy recently graduated, studied data scientists, studied data science, looking for a job as a data scientist, can't find anything, job market's

tough. So he applies for a logistic analyst job. He doesn't know anything about logistics, but hey, it's got analysts in the title. He goes in, he's interviewing with the guy. He's got a big map behind him. The boss does. Says, let's say you had to put a new distribution center. Here's our network right now. Where do you think you'd put a new distribution center? They thought about it for a Senate. When he put a pin right in the middle of a divided highway and the boss just laughs. He said, yeah, you're going to put our distribution center in the middle of a highway. That's not going to work. He said, sir, a normal distribution center is not going to work. He said, sir, a normal distribution center

is always on the median. I don't know. I like that one a lot. It's hilarious. The background. Oh, wow. I love it. I love it.

I lost to the Bayesian one. It's at best the second best joke that I got. It's amazing. All right. Well, where can listeners get the book? Predictablejokes.com.

Predictablejokes.com. It's on Amazon. You can find it, but I'm getting buried by all the people who know how to do good SEO and keywords. But Predictablejokes.com. You got to stand up on there. You got a few clips and you can get the book. The book is called Predictable Jokes.

Go to Evan's site. You know, we'd rather give him the money than... Yeah, exactly. Yes. Thank you. Thank you. I'm coming for you, Jeff. He started with the bookstore. You're starting with a book. That's right. Exactly. Exactly. Yeah. If we were doing synthetic controls, like he does his trajectory math well to mine. Yeah, I was going to say, I don't know if you're going to actually be able to produce a good synthetic control there. All righty. Well, Evan, so great to have you on the show. This is a great time. Excited to check out the book. Thanks for joining us.

Absolutely. Great conversation. Appreciate you all having me on. The Data Stack Show is brought to you by Rudderstack, the warehouse native customer data platform. Rudderstack is purpose-built to help data teams turn customer data into competitive advantage. Learn more at ruddersack.com.