Software Misadventures - Evan Estola - On recommendation systems going bad, hiring ML engineers, giving constructive feedback, filter bubbles and much more - #9

Episode Date: April 23, 2021

Evan Estola (https://twitter.com/estola) is a Director of Engineering at Flatiron Health where he's leading software engineering teams focused on building Machine Learning products. Throughout this episode, Evan shares various stories when recommendation systems didn’t work as expected, like this one time when members saw mathematically worst recommendations for meetups near them. He also shares why Schenectady, NY pops up on some lists of most popular cities and the story behind the Wall Street Journal article titled 'Orbitz steers Mac users to pricier hotels'. We also discuss skills Evan looks for when hiring ML engineers, how to give constructive feedback, filter bubbles and much more.

I would say one of the big things that most machine learning folks have is a lot of scar tissue around getting burnt by machine learning algorithms. One of the things that you learn is, oh, this algorithm has 99% accuracy. Awesome, I cracked it. And it's like, no, you didn't. You broke it. You put data in that it wasn't supposed to know. And constructing the problem of how you ask a machine learning question is a really hard thing to do. Welcome to the Software Misadventures podcast, where we sit down with software and DevOps experts to hear their stories from the trenches about how software breaks in production. We are your hosts, Ronak, Austin, and Guang. We've seen firsthand how stressful it is when something breaks in production,

but it's the best opportunity to learn about a system more deeply. When most of us started in this field, we didn't really know what to expect, and wish there were more resources on how veteran engineers overcame the daunting task of debugging complex systems. In these conversations, we discuss the principles and practical tips to build resilient software, as well as advice to grow as technical leaders. Hey everyone, this is Ronak here. Our guest in this episode is Evan Estola. Evan is a Director of Engineering at Flatiron Health, where he's leading engineering teams

focused on building machine learning products. Prior to this, he was a lead machine learning engineer at Meetup building recommendation algorithms, and before that, he worked on hotel recommendations at Orbitz. Guang and I had so much fun speaking with Evan. Throughout the episode, he shares various stories when recommendation systems did not work as expected. Like this one time when members on Meetup saw mathematically worse recommendations for Meetups near them. He also shares why Schenectady, New York pops up on some lists of most popular cities and the story behind a Wall Street Journal article about Orbit steering Mac users to book pricier hotels. We also discuss skills Evan looks for when hiring ML engineers, how to give constructive feedback, filter bubbles and much more.

Please enjoy this highly entertaining conversation with Evan Estola. Evan, super excited to talk to you today. Welcome to the show. Thank you very much for having me. So you're a director of engineering at Flatiron today, and you've been working on machine learning systems for majority of your engineering career. Can you tell us about how you got started into machine learning? Sure. So I think my path in machine learning is, has a lot of overlap with my path to becoming a software engineer at all. I, I was like a math kid. I loved, you know, that side of school. And I was never really a programmer. I didn't get exposed to it super young or anything like that. But I did get exposed to like open source and Linux type stuff. So I was used to

like tinkering around on my computer. And I initially went to school for biomedical engineering. And when I was there, I made some friends, some lifelong friends of people that I'm still good friends with to this day. And they were computer science majors. And not only that, but they were taking a data mining class. And they told me about data mining. And I was like, that's the coolest thing I've ever heard of. Why wouldn't I do that? And so basically just switched over soon after that and have been pursuing that ever since.

For listeners who are younger and have never heard this old-timey term, data mining. Yeah, I think it would be great to explain that. I think data mining was kind of, it involved machine learning techniques, but was really about any sort of systematic approach to using data sets to help make business decisions or anything like that. So it was a very sort of pragmatic angle into using data and into the machine learning world. And so that's what sort of sparked me from the very beginning was not just the tools and the algorithms and all that, but also the use of it and the business or whatever you want to apply it to. I spent a lot of time in undergrad working for an information retrieval lab.

I think there's a couple different ways that people get into the machine learning space. There's a couple types of labs that became machine learning labs. Everybody's machine learning space or there's a couple like types of labs that became machine learning labs like everybody's machine learning now but there's the sort of ai side and that has all this history from the 80s or whatever and and was not always using machine learning type approaches but some of this fancy knowledge stuff or or you know even the the old school like chess algorithms all using sort of knowledge bases and this sort of thing. But those groups got moved towards machine learning, or rather are definitely calling themselves machine learning.

Well, I guess AI is cool again, too, but whatever. I came from this sort of information retrieval side. So I think that how that changes the way I look at things is it's always practical it's always about impact in a business for a customer for a user whatever um not to say that people coming from the other side don't do that as well but i think that's always been the lens that i looked at applying machine learning through that's pretty cool uh so you mentioned a little bit about the math background and actually um i've heard different aspects from different people I've spoken with. Some will say that, hey, to do machine learning effectively, you need to understand math.

Whereas a lot of folks these days who, well, today, a lot of folks have the title of machine learning engineers. And some of those will say that, well, you need to know how to code. And having a CS background is, if not the same, maybe a little more important than having or understanding all the math behind it. I'm curious how much of machine learning these days is math versus CS or it depends. Like, I'm just curious to get your thoughts on this. Yeah, that's a great question. So I loved math. I'm not sure I was good at it.

Or rather, I thought I was good at it until I got to college and met people that were actually good at math. Oh, that's all of us. People that could actually good at math oh that's all of us in math that's all of us people that can just think in math I was like okay I'm I'm I'm not I guess I'm not a math person anymore but uh I mean I still like math I think an interesting thing is you can never know all the math right so when I was first learning machine learning stuff the math we did was all linear algebra, discrete. And now with, you know, deep learning type stuff, it's all back to continuous calculus, you know, differentiable functions and all this sort of thing.

So you're never going to know, you're never going to know all the math. You're never going to know the math that you need to know next. I would argue that you can never know too much statistics. Okay, maybe can never know too much statistics. Okay, maybe you can know too much statistics, but unless you're a statistics person,

you could never know too much statistics. Especially working in a business, you're often going to meet people with different backgrounds and need to explain things and need to just figure out how confident you are about something. The stats is always useful. In terms of the engineering side, when I started out, and I could probably tell a bunch of stories about how old I am or whatever,

but when I started out, you had to really know a lot of engineering to do any of this stuff. So we were rolling our own Hadoop clusters, and the amount of engineering that it took to process large amounts of data was pretty intense. Nowadays, tools have gotten a lot better. So there's a lot more space for folks that are coming in just on the either math or the sort of product plus ML math or whatever. So I think there's a lot of valid ways to get into this. And we kind of need everybody coming from all these different angles and you're never going to know everything. Um, so I think there's a lot of valid ways to get into this and we kind of need, we kind of need everybody coming from all these different angles and you're never going to know everything.

Yeah, makes sense. So I assume like you would be spending a lot of time, a lot of your time hiring, uh, being in this position that you are, uh, what sort of skill sets do you look for, uh, when you're interviewing folks? Is it like, depends on the kind of role you have on the team you were indexed on one side or the other, or you're looking at the candidate holistically i think it i think it depends uh i think it's a bit of both so sometimes we find people that are that are just great and seem like they're excited to work on hard problems and and you know especially one great thing at flatiron is we

are always looking for good communicators. We work in such a cross-functional space that communication and being able to explain things and being able to explain hard concepts, always super, super useful. Even back in my meetup days, you know, on the ML side, someone who can, someone who can approach a problem from a problem solving angle and not a, I want to use this cool algorithm side of things. Like we'd kind of set up our interview questions to, to get people to give us a naive approach first, like just use search or just use, you know, just do right by ranked by most popular, like don't go right to doing the craziest algorithm because that first thing might work pretty well. And so it was always about sort of problems first. And then, you know, especially the small company like that, being able to communicate with

product team, being able to communicate with the CEO. I always joke that the like, I think, you know, maybe we'll talk a bit more about, like, interpretability later. But I always joke that the CEO problem is if your CEO gets a bad result from one of your algorithms, you better be able to explain it. Because he might come right over to your desk, especially at a small company. Oh, yeah, they would be the best giveaway people at a company could ever have. Kind of, but they're also kind of the worst because the CEO has the weirdest data. They use the product for the weirdest thing. That is true.

They're not a normal user. This conversation is getting a little bit too real for me, but let's continue. So I understand that you're thinking of machine learning as a means to an end instead of just thinking of like, hey, I want to use this cool algorithm. In terms of just working with the product itself, and like I said, communicating with people, how do you think about learnability or teachability versus someone who has tons of experience coming into the job? Like, how do you balance that? Yeah, that's a great question. I think in our industry, anybody that has a lot of experience has learned a lot of stuff. Being a computer scientist is kind of being a permanent learner. I said, you never know all the math. You're never going to know all the software packages.

When I talk to people that are trying to get into the industry as a whole, that's the main thing I tell them. Just learn how to learn things because you can check every box on a resume or on an application in terms of the tools that you think you need to know. You're never going to know. Even if you know all the packages they use, you're still not going to know that code base. Everywhere you go, you have to learn. And so I definitely think that that's a key aspect. But that's not to say that people who have a lot of experience, they've probably been through a lot of those before. engineering where you are very close to the management and the business, I would imagine, then as compared to being an IC, how does it help you see the business aspect a little more closely? And how does it help you influence some of the decisions you make in terms of like the direction the team is taking? Oh, yeah, great question. I love I love working with the business. I love

working with my product partners. I love working with all the different cross-functional folks. Now that I'm in health tech, we have doctors and other clinicians, and it's really a whole range of people that are influencing things. I always love when problems... I love working on problems that are complicated enough that knowing a technical solution or knowing a different technical approach changes the way you approach the problem. And so being able to get engineers who want to understand the problems we're working on and can work cross-functionally and come into something and say, hey, actually, I know this technical, there's a possibility that if we look at it in this different lens we can solve

this problem in a way that that a non-technical person never would have come up with and so i love kind of bridging that gap and and that's a lot of what i do now in my role is just helping to frame the business problem helping you communicate technical things to non-technical people and really connect smart people to hard problems that's that's my favorite thing to do that's really cool and how has that changed from your last job working at meetup because to me one of the benefits of working in like a you know deeply technical field like ml or engineering is that it is somewhat independent of the domain itself obviously to excel at it you need to have a

lot of domain expertise, but it's usually not a prerequisite to get the job in the first place. Was there a steep learning curve that you felt like you had to kind of go through when you first joined Flatiron? Or what was that like? Yeah, that's a great question. So this is a question that's near and dear to my heart, because obviously a huge part of what I do is onboarding people into Flatiron. And absolutely, it's a steep learning curve. And we don't look for, if we only looked for machine learning experts that also have NLP application and also have a knowledge of cancer genomics, we probably wouldn't find many people. Luckily, there are a lot of people that have an interest in it. Like I, I, I'd like to joke that Flatiron probably has the highest percentage of people who thought they were

going to be doctors and ended up as computer scientists or statisticians or whatever. We, I think we have the highest percentage of those people in the world. Not that everyone at Flatiron comes from that background, but having some sort of interest in the biology side and the medical side helps a lot because there is a lot of just terminology to learn. I mean, that's the first few months onboarding at Flatiron is just learning a bunch of, you know, not only learning that tech that we were talking about, learning the tools we use and all this stuff, but also just learning all these medical concepts. It's a challenge, but it's also just a ton of fun.

And I think a lot of people in our world are just curious and just want to know how things work. There's a lot of opportunities through that. Going from not becoming a doctor to something else, having Chinese parents, I can definitely relate to how that feels. I was actually going to ask, you already shared some aspects of this, but what does a typical day for you look like as the director of engineering at Flatiron? Yeah, I'd say my job varies quite a bit. I get most of my work done through other people or working with other people. So naturally, I'm spending a trying to find the right people, put them together, help them understand what they should be working on.

I really love the management side of my job. I love giving feedback. I love giving positive feedback. I love giving constructive feedback. And I generally just like helping people see what they can do uh and and to help them get there and then to help you know impact impact the business and i've been lucky enough to work on businesses that i believe that what we're doing is also a good thing to do so you impact the business you impact the world it's like it works out oh so i i actually one of my personal goals is to get better at giving feedback.

I find it pretty difficult, especially constructive, right? Because I feel like usually you need to have social capital. You need to build a lot of trust before. And you also want to be very precise about the feedback you give such that it doesn't feel like, oh, just a feeling but it's based on evidence how is that something that just came naturally to you or is that a skill set like you develop over time like what do you have any tips or advice for me actually i would say plus one to that question i mean i think i think i think you nailed a lot of things right good good feedback is specific and actionable and comes from a place of care.

I think you're right that, you know, you can't just jump in and start giving people critical feedback when they don't know anything about you and they don't trust you. You have to get to that place of trust with people where they trust that you have their interests in mind as well when you're giving that feedback. So I wouldn't say it comes natural. I sometimes joke, my dad's an American football coach. That's his that's his profession. So I certainly grew up in a household with probably more constructive feedback than most people did. My dad has my dad has no problem yelling at people. Obviously, I frame things very differently than that. But I think that I guess I guess the the way that really connects is I remember when I was a kid and I remember you know coach kind of getting down on me for something I did on the football field and my dad said hey you know if

the coach if the coach is getting on you that means that means he thinks you're good it means he cares because he wants you to be good for the team. If he, if he was going to bench you, he wouldn't be wasting his time on you. So that's, that's kind of where I learned to love and seek critical feedback myself. And I think that's helped me learn to see that in others and see how critical feedback is good for people. And, and to, when you really know you've got it is when you've given somebody like critical feedback and, and it increases your relationship instead of, instead of costing. Obviously you have to have a relationship first. That's not, that's not, you can't, you can't use it from nothing. But, but once you've gone far enough, you can actually

develop more trust with somebody by giving them critical feedback because they know that you care. They know that you're willing to have an awkward conversation to help them out. And so it has to come from a place of caring. I wish I had that wisdom when my mom was yelling at me when I was growing up. Probably would have turned out better. I know we're digressing, but I have a follow-up on that. So one aspect is, well, when there's a feedback conversation, one aspect is giving constructive feedback and doing it the right way.

I think the other aspect of the conversation is also on the other side, where the person is willing to hear the feedback and understand it's coming from a place of care and this person wants me to improve. I'm assuming there might be situations where the acceptance of feedback coming from a place of care and this person wants me to improve. I'm assuming there might be situations where the acceptance of feedback is not immediate, or at least there is more like, hey, why do you think so? How do you handle those conversations in that case?

Yeah, that's a great question. In my mind, that's the first concern I have if I'm thinking about constructive feedback. It's like, what if this person doesn't believe what I say? How do you get over that? Ooh, yeah, great point. And certainly when I was starting off as a manager, to say any of this came naturally. Also, I've screwed this up many times. Probably some former reporter of mine could listen to this and go, that guy was a jerk.

What is he talking about here? It's something that I've definitely had to develop over time. I think there's a couple things that make it easier. One thing that's been really great is Flatiron has really well developed. I think a lot of companies are getting better at this. We have a great career ladder. I know I really sound like a director of engineering when I talk about how excited I get

about a good career ladder. A good career ladder is a beautiful thing because you can point to things in that document. You can help people understand it over time. And that really helps to contextualize things. And that, I think, can make things more clear um and and that helps a lot with the the receiver of the feedback knowing why you're giving them this feedback how that fits in with the bigger picture that sort of thing got it got it um really cool cool so changing gears a

little bit uh here on the podcast we love to, you know, hear stories. You know, very excited to hear some of your stories today about recommendation systems going bad. Before we start, can you give us kind of a TLDR on what a recommender system is and how does it work? Yeah. Yeah, I love that. Because I think it's not as obvious as it seems. I mean, I imagine some people are like, oh, yeah, I love that. Because I think it's not as obvious as it seems.

I mean, I imagine some people are like, oh, yeah, recommender system. Amazon, people who like this also bought this, right? But I think there's actually have more items that a user could potentially engage with than you have attention for that user to spend finding something to engage with. So, you know, the classic example is a little three boxes at the bottom of a product page that says, hey, if you liked this, you might also like these things. But it goes all the way up to Netflix.

Netflix, they have a recommender system of recommender systems. They build that whole page, the list of lists. That whole thing is optimized all together from what I understand. So there's a lot that goes into it. So a recommender system, in terms of how they work, they can be anything from a simple graph walk. Like I said, people who like this also like this. Even saying the words graph walk makes that sound more complicated than it is, right? You literally could just look up.

If you have that data stored, you could just look up whoever liked these other things. A lot of times you can model a recommendation algorithm as a, even just a classification problem. If I put this in front of somebody, will they click it? And then when, you know, the sort of next level up is you look at it as a ranking problem. It's like, okay, we have, we have n number of impressions that we can give. How do we put the best things that this user might engage with into those spots? And do we just put the things we think they're most likely to engage with? Do we use that space to give them new things that we want to know if they're going to like them or not? Do we use that

space to make sure they have a variety of options? Because maybe they're in different moods when they come to engage with the product. So there's a lot of different ways that it kind of comes together. We did a, I worked, my first ever job building recommender systems was at Orbitz. I built a little hotel recommendations module. So if you're looking at a hotel, we showed three other hotels that you might like. And one of the most successful algorithms we ever deployed for that, we found a hotel that was similar to the one you were looking at. We found a hotel that was at least $20 cheaper than the hotel you were looking at. And we found a hotel that was at least a star rating up from the hotel you were looking at. And it was this sort of Goldilocks sort of scenario from whatever buying psychology land. And that was one of the

most successful algorithms that we ever did was kind. So recommender systems nowadays, I'm sure you'd be doing all sorts of personalization and machine learning for that. But even just sort of that hand crafting thing really played a role in making that successful. Nice. So speaking of Orbitz, our first story starts with a Wall Street Journal article from a while back called Orbitz Steers Mac Users to Pricier Hotels. So, yeah, so what happened? So, I can't remember the exact details, but I think the first version of that headline was even worse. I think the first version of that headline really made it seem like Orbitz was actually charging Mac users more. So my story for this is strange because I was actually not at Orbitz when that article came out. I was in my first week at Meetup. So I had just left Orbitz, and I'm not there.

And so I can't speak to the response or what happened inside or anything like that. But I just remember I just moved to New York City, I'm in my new office and now there's this national news story, this thing, Wall Street Journal published it, the next day it's on Good Morning America this was, honestly looking back on it, this was one of the first

big, like, is tech bad news stories it's crazy to think about how that's a whole style of journalism now but like this was one of the first kind of big ones i think the story as i know it is that we had a group the team that i was working on this i was junior engineer pretty much a couple years out of school fresh out of school the team i was working with was exploring different data points

that we had available to us. A lot of people browsing Orbitz are not logged in, so we can't necessarily tie you to your user history and all that. It was hard to tie people to user history anyways. Like I said, we were rolling our own Hadoop clusters and doing our own thing. It was hard. It was hard.

But one of the things that we had available to us was the user agent. So we knew your operating system, your browser, and that sort of thing. So like I said, I was working on that hotel recommendations module, basically three boxes at the bottom of your hotel screen. And we did an A-B test of basically doing that kind of very simplistic graph walk algorithm where we took the hotel you were looking at and we looked at people who looked at that hotel, what hotels did they end up booking? And we showed you the hotels that people were most likely to book after looking at the hotel you're looking at. And then we thought, oh, well, we know this user

agent, maybe we can segregate the data that we use. So for Mac users, we'll only use Mac data. And for PC users, we'll only use PC data. And we had some reason to think that might work because we had done some data analysis that had shown that Mac users tended to spend like on average 20 or something. Maybe it was a lot. It might have been like $100 more per hotel night on average. So we knew that Mac users... And this makes sense. A Mac was two, three, four times as much as a PC. So that kind of made sense, fit with our intuition. We deployed the algorithm,

and it failed the A-B test test so i turned it off um in hindsight i think the fact that the user had already clicked on a hotel they pretty much already given us their price point when they clicked the first hotel so they they'd already given us more information than their browser was going to tell us about about how much money they were looking to spend on this room. Interesting. So the flip side then is, I think, I don't know who it was at Orbis or whatever, but the sense I get is that Wall Street Journal came in and was like, hey, we'd love to do a story on you.

And Orbis was like, oh, cool. We have this team doing all this really cool data stuff and doing all this really smart stuff with data. And here's one of the cool things we found is that, you know, people on Macs spend more on hotels than PC users do. So we're using that to, like, you know, to influence our algorithms or whatever. I think the article implies that it was used in search. As far as I know, it was never used in search. As far as I know, the only time we ever used it was in that recommendation engine ab test that lost and i turned off

and it was national news story like i said it wasn't like good morning america that's a that's a crossover you know that's a that's a that's a cultural touch point that's not just the tech or business world i feel like one of the takeaways for me is kind of ties back to what you were saying before about can't know enough statistics because i feel like a lot of the concepts in stats is it's not super intuitive but you so it takes time to become comfortable with the um with the concepts like here i don't know you know causal inference is the reason why sort of that gap in terms of like how we're using it versus how you know journalists are

perceiving it but i definitely see those kind of discrepancies pop up um where and then it kind of causes mass confusion in terms of like okay what what's happening here um so so that's really cool the takeaway i've always had from it i think that's totally valid. The other takeaway that I've sort of reflected on over the years was that the problem was, I suspect whoever shared this information with the Wall Street Journal, they did so willingly, by the way. They thought it was going to be an article about how smart and cool the team's work was. And I think the problem was, it was about how smart and cool the team's work was and not about what value it was providing to the customer. And so if you frame something as the value that it's providing for someone,

it's a lot, it's a lot harder to be accidentally taken as, Oh, we're tricking people or we're doing this, we're doing this thing to scam people. So I've always used that as a motivation to like continue keeping the customer in mind that's really well said because then you're not just distracted

by this shiny algorithm without looking at what actually comes out any other stories about these lines? so at Meetup we had a couple that I enjoyed one of my favorites

regards Schenectady, New York. Have either of you ever heard of Schenectady, New York? Nope. I haven't. So Schenectady, New York is a small town essentially right outside of Albany, the capital city of New York. And Schenectady, now that you've heard of it,

I bet you'll see it somewhere someday. Because it frequently pops up anytime you're looking at cities and when you're collecting your geography data in a certain way. And so I'm not giving it away just yet in case you or the listeners want to try and figure it out. So I'll give you an example. We did the first time I ran into Schenectady. We're just trying to figure out what are the biggest cities for using Meetup in the country. So pulled some geography, pulled like census data and then took our data, divided it.

And Schenectady pops to the top there was a higher percentage of the population of schenectady were using meetup than anywhere else in the world um i've seen this as well do you remember ashley madison it's like a danian website that had some sort of oh yeah the the the the adultery thingy i think that was the i think that was the, I think that was the gist. I'm not deeply familiar or anything, but I saw a news... I thought that was a trip question there,

but okay. No, I saw a news story once that said, what are the top cities for Ashley Madison users? And it was like, LA, New York, Schenectady.

So this pops up. Any guesses? Have you figured out why this happens yet? Are a lot of the things being routed through it. So it's like the data is collected there, even though the users are not actually there. Is it something like that? That's a good guess. That's a good guess. So the, the, the trick is it's actually user generated data and it's user generated. If you ask the user for their zip code, because a percentage of people, when you ask them for their zip code, are going to type in 1, 2, 3, 4, 5.

Is that the actual zip code for the city? So it's not even the zip code for the main city. It's actually a zip code for a GE plant that sits in the city. And GE got this. It was like an honorary zip code the whole the background the story of zip codes is fascinating but they didn't used to exist and now they're like the main way a letter gets to where it's going like you could write anything you want as the city but if you put a zip code that's the post office it goes to so GE was given this

honorary zip code when they were first giving out zip codes i guess and now they have to staff this huge mail room because anybody that puts one two three four five on a letter letters to santa like they all end up at this mail room at ge um i feel bad for ge here a friend of a friend of mine was a was actually a journey journalist at albany at one point did a whole story on this and like interviewed people in the newsroom and stuff. But yeah, so keep an eye out for Schenectady. It's really just a parable of make sure your data is clean and don't trust

user-generated data. But yeah, there's a lot of... Sorry, go ahead. No, that's actually kind of cool because part of it is sort of how you discovered it is sanity check, right? It's like you've done analysis and then it doesn't. And I think, I mean, that's kind of a commentary

to a lot of, you know, machine learning products or problems you want to solve, right? Because I can imagine, you know, me working on this 2 a.m., you know, filling a ticket and then it's like, oh yeah, let me just pull up the top 10 and then, all right, looks good to me. Do you like have sort of a process in terms of like, other than just, you know, because I think you do have to care a lot about the problem and also having a

good standard to your work, right. In order to like examine these things. This case, it could be very obvious, but sometimes it's, it's, it's more subtle. How do you, I'm curious, like, how do you kind of go about getting people to do like a lot of sanity checks yeah good question one of my former co-workers randy i think he's randy underscore au on twitter he's great follow um i think he he has always described his job as either counting things or or data cleaning or something like that and he has like one of his constant refrains is know your data. And so if you're if you're going to be trying to do something with with data,

you can't spend too much time getting to know that data. And often the best way to get to know data is just, you know, run some top 10s, check it out, check the how long how long does the tail go, check the bottom of it. And just, you know, know, know, know what you're working with building up that intuition. And like we said earlier, this is one of the big challenges with a place like Flatiron, it's very hard to get to know, but it's insanely complicated cancer data. But it's really the only path to building up that intuition. And it really becomes a sort of superpower of being able to understand, you know, what am I expecting my model to see? Because I know the kind of underlying data and what that means.

So before Flatiron, you led the team at Meetup. And I can imagine one of the core ML problems there, right, is how do you recommend the best meetups for users? And I think you were telling us like one of the times this didn't exactly happen as intended. What happened? Yeah, so at one time, my team put out an algorithm change, and we had a pretty good workflow,

so we could launch stuff pretty quickly. And we put out a change. I probably code reviewed it myself. Looks good. And we ended up reversing our recommendations. So the way we did recommendations at Meetup is we would select all of the Meetup groups near you. It's geography bound.

We'd find all the Meetups near you. And then we had enough time. Our algorithms were fast enough and our model was simple enough. We could score every meetup in your area and then just sort them and show you the top three or five or however many spots we had on that page. And we could even reverse that and say for a given group, who are the people that are most likely to join? And that's who we would email when a new group started. So using basically the same algorithm. But yeah, at one point, we accidentally put out a change that reversed the order.

So we were literally showing the mathematically worst meetups for all of our users. And most of it didn't reveal anything too deep about the psyche it wasn't like it wasn't like we found like who is the opposite of you it was mostly just showing just like kind of the most garbage things on on the site you know the most empty things are the most sort of like weird spammy things and that sort of thing um which was good it actually made us feel really good about recommendation system we were like hey i think this thing's actually working because we're looking at the bottom really tells you something um yeah and how did you guys discover the problem like after i think i think we just discovered it just by looking at looking

at the recommendations i don't think we had i don't think we i don't think we had, I don't think we, I don't think we had time to notice from the user engagement. We had a lot of graphs about, you know, how, how much are people, all of our AB testing stuff was all tied into the, to the systems. And so we monitored them constantly. I mean, this was, it was a relatively small team and the, you know, the tools, all the tools that exist today didn't quite exist yet. And so I just spent a lot of my life in Graphite just looking at graphs of how things were going. But I don't even think we noticed it in the monitoring because I think somebody pretty immediately noticed. They were like, these look bad.

And I think we knew enough about the release that had happened to have a guess as to like, ooh, it was probably that code. And then pretty quickly figured out that like, oh man, we just literally, the minus sign on this. Kind of extrapolating on that. So for software, I feel it's more straightforward to monitor for when something breaks. Austin is not here today. Otherwise, he'd be rolling his eyes at me, I i think at that statement uh having worked on monitoring infra but um but you get an

error right at the end of the day usually you know compilation or runtime or something else but then for ml i do think it's a lot harder because everything can compile just fine but then you know the model can be spewing out sort of total nonsense um i think there are easier cases right if you're trying to catch fraud and everything is flagging as fraud then you know maybe it doesn't even pass your ci process and maybe you you add some kind of um rule to say like you know if this doesn't just looks like completely out of whack you know we need to stop but but for some of the more subtle sort of things um you know it becomes a lot more difficult but um yeah i guess curious to get your thought around like you know debugging

some of these problems and things like that yeah i think one one one thought that that brings to mind is related to just um deploying machine learning models and using them in production and A-B testing things in general. One of the great, one of the things that I think everybody who deploys ML models in a consumer scenario, especially any sort of recommender system or search or anything like that, I think a lot of people, I mean, maybe not everybody, I'm sure there's problems where this doesn't apply to, but a lot of people eventually run into a place where your offline model performance is not predictive of the online performance. And there's sort of a key, especially in this sort of user consumer scenario, there's sort of a key reason why that happens. And it's users can't find a show that they've never heard of.

On Spotify, I will never click on an artist that I've never heard of or that I've never seen. If it's never been recommended to me, if I haven't searched for them, if they're not related to another artist that I already listened to, I'll never hear about them. So there's no way I would ever find them. And that's inherently going to influence your model. So your model just can't know. I can't know how people feel about, if you can expand the you're approaching it as a

strictly classification problem, will this person like this artist or will they like this artist, you can only know that I've liked artists that I've listened to before. And so your offline model might be great. And it's not going to necessarily work once you deploy it. And I think that's how most people learn this is they they get an offline model that just crushes it. It's like, oh, it's not going to necessarily work once you deploy it. And I think that's how most people learn this is they get an offline model that just crushes it. It's like, oh, it's 5% improvement over the previous model. You deploy it, no better. And then the flip side of that, you realize like, oh, man, some of those models that weren't, you know, maybe not slam dunks offline. Maybe they would actually work if we deployed them because they used some new data source or they did something interesting they had some new interesting idea to them so um that's one of the

huge things you have to test everything offline performance does not predict online performance especially in a scenario where there's more items than anyone's ever able to interact with um i think you have to have monitoring around all those things monitoring you know the click-through rate on your algorithms is super important. In the world that I'm in now, we don't deploy algorithms that are used by consumers, but we do have data sources that are increasing all the time. And so we are constantly retraining our algorithms against new data to make sure that they're still hitting the same performance characteristics that we expect, make sure we're not introducing new bias into our algorithms or anything like that.

So there's a lot of different things. And it depends on the problem that your machine learning algorithm is solving. But I definitely think continually monitoring performance is an important factor. That's really interesting. So would you say that a lot of orgs, I guess, undervalue the infra, like how important the

infra piece, right? Because what you're saying, right, is that things might not work in offline as well in online. So we have to run everything through, you know, through prod. But then that means setting up the infra such that maybe you can test out multiple models and then, you know, do your things. But then you also need to, if you're doing sort of a CI, cicd pipeline you also need to have really good testing coverage and you know all these sort of things and then like you said also monitoring but that's also now trivial again

because you need to probably run through some samples and then maybe you have like a you know golden test set that you always run and then you have to look at distributions so and then that that feels like a lot of that just you know it's just infra work right it's not ml specific at all and is that something that i guess yeah that's that's been the case absolutely i mean i know i know that your the your your backgrounds are in data engineering i know a lot of the people that have been on the uh on the podcast come from the sort of infrastructure side and And then you talk to a couple of other people about chaos engineering and that sort of stuff. And it is,

I can't understate how important the infrastructure side is, especially AB testing. But really if you like, I, I don't, I'm sure there's a bunch of different ways to go about it. I don't know. I've been out of this world for a couple of years, but I don't know if there's like a product now that you should use for A-B testing and feature flags and if all that stuff kind of like fits together. But there's really a lot of overlap between those things. And so I think, you know, trying to approach them, I hope everybody's not still relaunching their own. I swear I probably wrote like three or four A-B testing frameworks in my career.

I hope I don't have to do another one. But yeah, it's super important. And like you said, it's such a crossing over. It's such a intersection of the product side, you know, building good monitoring, you got to know what's important about the product, the infrastructure side, the algorithm and data side, it all comes together. So this is something that might be obvious for folks who work on machine learning, but I don't, so it's not obvious for me, but can you share some examples of like,

how do you monitor performance of a model that's in production? Like for, again, for software systems, I know, well, counters, gauges, latencies, I can think of the obvious things, but what are some of those like not obvious or some of the obvious things in the ML world? Yeah, I think, um, I think most of the stuff is similar. You know, you just count the number of people that interact with a given module. And if people are usually, you know, usually 5% of people on a page interact with this module and all of a sudden that drops distinctly, you've probably got a problem there. You know, one of the problems is that things, things are rarely going to go to zero. And sometimes, sometimes a subtle effect

can be, I saw a great talk from someone at Uber once who was talking about all the things they do to predict traffic and all the monitoring that they put in to to try and look for spikes in traffic and all this sort of thing and down to the level of like oh they're a rihanna concert just got out and so now there's a spike of traffic in this area um and i thought that was really cool and that i never got around to to building like that good of a system but that's what i'd love to have is like basically over time what are the fluctuations in engagement with this module because you know it's going to vary by location and by all these features and that's what makes it really hard to detect like you could you could tank your algorithm in texas but if that's one percent of your users

you might not notice that difference in your metrics. You might just think, oh, there's not many great things this weekend, or maybe it's a holiday. It's very hard to tell the difference between holiday or some other thing where the engagement goes down. So yeah, there's no, unfortunately, easy playbook that I know of for like, oh, here's the things that you need to make sure you're monitoring as well. But there's definitely a lot of things you can take into account to get to the ideal system. So being on the infra side, I love the fact that you're talking about monitoring and you actually care about it. In the infra world, we usually have like, hey, there's a production checklist you have to go through before launching something in production. And one of the things

actually is, do you have metrics that you can monitor? And if the system goes down, you need to know, you don't need the customer to find out that your system went down. Do ML teams also have similar, if not checklists, but similar procedures to say, well, before shipping the model, you have to make sure you have the right metrics you're looking at? Yeah, I think there's a lot of, I think, especially on this, on this, more than anything else, at least, like I said, coming from the land where our data engineering team and our ML team were the same team, we were called the data team, we had to do both. Because, you know, we were the team that wanted to start tracking clicks so that we could use it as a feature in our in our recommendation algorithm. So we had to build the data warehouse to be able to store all those clicks somewhere. And so we always wanted, you know, we wanted to track and monitor everything because, you know, if you can monitor something, then hopefully you're keeping track of it.

And if you can keep track of it, then you can, you know, hopefully use it as a feature in your model. And so I think there's a lot of overlap between those things. That's pretty cool, actually. Right. I remember you were talking about this.

So having a team that's composed of both sort of people working on ML, but also data because it is so tied together. How is that your general philosophy in terms of like for most like ML teams? Because one issue I can imagine is getting people that are specifically working on ML to care about both the production aspect as well as how the data is created. But then also pushing people who are working on data to also care about, hey, where are you going to shove all this data into? Has that been a challenge? Yeah, and I've already tried to make the case that I really want to hire machine learning people who really deeply care about the product space as well, right? So now you care about the infrastructure and the product side and the data and the machine learning.

So yeah, it's definitely hard to cover all those things. And I think composing teams of people that care about different aspects of that and, you know, can share that with that knowledge and interest with other people. And, you know, it certainly as you scale, like most of the lessons that I learned came from working at Meetup where we had a pretty small team at times, you know, it was like three of us probably at one point up to, you know, maybe 10 by the end, but like, yeah, so it was pretty tight knit. We knew so it was it was it was pretty pretty pretty tight-knit we knew who cared about what it was pretty easy to remind somebody like hey you gotta let me know when you're doing that thing because i you know i need to do this other thing um and it's hard and i don't know the answers to how to scale this up indefinitely certainly the the the really big

organizations that that seem to do this very well, obviously your Facebooks and your Googles, they just have a ton of... There's a reason why we get paid to do what we do. It's because there's a limited number of people that can do all these things. And Google wants all of them if they could. So it's hard to find people that can put all these aspects together. But as you grow, then you can hire more and more specialized people, build more specialized teams, and try and divide out those problems. But it's definitely good to know at least a little bit of all these different factors. At least know they exist.

At least know that somebody cares about them because then you hopefully won't have a gap in your approach. And our last story, I think, is about you're speaking at conferences about racist and sexist algorithms. So model, fairness, interpretability. Yeah, tell us more. What happened? Yeah, so after a lot of my experience working and deploying recommendation algorithms and that sort of thing, I put together a talk called When Recommender Systems Go Bad. And I went around talking about a bunch, basically had examples of times where companies had built algorithms that then turned out like,

oh, you trained an algorithm on Twitter data, and now it says a bunch of racist stuff. And it's like, well, yeah, you probably shouldn't have trained it says a bunch of racist stuff and it's like well yeah you probably shouldn't have trained it on a bunch of racist data um but then you know from there all the way up to like really scary stuff like models for predicting recidivism like basically models that predict whether um someone that's been in jail is going to commit another crime and and how these things can be impacted by by by race and stuff whether you, whether it's out of ignorance or whatever, that can happen. And in fact, I think as someone who builds machine learning algorithms, I think we need

to have this top of mind in our day to day and really think a lot about how is this model going to be biased, not just in the ML sense. We talk about bias in the machine learning sense of over-relying on certain data or whatever. But biased in the societal sense. And one of the problems with that is that society is biased. And so it is hard to pull that out of your data and get your model to not learn it and to not perpetuate that. But I used to end all my talks by saying, you know, racist computers are a bad idea. Don't let your company invent racist computers. Uh, which I feel pretty, I think

most people would agree with that. Um, but it's hard, it's hard to figure out how, how to do that. And so I don't, I never had all the answers, but I definitely just wanted people to be aware of it and think about it and think how, you know, like there's problems with society. And we probably shouldn't encapsulate that in our algorithms if we can avoid it. Now, did I always do this perfectly myself? Absolutely not. At one point, I think I'd already been giving this talk for a while. And it was at work one day, and one of our community members said, hey, I just got an email from an organizer. And they were kind of offended by the topic recommendations that they got. So one of the ways that we got around, you know, trying to bootstrap our algorithms with

data was we would have people pick topics, we have this big topic graph on Meetup, just things that you're interested in, snowboarding, knitting, whatever. And organizers who were starting a Meetup group could pick what topics the Meetup group was about, and users could pick what Meetup groups they, or what topics they were interested in. That's kind of the base way that we bootstrapped our recommender system.

So the algorithm for this particular user, the person starting to meet a group was trying to start a group for women's business networking. Cool. And then the topics that we recommended, I think the organizer had picked like women business owners or something like that as their sort of core topic. And then from there, we recommended fashion, shopping, skincare, makeup. And that was probably pretty offensive to that person because they were trying to start a business. And we gave all these sort of generically, like, stereotypical topic recommendations. And so, you know, looking at that, we thought, okay, what are the inputs in this algorithm? Why is it doing this, we realized that the topic recommendation algorithm

was based on user preferences, not on basically what so you know, this sort of classic graph walk, people who did this also did this collaborative filtering recommendation algorithm, we were basing it on user preferences. So users that had picked women business owners were also likely to pick fashion, shopping, etc. But groups that were started about women business owners were not about all of those other things. And so we just had to change what data we were putting into that algorithm and base it on the group topic graph instead of the user topic graph and then we got much much less sort of stereotyping and uh and and gendered sort of uh recommendations but

these things are hard there's no like you can't predict all the different ways that it'll happen um yeah that's that was that was one of the times where I had to... It's so interesting that, I mean, what goes in comes out. In this case, you mentioned that an ML model can reflect the reality of the society. The biases or inherent biases that exist in the data itself will come across, even though unintentional. There's the other aspect to it as well, where a lot of the information we consume these days is through recommendation systems or systems which are powered through machine learning, like the movies you watch, the news you read, the ads you see, like all of it. So it also creates this information bubble around you where it kind of influences the way you see the world on a daily

basis. How have you, or do you have any other thoughts on this in terms of like, how has this information bubble been been affecting the society? I mean, we don't have to talk about the elections that happened over the last four years. There are a lot of documentaries about that. But in general, like how as an industry, are we even recognizing that, hey, there is a problem which kind of we created for ourselves, even though unintentionally, how can we make that better? Yeah, it's such a tough question. I think, you know, algorithms are just designed to maximize engagement, right? Everybody just wants people to like their website and use it.

Yeah, because that drives ad revenue or whatever, you know, solves whatever thing you're trying to do. I think maximizing engagement is not an algorithm only thing, right? So newspapers are trying to maximize engagement. TV networks are trying to maximize engagement. And so, you know, when you look at certain TV networks or news sources or, you know, online, some like whatever far whatever blog, like they're just trying to say things that their audience is going to respond to and share or engage with whatever. Another great Twitter follow is Carl Higley. I've never actually met him, but I think he's the best follow in Rex's world on Twitter. So hit him up. He was just ranting about this, like this week, I think, and saying that like maximizing engagement isn't inherently wrong. Everybody like the, the, the, the, the newspapers publishing the story about how algorithms maximizing engagement caused all these problems. Those newspapers are also maximizing engagement by writing that article.

Absolutely, absolutely. So it's really hard to figure out where the solution is here. And I'm not saying at all that it's not a problem. You know, if anything, okay, so an algorithm, we used to talk a lot about the filter bubble. So kind of like what I said earlier, if you want to bring in other data sources to break someone out of, you know, if you only ever listen to Led Zeppelin on Spotify, they don't only show you Led Zeppelin because they've gotten good enough at these algorithms to find the other things that you might like. But with the very basic algorithms, it was really hard to get you out of that filter bubble and only showing you the things that you've already sort of engaged with or listened to. So an algorithm that only ever shows you one type of thing is probably not maximizing engagement enough because it could probably show you other things.

So the filter bubble and the information bubble, I think slightly different problems, or at least they have definitely different solutions. I guess if anything, I would say, and I wish that media and news weren't so intertwined. I think that's one of the problems is that like news and information probably shouldn't be the same sources as entertainment because that incentivizes the news producers to be entertaining. Or at least on the same platform within, I'm sure we're all thinking of the same platform where people go for entertainment and end up getting radicalized. Not great. I also think that one of the things that comes up on these and i'm no expert on these this sort of thing but freedom of speech is the is the big rally like oh you know these platforms are hurting my freedom of speech freedom of speech is not freedom of distribution

there's no rule that says youtube has to promote uh either radical stuff as much as they promote you know whatever your favorite youtube channel is um so i think as these platforms i think if i was working on them i would want to aggressively aggressively deprioritize any sort of hate speech any sort of racism any of those sorts of things because there's no there's no there's no law that you have to promote those things. You don't have to guarantee those people a platform. It's so interesting. A lot of recommendation systems or any of these algorithms to improve engagement are in one way for information discovery, where it's like, hey, you have way

too much stuff you cannot go through. So let us suggest what you can go through to be on the platform. And I mean, there's a show on Netflix, which we don't want to talk about, but it was like portrays all these social media companies as evil. And I feel that it's giving tech too much credit. I don't think we thought this through that, hey, in the next 10 years, when we have all these amazing platforms with internet being ubiquitous in the entire world, we'll have this control over the society in general. We can control opinions. We can show them the world which you want them to see.

And I feel it's too much powers in the hands of engineers who are writing code on every single day without realizing the long-term impact of what they end up creating if nothing else at least i hope us i mean the tech industry just i think this conversation is happening which is a good thing but we realize and recognize that hey this is a problem and we if nothing else we at least need to be conscious about what we are building. And just optimizing for engagement

is not the only metric that's going to help us grow. And I think sort of a side note to that, to me, I feel like the contrast, I really like how you put it, Evan, which is that in the tech world, they have the same objective functions, right? Which is to say, maximize, say, retention or something like click-through rate or whatever.

But then in the old world, like without tech, I think there's a lot more sort of qualitative checks along the way where like, oh yeah, like maybe this is an idea I have, I'm going to implement. And then, but then as I'm doing it, I realize, oh, because it's a person at the end of the day that's doing that right versus in the computer world once you very specifically or specifically say

what your objective is you kind of unless you specifically add also like right checks and guard guardrails otherwise you would just literally like do what you tell it to do, which is to do that. So there's, I feel like that aspect as well. Yeah. And that was something I was trying to, part of that, or one extreme of that attitude is something I was trying to fight against with the talk that I used to give about societal bias in algorithms, which was, you know, so the example we used at meetup and something i was very proud of was we made sure our algorithms didn't combine explicitly combined like gender and interests the the sort of key example for us was coding meetups so if you were to look at coding meetups on on meetup yes a higher percentage of

men were interested in coding than women. But we didn't believe that we should let that impact who we showed coding Meetups to. And we decided as a company, we didn't want to look at gender that way. And so we took a stand. And so we used that idea, that value that we held to choose what our algorithms did. And what I definitely can't abide are the engineers that say, oh, well, you know, that's suboptimal. Wasn't it suboptimal to not show coding minutes? Like, maybe, but like, it probably, it mattered more to us to do what we did. And so I think the same thing is what I would like to see. And if I was working on those platforms, at least, it's hard to hard to know all the factors, but I would want them to take a stand

and say, actually, we don't we don't let this kind of hate speech or this kind of ideas propagate on our platform. And so, you know, we bake it into our algorithms to not do so. Thanks for sharing that stance. I don't think it's, it's easy to take that stance openly. So we appreciate that. So going back to one thing. So I was recently in India, I was meeting my family and my mom has started using some of these social media systems.

And she was like, hey, it's amazing. She recently became friends with a few of our relatives and she saw other people being recommended to her. And she was like, this is pretty cool. How is it doing that? My family's not into tech. So how do you explain, well, let's just say a recommendation system to someone who is not in tech.

Like explain it to me as if I have no idea or I'm a 10-year-old child. Sounds like an interview question. No, I'm just trying to figure out, like, how can I talk to my family or use some vocabulary that is... Because if I go and say it's a recommendation system, they're like, well, what's a recommendation system? And I was like, it takes too much words to explain. And I'm like, hmm, there should be a better way of doing it i think the i think where it gets really tough is when somebody says like oh is my phone listening to me because the other one all the time and now and now i keep seeing now i keep seeing wallet ads and i'm like i don't know if it's listening to you i hope not oh so

so this happened like uh again this this happened recently in india some of my friends had uh an assistant in their home and they interact with it through voice and they're like uh again this this happened recently in india some of my friends had uh an assistant in their home and they interact with it through voice and they're like oh is this listening i see this ad on instagram and i'm like i don't think it works that way uh but yeah anyway hope not yeah as far as how to define a recommender system yeah i mean i think the easiest way is probably just in that just tom collaborative filtering you know they're just uh um no but the easiest way is that is that sort of like oh they they know that you're friends with so and so or like you know you're friends with five people that are also friends with this person so that's how they that's how they can guess that

you're that you know them um now truth is, it's probably a lot more complicated than that. Maybe they have access to your contacts in your phone, because you installed the app, and they know that you have a contact for that person already. The number of factors that could be weighed in are a lot, and honestly, not everybody knows all those factors

and might feel a little... The companies that are building things don't necessarily want everybody to be completely aware of all those things. So they'd have to explain all that. Part of it's just hard. And for the most part, we get used to these things as we go along. Like people used to freak out that they looked at a, at a, you know, looked at some shoes and then a week later saw an ad for those shoes. We've all accepted that by now, right? Everybody has some sense of like, oh, cookies or whatever, right? But yeah, it's

hard to know what the right line is and I think we're getting better and better at transparency and as users understand more and more of where they want to draw that line. And so, yeah, I hope we're moving in a good direction on those things. Makes sense. So we're starting to wrap up.

And I have one question that is probably a little more personal as well. For folks who don't work on ML yet, but are interested in dipping their toes into it, do you have any advice? I love this question. So on one hand, there's tons of ways to go and get exposed to ML and learn some of the techniques and do practice problems. And there's, you know, there's Kaggle. There's so many great ways to go and learn machine learning now. And I think that's all awesome.

If you're hoping to do it at work, if you want to do it in prod at your company, I think the coolest sort of route or the most effective route is to come at it from the problem side. So you know something about a problem that you have on that your team is trying to solve, whether it's a product problem, whether it's an infrastructure problem, any of these sorts of things, you know more about the problem than, than somebody else is going to maybe, maybe. Maybe you have a machine learning team already. Maybe you don't even have anybody at your company that does machine learning already. So thinking about solving the problem first and using your knowledge of the problem is going to be one of the best sort of angles into it. And try not machine learning first.

I tell that to every machine learning engineer. What's the first thing we could do that's not machine learning? So try those things first. And then but at some point, maybe, you know, there's a reason machine learning exists and is so popular is because it can you do happen to like go to your machine learning team or something like that, and you know, want to want to maybe co-deploy a machine learning algorithm for the problem that you're trying to solve. I would say one of the, one of the big things that most machine learning folks have is a lot of, a lot of scar tissue around getting burnt by machine learning algorithms. You know, one of the things that you learn is, oh, this algorithm has 99% accuracy. Awesome, I cracked it. And it's like, no, you didn't.

You broke it. You put data in that it wasn't supposed to know. And constructing the problem of how you ask a machine learning question is a really hard thing to do. And so the worst thing you could do is go to your ML team and say, hey, I wrote this algorithm and it has perfect performance for my problem. Can you help me deploy it?

And they're like, first of all, I like making the algorithms, not deploying them. So don't come and ask me to deploy your algorithm. And second of all, it does not have 99.999% accuracy, because if it did, it probably wouldn't be worth doing with machine learning or it's just broken. And so, you know, developing that bit of scar tissue, a lot of skepticism about algorithms, figure out ways, the better you can understand your problem, if you understand your problem well enough to really know how to test the machine learning algorithm and see if the machine learning algorithm is working or not, that's a really good sign that you're thinking about your problem deep enough to go in and apply machine learning to it. So like my constant referring, keep the problem in mind. Think about what you're trying to do, not just how you want to do it.

I love that advice. If nothing else, I'm going to send this, like, hey, try something that's not machine learning first to a lot of people. Thanks for sharing that. And I can say it's coming from an expert. I am not the one saying that. So we have a question which we ask everyone towards the end. It's what was the last tool you discovered and really liked?

I certainly didn't discover it. I guess probably the answers that I need to give to this I certainly didn't discover it because probably the answers that I need to give to this are things that have existed for a very long time but I'm like finally getting a little good I can finally write an awk command and get it right the first time like every once

in a while. You're a better engineer than I am I mean only simple stuff only like print column nothing crazy but that feels really good I mean, only simple stuff. Only like print column. Nothing crazy. But that feels really good. I feel like that's a bit of a power that I've been learning. I also really like I think it's called

process substitution, where you do like angle bracket, parenthesis, and then you can put in a whole command, and it takes the output of that command and treats it as if a document. It's like having a named pipe. So sometimes it's just a lot easier to construct.

I always struggled using XARGs. And so for certain types of things, I find it way easier to just treat it as if I had a file on disk, but I don't want to write the file somewhere. I just want to take this file and then put it through sort first, but then treat it as a file in the command. So that's what I'm going to go. I'm going to go some real command line. Evan is really trying to impress our audience.

Yeah, we love Bash. Is this good for this audience? Oh, yes. It's the best. Anything else you would like to share with our audience, Evans? I guess I would say to people trying to construct their careers this is just one of my my favorite things i like to tell people you can't go wrong working with people that you like on things that like that you like but you especially can't go wrong working on things

that you believe in and things that matter and so if i could if i could have any influence on the the tech landscape i would say go work on go work on problems that matter. And so if I could, if I could have any influence on the tech landscape, I would say, go work on go work on problems that matter things you believe in, it's going to be good for your career, it's going to be good for your, your, your feeling about how you do things. And it's gonna be good for the world. Oh, that's good advice. Thank you so much for taking the time. I mean, this was awesome. Thank you so much. Thanks for having me.

Hey, thank you so much for listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremisadventures.com. You can also write to us at hello at softwaremisadventures.com. We would love to hear from you. Until next time, take care.