Coding Blocks - Site Reliability Engineering – Eliminating Toil

Episode Date: May 9, 2022

We say "toil" a lot this episode while Joe saw a movie, Michael says something controversial, and Allen's tip is to figure it out yourself, all while learning how to eliminate toil.

You're listening to Coding Blocks, episode 184. Subscribe to us on iTunes, Spotify, Stitcher. I don't have to tell you where to find us. You can find us. We're there, I hope. If we're not, complain to Alan, maybe Jay-Z. I don't know. That's fine.

Yeah, Audible. We never mention Audible, but we're on Audible, right? Yeah, we should be. We should update our notes. You can visit us at codingblocks.net where you can find our show notes, examples, discussion, and more and send your feedback, questions, and rants to – we should drop the email at some point. But right now it's comments at codingblocks.net. It is kind of like an old way to communicate, right? Like it's so old now.

Yeah, just go up to the website and click contact us and and fill it out there, and then you'll get to us. And you don't have to remember anything. It's going to be like one of those days where, like, you know, like, grandma still sends you a card, right? You know, for every major holiday or birthday or whatever, you know, grandma is like, you know you're getting a card, right? And pretty soon, you know, it's going to be like, you know, every major holiday or birthday, you know you're getting an email from grandma, right? That's right. That's going to be the thing. The times there are changing. I actually

went through my Facebook and disabled everything today. I didn't shut down the account. I just turned off the tagging and just all the step one stuff. Nice. We do have a Facebook and you should like and subscribe because

it's great. Yeah, because Jay-Z will never know. Yeah. But if you Slack me, that's the best way, honestly. It really is. And there's a bunch of other people there who are probably thinking about and talking about the same kind of stuff you are. So definitely, that's the place to be, codingblocks.net slash Slack. But we also do have a Twitter at Cutting Blocks

and we have a website with all the dillies at the top of the page. With that, I'm Joe Zach. I'm Michael Outlaw. And I am Alan Underwood. This episode is sponsored by Retool. Stop wrestling with UI libraries, hacking together data sources

and figuring out access controls and instead start shipping apps that move your business forward. And shortcut, you shouldn't have to project manage your project management. All right. So this episode, we are continuing along the SRE book from Google. And this time we're talking about eliminating toil. So this should be fun because I have a feeling we all have some opinions on this.

But before we do that, this is the favorite time of the show where outlaw gets to read the names so oh i have to read i have to do all of them by myself okay i'm up for the challenge we don't have enough of them to to really split them up so you you go ahead and do this okay uh all right well double a from audible thank. And okay. So here we go from iTunes. Franklin McDonough do. McDonough do. I'm going to say it like that. McDonough do. It could be either. I'll buy that for a dollar. Billy VL. Wait, in Roman numerals, though, wouldn't that be like something else? What's L in Roman numerals? It's 5 before 100, so 95, maybe? I don't know.

The 5 is first. Oh, whatever. We're off to a bad start. I could program it, but I can't read it. All right, now we're into Elite Speak, and I'm definitely going to mess this one up. So, Domage? Yeah.

Maybe? No? It's better than I was going to do. Pretty good. There's some threes in there, but whatever. Okay, let's hear yours. What was yours? Oh, Dom 3 AG. Okay. But you were dropping the last three. there but whatever well okay let's hear yours what was yours oh done through ag okay but you're

dropping the last three yeah okay yeah i don't know what i'm doing i mean don't listen to me what about you i got nothing there i got nothing i don't so i don't know pretty good then is what i'm hearing yeah it's only taking like what? Nine years. I'm finally getting the hang of this thing. Yeah. We, we have been around for a minute. Oh, but I got no high hat in my right ear. It's really killing.

Oh yeah. That's so frustrating. It's, it's hard, man. Okay. I get the M and M reference, but I'm not sure like how that relates.

Like I got no snare. I got no snare. It's really the thing that you meant to say. No, no, that's my hi-hat oh okay are you like

the Monopoly guy now you don't get it twisted it was a hi-hat no my right headphone thing isn't working so I'm hearing you guys

out of my left and it's like oh man oh that's what you mean okay because because like I wasn't sure

where you were going with this because this week it was announced that like all the candidates for the induction into the Rock and Roll Hall of Fame were announced. And Eminem is one of the candidates. And so that's where I thought you were going with it. I don't know if you know, but the Rock and Roll Hall of Fame has all kinds of people in it. No, I didn't. NWA is in it.

Dolly Parton is in it. It's called the Rock and Roll Hall of Fame, but you know. Well, why have a Country Music Hall of Fame and all that then? That doesn't make any sense to me. The way they induct people is really suspect too. It's just ridiculous. Okay. All right.

That's fair. Yeah. That's fair. But I swear there was something I was thinking about too, that I wanted to bring up here, but I totally cannot remember. So I guess I kind of take issue with you saying that Eminem isn't rock and roll. I mean, there's a,

there's a comment to the right. Is that what you meant to in the notes? Uh, no, not that one. Although it was good. One of the reviews, um,

he, he modded our, one of our dad jokes. Um, and basically what, One of the reviews, he modded one of our dad jokes. And basically, what was it? Something like, what do Spanish programmers write in? And he thinks that the answer should have been Lisp. Because in true Spanish, Spanish, in true Spain Spanish, the THs sound like Ss. So that,

that was a pretty good addendum or, or a modification to a dad joke. Yeah, I'll take it. Yeah. Thanks for that. So I can't remember what I was going to say. So I guess,

Oh, Oh, I do remember. I've made a Saturday meetup folks. I made it to our coding blocks. I missed it Saturday. Yes. Nobody else came there,

but I was there representing for a change wait you were literally like the only one on the call that's worse well i mean it's good to talk to yourself on occasion right like monologues are healthy in some situations but no no i mean um there there were a decent number of folks there so um definitely had a great time so appreciate everybody humoring me while i drove around picking up furniture you know dropping the phone running off so yeah that was funny yeah good time i made it it's been like the first time in six or seven months so yeah nice well if we're gonna if we're gonna like modify dad jokes how about we start off with one let's do it how do locomotives know where they're going? Choo-choo-choose.

Sounds like choose. Lots of training. That's pretty good. That's pretty good. All right. All right. So I guess we'll go ahead and get on with the show here. Oh, we're done with the toil. We and get on with the show here. Oh, so we're done with the toil.

We can get on with the show? You know what? I don't think in the history of the 183 episodes of this show that you've listened to, we have ever said the word toil. Tonight, folks, we're going to say it a lot. We really are. We're going to say it with some vehemence and some gusto it's such a really truthful comment though because like how often have you used that word ever in your like you know your daily lexicon like how often does that word enter into something you were going to say well tomorrow it will be at work i used it today i used it today with the call with jc it was hilarious

yeah and it actually fit i was like oh snap chapter five what up that's awesome all right so let's get into it the toil um what is it it is simply work you don't want to do which is amazing i mean i guess most things in life are toil i don't know literally the first sentence is it's not just work i don't like to do and you're saying it's work is work no no it's what you don't want to do yeah it's totally work you don't want to do but it has to we define it a little bit better further down um, it's not just admitted like administrative work or tedious tasks. That's not necessarily toil company-wide training. Yeah. Like that, that is not toil. That's required stuff that you have to do. So you can't call it toil. Um, like they even talk

about, right? Like some administrative work has to be done, whatever it is. So those don't count HR trainings, meetings, all that kind of garbage. You're going to have to be part of it. Can't do anything about it. Um, and they even say that some things that you do that pay long-term dividends, even, even though it seems like it's just really, it's brutal getting through it. That also is not toil because it does pay long-term dividends. One of the examples that they gave was like, if you're cleaning up service configurations that have just been like hanging around for a while and needed to be cleaned up, that's not toil because it paid dividends. An example that I was thinking of though, would be like,

if you, if you have a build script to like build your you know next version right but you'd never automate that thing and you just like hey every monday uh i'm going to issue a new build i am going to actually run that and enter in all the parameters that are necessary to create the new build yeah totally i'll give you a real life one that just came up this past week is, you know, we have like a rotation of on-call people and the system that we use is so bad that only a handful of us can get the listing of who's on call any particular day. So like one of our team members is like, yeah, I'll go in and screenshot the thing and share it every week. Right? Like that's toil. That's, that's, that's something that should be automated and shouldn't be difficult,

especially given the amount of systems that everybody has access to nowadays, but whatever. So to further define toil though, here is it is work that is oftentimes manual, repetitive, can be automated, has no real value, or it shouldn't be done that way, and grows as the service does linearly. So as the service scales to more users or more requests or whatever, you need to add more tasks to manage it, that's toil. And it's something that a person does, right? So if a person has to come in and do like what I said, screenshot something and send out a thing, that's toil.

I mean, a machine could do my HR stuff. That would be amazing. Matter of fact, I would probably pay good money for that. I could totally like, you know, let it watch the video. Did it learn anything? I don't know.

I have a really good example of this. And this isn't even related to what we do every day in terms of programming and stuff, but it falls in the same sort of bucket if you are somebody that has a bunch of WordPress sites. When there's an update that comes out, it's gotten better about it. Like it'll auto-update and do that kind of thing. But it used to be you would have to log into every site, update WordPress. You'd have to then go to the plug-in page, select all the plug-ins, update it, right? Like that's stuff that really shouldn't require human intervention, right?

That would be toil. That would be considered toil if you're managing websites. human intervention, that would be toil. That would be considered toil if you were managing websites. That's something that should be automated. Sorry, I was tweeting. Wait, I thought you canceled all that. No, I'm still on the bird site. Yeah, I was trying to think, but I guess it doesn't really apply because I was thinking about, you were talking about the WordPress updates

and everything because immediately my mind went to, oh, well, remember the administrators? They would be like, oh, well, I'm not going to let the updates go through until I've had time to vet them. And you're like, did you vet them? Right. Yeah, really? What did you do?

Yeah, exactly. What did you do? You're like, oh, it installed successfully, so okay. Yeah, Really? What'd you do? Yeah, exactly. What'd you do? Just like, Oh, it installed successfully. So, okay. Yeah, that's good. Like a GKE update. Everything's fine. Yeah. It's fine. Um, yeah. And so if something is repetitive and it's manual and if it can be automated, then you want a machine to do it, right? Like that's awesome. The, the only thing that they kind of leave as a caveat here is if

it's something that requires like human judgment or, or thought to take into consideration here, then, then that's not toil, right? That, that might be something that you just have to do because you need to look at it and make a decision. Code reviews. Yeah, that's a great example. Not an example of toil. Right. It feels like you totally have that thing auto-merge. That's actually a really good example, though.

We've got all kinds of build gates and things in place, right? So if you put in a PR, you can't merge that thing until it passes a build and it passes the merge properly yeah that doesn't the build includes test don't don't don't shortcut it right um but that's actually a really good uh good point right like you want somebody to pass judgment on it before you let it go in because somebody might look at it and be like oh you missed something missed something there. And while it builds, it's really going to mess up the system, right? So that's a great point out.

And tactile, so one of the things that humans are good at is strategy, making the interesting decisions that kind of govern the long-term directions that you want to grow. So the things that are toil are things that are tactile, so things that are more tactics based interrupt driven have to do with the kind of moment to moment operations. These are things that you may not be able to eliminate completely, but the goal is just to kind of minimize those sort of minor decisions. And they also say if there's no enduring value, right?

Like if your service didn't change state after you did something, if you didn't make it better for the long run then that's toil right like that yeah they said oh sorry no you're good i was saying this is something they keep coming back to and that you mentioned a phrase that they keep using too in the beginning of it pays dividends it means you do the work now and then you reap the benefit uh you know over time so you may spend a couple hours to set up those scripts or automate that thing but then you save an hour every week. So you may spend a couple hours to set up those scripts or automate that thing, but then you save an hour every week or you save 15 minutes every day or something like that.

It pays dividends. And I don't think they called it out explicitly in this chapter. I don't remember seeing it. The thing that really stinks about toil-type items is it's not that it just takes time. Usually, if you're manually doing repetitive tasks, your chance of making a mistake in those tasks are really high as opposed to like when you automate them, right? Like you automate things because you've, you've done that series of steps multiple times. So you know exactly what you need to do to make that happen. Whereas if you're doing

it as a person, somebody comes over and interrupts you in the middle of it. Maybe you forgot to copy and paste that thing you needed to do, right? Like there's, there's so many things that can go wrong in that, in that loop. Yeah. It always works out to where, you know, you don't know at first. So you're like willing to, you know, manually do it a few times, but I don't know about you, but like there's, it doesn't take too long before I'm like, okay, I'm at least going to like write a script for this or something so that like I can just reuse that thing. And, you know, depending on how complicated it is, like we've talked about like using your command history to like, you know, uh, control R and you search through your history. So if it's super simple, like maybe that's good enough, but, but you know, if it has any complexity, especially like with things that are

changing, like values that are changing, like before long, you're like, okay, let me just script this thing out. And then if you ever noticed, like in the beginning, you're like kind of just having fun with like trying to automate that thing, that task. Right. You know, but then over time you like start to take it for granted because you're like oh my god it's so much better like i you know why did i not start with this but you never start with that you never do you always like iterate your way to it because you don't know what you need until you like get into it right yep it's like the first time you do something you just kind of do it the second time maybe you take some notes third time you script you know save the scripts the fourth time you really know and then yeah that's that's uh how i tend to go and then uh you know

i always wish i'd done it the first time but then again there's tons of scripts i've written i've never used again so that's pretty annoying as well so yeah it's true i mean it's hard to find the right balance right um but yeah this is also where i want to give a shout out to powershell though like we talk about shell scripts a lot here and I truly hate shell scripts. Like with, with an undying burning passion, do I hate shell scripts because getting parameters and doing all that, it's just dirty. I wish that I did PowerShell on everything because anytime I've ever dealt

with PowerShell, it just looks readable and it's easy to reason about. I don't know how you guys feel about it but oh that's great it's the best just like little stuff like uh parameters like having an easy way to declare parameters to a script and like if someone runs a script without it'll guide you to make you enter the ones that are required or not you get auto complete on the arguments like stuff like that uh if you want to pipe something in you know you get strongly typed objects it's wonderful yeah it's wonderful yeah i can't tell you how many times i i google like how do i tell if a variable is like null and bash and you're like okay well wait wait a minute uh because null or empty string is different in some cases if it's zero and if it's bash it's this if it's fish it's that it's well it's like oh my gosh yeah man i i kind of hate

shell scripts i do them a lot but i i really i always consider how bad would it be to add power shell to to my container or whatever right like is anybody gonna get that mad if i do it well let introduce you to your new build system yeah just all the way down hey veteran python that's you know what i oh man we're getting some hate now come on yeah hey i would rather do python than shell all day all day every day right um but at any rate all right so back back to this so the last little bit right, to identify toil is, we're going to go big O notation. Again, we haven't done this in a minute, is if you've got O of N growth with the service, right?

So we already mentioned it. If as your service grows, so does the amount of time you're spending on tedious tasks, that's toil. Yeah, like if you had to every time, let's say that you had a SAS service, a software as a service kind of offering. And every time you wanted to onboard a new customer, you had to manually do something right in the beginning. You're like, okay, whatever. Like, you know,

I brought on the first customer, I brought on the second customer, not such a big deal. Once you start bringing in like a hundred customers a day, then that becomes a big chore if you're still having to do that thing manually, right? Yep. We even talked about in previous lives, like looking at the load on servers and that kind of stuff. And if you're having to constantly monitor loads to figure out if you need to add another server to the stack to, to make it handle the, the request coming in, that's toil that,

that should be automated. Can we just admit Kubernetes has spoiled us? Like, Oh, no doubt. You know, think about like how we used to manage servers before, you know,

and like, yeah, like looking at like, you know, what's the utilization, what's the disc IO on it? What's the network IO on it? Like, you know, I think we're okay for a little bit, you know, but you know and like yeah like looking at like you know what's the utilization what's the disk io on it what's the network io on it like i have you know i think we're okay for a little bit you know but you know before like a big sale comes along like you know as we get closer to christmas like we probably want to like bring on you know a few more but now we're like whatever i don't care

you know it'll figure it out i love I love it that we derail so easily. Did I do that? I feel like Urkel. Did I do that? There's a dated reference for you. I just saw him in a movie that came out. Really? Yeah, like last week. Jaleel White. Wait, you watched a movie?

Yeah, it wasn't that. Yeah, it was not my choice. It's not a good movie. I'm not even gonna say the name of the movie. Hated it. Hated it. That's so awesome. I gotta know the name of this movie now.

Right? Yeah. I gotta know the name. He'll find it. He doesn't even know off the top of his head. That's even better. Oh, it'll be like some major blockbuster too.

Like, you know, the new doctor Strange or something. And Jay-Z is like the one person to say like, oh, I hated it. You know what's even better about this is I derailed from our derail. So going back to Kubernetes, you know what I actually like about it? Not the system itself. It's the fact that when you are designing software, you start designing in a way that makes

it be stateless, right? Like back in, in, I say back in the day when you'd write software, you just kind of wrote it knowing that, you know, the thing's going to be running, server's going to be fine. But when you write in Kubernetes, you have to know that that thing can just get blown away at any moment and it needs to be able to recover. And so you start just in prison intrinsically working that way. And and so your software as a whole becomes more stable because, hey, if Kubernetes kills that pod, you know, it's going to come back and it's fine. It's just going to pick back up where it was. Right. So that's I kind of like it, that it forces you to think about your software in a decoupled restartable state.

It reminds me of like I remember back when we were going through the 12 factor app series and like there were so many great things about it, about that series that like at the time I was thinking like, oh, I want to live in this world. This sounds amazing. But it also seemed like so unattainable. But now like in this, you know, Kubernetes world that we live in, it's like, oh yeah, like this is a lot of the, you know, there's similar concepts. There's a lot of overlap between the, this and the 12 factor app. Totally. Hey, and by the way this and the 12 factor app. Totally.

Hey, and by the way, Jay-Z did post this movie. I don't feel like we should say the title of it because it's got a few reviews and it's not high. Yeah. Good. Yeah.

It only has a few reviews here. So why, why? Okay. It's called the greatest inheritance. Yeah. Why? So the movie tricked me. Iest Inheritance. Yeah. Why? So the movie tricked me.

I won't go into why I hate it because I can talk about things I hate all day long. But the gist is. What made you go to it is what I'm trying to get out. You saw this and you're like, oh. Yeah, the preview looked okay. So I acquiesced. But the deal was someone dies and they go to the funeral. And one of the kids asked, so where's the will?

And there was a stipulation in the will that said if somebody asked about my will within five minutes of my funeral, then it kicked off this whole kind of almost like a survival kind of survivor kind of adventure where they had to kind of like the kids were trying to do these like little challenges around the house in order to try to find the will and inherit the property. And it looked like it was going to be funny. And will and uh inherit the property uh and it looked like it was gonna be funny and it was not funny it was like it was like a hallmark movie but the preview made it look like a comedy it was not it was like i don't know so let me let me let me just go ahead and tell you i mean this should like you need to look at reviews jd it's got a 3.4 out of 10 on imdb now without looking don't cheat do you care to guess the rotten tomato score on this movie oh i looked at that already it's a five no it doesn't even have one oh oh oh well then i don't know people are even watching this movie to give it a review jay-z like that let's be honest it's not because you thought it was gonna be funny it was because

you saw urkel and you were like you know what that's a throwback i didn't know it was urkel that's the i was like watching the movie and i was like holy crap that is urkel and then i for some reason i remembered his his name, his real name. And I was like, that's Jaleel White. I know this. And we looked it up on IMDB. And then we're like,

wait, this movie sucks. What are we, what are we doing? Oh yeah. Holy God. And like the first view, uh,

basically says the same kind of thing. Like you got tricked. Like the preview looks like it's comedy and no, it's like heartwarming and touching or something. It's gross. Don't nobody want that. Yeah. It's a family movie.

I know I've got my family. We've got to watch comedy. I want to see like brothers and sisters, like, you know, yanking each other by the hair and crashing cars and stuff. I don't know. That's amazing.

All right. Yeah. Digression. Sorry. It's a good one. All right. So, so getting back into this one then so why is less

toil better anybody i don't know i'm still thinking about the movie yeah i got i got totally derailed because i was thinking like the type of movie that jay-z was describing sounds more like knives out i would i would watch times out I've heard it was good. It was pretty interesting. All right. So, all right then. So you're going to ask a serious stuff. Okay,

fine. Why is toil better? Less toil better. All right. So this is kind of cool. They talk about at Google, right? The goal is to keep each SREs toil at less than 50%.

And when they say to keep their toil at, that means the amount of tedious, um, non amazing work at less than 50%. And when they say to keep their toilette, that means the amount of tedious, non-amazing work at less than 50% of their time, right? Like stuff that doesn't pay future dividends. The repetitive stuff that should have been automated. Yes. The other 50% of their time should be developing solutions

to reduce their toil even further, right? Or make new features for a service. Now, this was interesting because when I read that particular part, I was like, wait a second, are these feature devs or are they SREs? And they actually go on to explain that in their next bit, which is features for a service are improving reliability, performance, and utilization, right? So it's not making a brand new thing for the service. It's making sure that service is better overall. The point is, is that if you spend more than 50% of your time on toil, which is stuff that you can automate but haven't, then that's just taking away

developers' times. And if you were to let that go on tractor on, you know, uh, monitored or whatever, and let that expand out to other people within the team, you could eventually have like entire teams that are just, that's their job is whatever the toil is. And it's easy for that to happen too, right? Like we've seen it before. Like a lot of times, like if you are a person that's like a go-to person, you know, the system or whatever, it's easy for you to get roped into helping with every single thing that comes along. And so all of a sudden your toil, whether you realized it or not is growing, right? And now you're spending 70, 75% of your time looking at things that should have been automated in some sort of fashion. It kind of reminds me of Scott Hanselman because remember,

I remember that he had this approach to where he was like, well, you know what, if I'm going to write it down once as an email, then instead I'm going to write it as a blog so that I can just share it with everybody. And that way I never have to say it a second time or write it a second time. Yeah. The key strokes, but they, but they do go on to say it a second time or write it a second time. That's a good point. The keystrokes. But they do go on to say, you have to be careful because there are people who would be happy with that toil.

They will shovel that same ditch every day, day in, day out, if that's what you're going to pay them to do. And they will happily do it, right? Yeah. It's just not a good use of that time and money. It doesn't help the company is the problem. Right, right. I was just trying to think, have we really given many examples of toil?

I think we kind of maybe take it for granted. We've given a few, like the GitHub thingpress i mean um i would say even deployments right like if you have a deployment that that you constantly have to interact with uh script example the build script yeah yeah so we have the show for joe yeah we can do that he's still thinking about the movie joe in summary what we're talking about that movie yeah movie. Well, I was just thinking about stuff like the show. I was literally looking at the show and I was thinking about what it takes to actually produce a podcast. There's a lot of stuff that happens. I've been lazy about automating, like setting up the links, creating the post, scheduling.

There's a lot of stuff that can be automated there. And it doesn't take ultimately a lot of time each time to do it but you know pay a dividends to do it but it's you know a trade-off so i don't know i'm just kind of thinking i'm thinking myself in circles is like a lot of times the things that you don't automate like what i was thinking like what's the toil that people don't you know if we're saying why is this toil better like what's the toil that people are most reticent to to like let go of and it's stuff where things are important or you don't do very often or you know they're kind of mission critical so it's again like things like releases uh end of sprint type stuff like sometimes you'll have

people go like write you know queries to kind of pull some stats and uh and things change you know maybe you know every two weeks whatever so it's processes that are kind of evolving and the downside of kind of automating is that it kind of locks you into processes sometimes. And sometimes those processes, like you get enough stuff built up around them and it's like casting them in concrete and then you can't change. That's true. So,

yeah, but I think so that actually sort of bleeds well into the next point, which is when you're spending your time reducing toil, that's the E and SRE. That's the engineering. And it's that time that allows you to scale the service with less time for an SRE or anybody else to keep it running proficiently. So like what you were talking about is when you build up processes around something like a release or whatever it is, it may seem like it's in stone, but if you are spending time engineering solutions

to make things better, that could be part of your time to make it not so set in stone, right. To, to, to free it up. And I think that's the important part is it's not like once you build something, just like in, in any kind of developer position, just cause you built it the one time doesn't mean it can't be touched again. Right? Like you should be able to go back in and, and make it better somehow if you need to so i don't know that's the human element right that's the thing that's the human decisions that are on strategy that need to happen like you can imagine like if you just had this company and you built up stuff a couple years ago around like mesos that was like a kind of a automation platform for you know releasing type stuff and you build up all

this automation all these processes all these wikis all these on mesos. And then nuke it on the block. Kubernetes comes in and people start swapping over. But now you've invested so heavily, but that's where you need a human to kind of say, well, uh, you know, it's time to change.

And you have to be disciplined about in smart about when to make those changes and when to, to not. Do you remember swarm? Does anybody remember swarm you know it's still part of docker right like you could still have it it'll ask you hey do you want to convert these things into a swarm no thing or whatever yeah no exactly yeah it sounded like you had promised but i mean kubernetes just caught on too strong um so they also say here when google

hires an sre it's like their promise to the SRE that they're going to keep their toil at 50% or less. And, and Google takes it very seriously. They try very hard to ensure that the group doesn't turn into a full-time ops organization, right? Because they hired people that have software skills to help solve these problems, right? Like they don't want them doing a bunch of repetitive tasks. You know, let me see if I can find this again, because I found this interesting thing. I don't know if you agree with this, because when you said the ops focus team, that's what made me think about this. So I was doing a Google search just to see like what else SRE related

might come be out there. And, uh, you know how Google will give you like other suggestions. Like people also ask one of them is what, what is SRE versus DevOps? And, uh, the little synopsis that came back and it was in a nutshell, DevOps engineers in picture, like a Venn diagram, right? Where there's like a, just a little bit of overlap between SRE and DevOps in a nutshell, DevOps engineers are ops focused engineers who solve development pipeline problems, development pipeline problems, site reliability engineers are development focused.

So not ops focused, but development focused engineers who solve operational scale and reliability problems. Now agree, disagree. I think I disagree only because, so I joked with you a lot about it. Just see Twitch back when we were doing the DevOps stuff, right?

And it wasn't just you, right? Like our buddy Bobby, right? Like, I just love to see him be like, no, you know, seven minute abs. Um, the, the whole DevOps as a role thing or as a title, I still sort of agree that it shouldn't be, right? Like everybody should be involved in it. So I guess if you're saying a DevOps engineer, maybe. I don't know. But I see DevOps as you're a software developer who has to be involved in the operations, right? But I also see that kind of being the biggest difference to me between

DevOps and SRE, at least from the things that we've read and things that I've seen is DevOps is typically a developer is software feature-based type stuff. And SRE is literally on stability, performance of the software that was already written, right? Like in, in writing software to make that experience better. So I don't see them in the light that that thing said, but it sounded like you just kind of summarized it though. You said DevOps was more operations focused. I'm saying they're more focused engineers who solve development pipeline problems, right?

That's what the SRE sounds like to me more so than what the... I put the quote in the notes there for you. You could see it. Let me read it again. Yeah, no. No, I'm saying DevOps. So again, we've talked about it. Like DevOps as an engineer title doesn't...

I mean, maybe it exists out there. But when I talk about DevOps, I'm talking about software developers that write software that people use. And then you have to be involved in the operations because you're the one who knows how it works, right? So it's not fair to hand it off to an operations team, not ops focused. You are focused on building software that people use, but you're involved in the operations to help the operations team be able to deploy it, deliver it, whatever. So that's where it's different for me. I don't think they

are ops focused. There are ops involved. SRE seems to be exactly what that says, which is they are ops focused and focused on making the operations better through software. So, well, here's what I'm going to do. I'm going to have this quote in there in the show notes along with the link, and I'm going to say, Alan approved. And, you know, I'm probably going to quote it as Alan said this. But, you know, if you're listening to this and you're like, whoa, wait a minute, I totally agree or I totally disagree, or I almost said you totally don't disagree, put a comment in the show notes for this episode. This is episode 184, so codingblocks.net slash episode 184.

And you can have a chance to win a physical or audible copy of the book. How's that? That's right. That's right. Good call. Hey, what did you think, Jay-Z,

this DevOps engineer versus SRE? I think they're, they can both be roles. And I don't really, yeah, I mean, I guess I do think of DevOps as being more associated with actual, like continuous integration pipeline.

And the SRE seems to be more about kind of heavier on the ops and just more almost like serving the business. So there's a lot of things that we didn't talk about with DevOps, like figuring out budgets for things and almost like release schedules and things like that. So if we're all going to give our take, I would say that I think that there's a lot of overlap between them and that, um, the DevOps is more of like the culture of like, Hey, we're gonna, you know, automate things and we're gonna, you know, we're, we're in, you know, Alan said we're ops involved. So, so, uh, coin that sir. But, but the, um,

the SRE, I think, you know, there's a lot of DevOps principles that they're following too, but they are definitely focused on like, okay, how can we take what was created and make it in better for at scale and, and improve the reliability.

That's what I've gathered so far is the SRE responsibility. Whereas the feature teams that are developing new features, they're definitely going to... I would assume that they would have in Google some very DevOps-centric minded processes, even for teams that are developing new features. Right. But it's not, I don't see it as just necessarily build only related. I think one of you said something similar to that.

Yeah. Yeah, I did. It's just kind of my biases. He's trying to try to put people in a corner, put baby in the corner. All right. So the next part, I actually like this one quite a bit. Calculating toil. So here we go. Math on my chicken. That's right. You're ready for some numbers. That's right. 1% baby some numbers. That's right.

1%, baby. It's going to be addition and division. It's something. So they gave an example. And what's interesting is they kind of go hand in hand. They give an example of a six-person team. So it looks like for every person, they add a week cycle to it. So if you have six persons, then you have six weeks and that's your cycle. So they said, you have to assume every, every person has one week of primary on call and

one week of secondary on call. And that means then that each person, two of their six week cycle is going to be on call with interrupt type work, which would be toil, right? And so that means that given that ratio, two out of six, you're spending 33% of your time on toil. At most. Oh, at most is what it's supposed to be. If you have an eight-person team, then again, two of that eight weeks is going to be on, you know, interrupt based work. So now two out of eight, you're at 25%. So again, that should sort of be now they didn't say it most.

They actually said that's your floor, right? That's, that's the, uh, that's the least amount of toil you'll spend your time on. So you're guaranteed when you're on call that you're spending those two weeks on toil. And then there may be additional toil afterwards if there's a problem or an outage or whatever. Right. So, so those are your floor.

That's actually what they mentioned. Was it that way? I thought it was like, that was the most amount of toil that you would get. Cause like, hopefully you're not. Toil lower bound is what they called it. So you're on call weeks as a port, as a percentage of your total week in the cycle is your lower bound is what they called it. So your on-call weeks as a port,

as a percentage of your total week in the cycle is your lower bound because that's kind of what they got at is when you're on call, you're not really doing any engineering work. You're looking at logs, you're making sure things are good, all that. Right. Um,

well, it must be different than my on-call cause, Oh wait, right. Yeah. No, you're all call is we're working and we're managing toil i just can you imagine you're like a google interviewer like well i mean come on now

it's done i mean come on when you're on call you're not really on call right right but but i did like the uh the i don, this was totally the wrong thing to take away from this part though. But I was like, Oh, a whole week on. And then, you know, you're done. Right. Right. Just get it one and done it out of the way. Yeah. And I was like, Oh, imagine, you know, like, but there was also something to be said about it again, take the wrong, complete wrong takeaways. Right. Cause I'm like, well, if you, if you were on call for that long, like you would feel some pains, right?

You would know like, oh, these are the problems, right? Whereas if you only did it like one day out of the month, for example, it'd be really easy for you to be like, oh, I got it on a Sunday. Like, you know, yeah, exactly, exactly. Yeah, that's a really good point. I didn't think about that. I just thought one out of six. I had the opposite. I was like, one out of six. I had the opposite.

I was like, one out of six, I'm on call. But, I mean, it's great to be able to see the same kind of problems. Like, that's really interesting. Also, like, by the way, how you're like, here's a bad take. Let's double click on it. Just saying. Wait, there was a bad – I clicked on what just thought you're like here's the bad take this is the wrong takeaway oh let's go let's dive in it was funny i thought

it was funny yeah yeah yeah i like it i although i do like that whole that i think it was a good takeaway the bad take is a good takeaway because what you said is totally legit. If you spend a week in pain, then that will probably help drive the strategy on what you plan on engineering when it's, when it's time to go fix some stuff. Right. Now, this is when I thought that that 25 or 33% was like the upper bound. I didn't, you know, I, I misread that I guess. Cause I was reading as I was reading, I was had this other fantasy in my mind of like oh, yeah, you would actually get to see it. And I missed the lower bound part, but yeah. Yeah.

So check this out. They actually do internal surveys at Google to find out what people say that their actual toil time spent was. And so the SREs reported that their toil is spent most on interrupts, which are non-urgent service-related messages. Then on-call urgent responses, so when things go wrong, and then releases and pushes. So that's kind of good. That's sort of what you'd expect or what you'd hope to hear, right? And then they also said that on average, which I thought this was hilarious, especially after

the previous chapter where they said never use use averages because they suck. Yeah. And then they're like, here's some averages. Here's an average. I was like, oh, yeah. All right. That's fine. You try to bury some numbers. But what they did say is the average SRE spends less than 30 percent or close to 33 percent of their time on toil.

So about a third. Right. So in that six week cycle, they're really only spending two weeks on it um but they did mention that they had outliers right yeah it was interesting they said some people uh actually spent zero time toiling which is pretty cool and some people went as high as 80 which is you know interesting that it wasn't higher yeah so the person who got zero time toiling like their week happened to always line up on company holidays, like Christmas or something or Thanksgiving. And they're like, oh, it's so hard. Yeah.

I was wondering, like, when did they like when did they survey these people? When did they watch? Like, was it over a two week period? They happen to be on vacation, you know, or like, was it over a year? And if it's over a year, like, think about someone measuring, like, how much time you spend in jira and how much time you spend in the wiki like oh for a year does it did it not make you think that maybe these people that spend zero percent of time on toil are the ones that everybody's like i don't want to deal with that dude yeah oh yeah absolutely absolutely or this guy this girl whatever like no i don't want to talk to Yeah, I figured like instantly my mind was like one of two things.

They're kicking back and making all the strategic decisions for everybody. Thanks. Or no one wants them to touch anything. Right. Thanks. They did say, though, that when they find that somebody or a team of people are taking on too much toil, that it's up to the manager to fix that, right? To spread that out somehow. So, you know, they're constantly evaluating it, which is, you know, kudos to Google for

doing that, right? Because ultimately it helps them as a company. Yeah, it was good. This episode is sponsored by Retool. Building internal tools from scratch is slow. It takes a lot of engineering time and resources, so most companies just resign to prioritizing a select few and settling for inefficient hacks and workarounds for every other internal business process. Retool helps developers build internal tools faster so they can focus development time on the core product. Retool

offers a complete UI component library, so building forms, tables, and workflows is as easy as drag and drop. More importantly, Retool connects to basically any data source, database, or API, offers app environments, permissions, and SSO out of the box, and offers an escape hatch to use custom JavaScript when you need it. With Retool, you can build user dashboards, database GUIs, CRUD apps, and any other software to speed up and simplify your work without Googling for component libraries, debugging dependencies, or rewriting boilerplate code.

Thousands of teams at companies like Amazon, DoorDash, Peloton, and Brex collaborate around custom-built Retool apps to solve internal workflows. So to learn more, go to retool.com. That's R-E-T-O-O-L.com and learn more today. All right. Well, which one of us are – it can't be Jay-Z this time. It just can't be. Like I think we learned our lesson.

It worked. It did. I smashed can't be. I think we learned our lesson. It worked. I did it. I smashed that zero stars. Yeah. Real quick, if you have bad reviews, go ahead and leave it. We'll love it. What? No. I just did it. Boom. Stuck it in there. Done.

So let's cut his mic off. Never let him speak again. Can you just go ahead and mute him right now? There we go. And I will be sure to cut out all of his audio from the show. But if you haven't already, you could leave us a review. We would greatly appreciate it. You can find some helpful links at www.codingblocks.net slash review.

And with that, we head into my favorite portion of the show. Survey says. All right. So a few episodes back, we asked, what percentage of time does your team devote to technical debt per release cycle? This one actually lines up pretty well with, you know, this SRE topic, I thought. All right. So your choices were 100%. It's all we do. I don't even know if we have customers. Or 75%-ish, we don't care for new features. Or about 50%. We're equally slaying last release's technical debt while we introduced this release's technical debt. Or around 25%. We're equally slaying last release's technical debt while we introduce this release's technical debt. Or around 25%. We're accumulating technical debt faster than we're paying it off.

Or roughly 10%. We've got too many new features to deliver to care. Or lastly, technical debt. Why would anyone address that? You'll completely rewrite the application before it comes due. So one blah, blah, blah, 84. So I forgot the rules. So it's Jay-Z's turn.

Okay, good. I know the answer. Oh, you did. Okay, good. The answer is about 50% because, you know, you got new stuff you're doing, but also you're dealing with the problems from last release. And I don't think that's 50% fixing it. The question was whether you devote time to it.

So that includes toil in my interpretation. And I think a lot of people are dealing with a lot of toil related to decisions they made in the past to get stuff out sooner. So I'm sticking with 50%. And I'm going to say 50%. Oh, come strong. That's right. All right. You thought about that way too deep.

I don't think anybody else did when they read this. So for that reason alone, I'm going to say you're wrong. Okay. You're going to go with probably, you're probably right. Let's go 25%. And I'm going to go 25% sticking with the,

the times here. Okay. So 50, 50 and 25, 25. There we are. And it all adds up to 50. Not really.

Yeah. Right. Carry the one. Yo. Yeah, it does. Yeah, it's pretty close.

Uh, yeah. Well, um, so Jay-Z won and lost all at the same time. Wow. So it was 50%, about 50%, but it was 38% of the vote.

Oh, that's impressive. So pretty good though. I mean, if you round up, it might be like the first time Jay-Z's ever won in the history of the show. It might be. While still losing. That's how I do.

Chaos engineer. Chaos engineer. All right, well, how about this one? I'll give you a quick joke before we go, because I just wanted to let you know that our wedding was so beautiful, even the cake was in tears. Wait for it. Alan's thinking. I see the eyes looking around

like, wait, what? Cake was in tears? Oh, I got it. That's my favorite part of these. Yeah, I got it instantly because, know good idea that's how you win and lose at the same time of course he got it i was thinking costco's got good cakes look at that trace call my jay they do have good wait over public's though because oh not even i mean public's not even close wait I mean, Publix. Not even close. Wait, Publix has one kind of cake. Publix is not close to Costco?

In terms of their quality of cakes? No, sir. Okay, come on. This is where I just know that Alan, he's drinking the Costco Kool-Aid, and he's like, let me just tell you. Ask anybody. Everything there at Costco is better. Ask Microgy.

Ask anybody, man. The tires at Costco are way better than tires you'll get anywhere know ask anybody man the the tires at costco are way better than tires you'll get anywhere else you want no the same tires a hot dog you want a cake you want whatever broccoli okay i can't even jay-z are they better than public's cakes yeah oh yeah totally and they have different kinds yeah man yeah dude not even close okay never i'm gonna buy you a cake what kind of cake you like outlaw i'm gonna buy you like from costco when it when it's when it's birthday time because i don't i try to stay away from them because i'm like they're just deadly but when it's birthday time i'm like okay you know what i can splurge i can have a slice of the the buttercream icing from uh cost from public's see

now you got me saying it yep i almost said it yeah you're gonna you're gonna learn one day but but no no like what kind of cake is it that you want because i'm gonna buy you one from public's and i'm gonna buy you one from costco and we're gonna eat see this is why no i can't listen to this i already can't do this i'd have to to go mountain biking like 85 miles just to burn off a slice of that. And then you'd be like, all right, now here's your Costco cake. That's right. It's going to be big, too. You know that?

They are. Oh, yeah. It's a different thing, too. The whole Costco cake is probably the size of a football field. But I can only pay $15 for it. That's right. And you can feed everybody at Costco with it. They're amazing man like i actually won the guinness book of world records for largest

cake as i was purchasing this it's crazy it's very efficient uh you know a lot not a lot of toil involved in those calories that's right that's right all right well since we're talking about toil i thought i'd ask for this episode, does your job include any toil? And your answers, your choices are your answers. Let me just tell you your answer. Now, your choices are, of course, it includes some, but it's a reasonable amount. Or this topic is opening my eyes to how much toil my job has. Or I think my job includes too much toil, but my team won't do anything to change it.

Or, oh my God, if you remove the toil from my job, I'd have no job left. All right, fine then, Mr. Smarty Pants, who's like, you know, I knew tears off the top of my head. I didn't actually. Joe, this one's Joe only. All right. What's brown and sticky toffee a stick hey toffee too though dude that might be my favorite good because i got dogs you know so it took me a minute

to think of something i could say on the radio the radio what's that dude i swear when he said that i was thinking of uh um adam Sandler with the AM radio. Yeah, just press number three on your eight track. That's right. Oh, good times. This episode is sponsored by Shortcut. Have you ever been really happy with your project management tool? Most are either too simple for a growing engineering team to manage everything or too complex for anyone to want to use them

without constant prodding. Shortcut is different though because it's better. Shortcut is project management built specifically for software teams and they're fast, intuitive, flexible, and powerful. So let's look at some of these highlights. We got team-based workflows. Individual teams can use Shortcut's default workflows or customize them to match the way they work. Organization-wide goals and roadmaps. The work in these workflows is automatically tied into larger company goals. It takes one click to move from a roadmap to a team's work to individual updates and vice versa. Tight version control integrations, whether you use GitHub, GitLab, or Bitbucket, Shortcut ties directly to them

so you can update progress from the command line. They have a keyboard-friendly interface. The rest of Shortcut is just as keyboard-friendly with their power bar, allowing you to do virtually anything without touching your mouse. And iterations planning. You can set weekly priorities and then let Shortcut run the schedule for you with accompanying burndown charts and other reporting. So give it a try at shortcut.com slash coding blocks. Again, that's shortcut.com slash coding blocks shortcut because you shouldn't have to project manage your project management. All right. So here we go back into the serious bits.

What qualifies as engineering? Anybody? Is this a dad joke? It could be probably. Yeah, I like this. We've already kind of hit on this, but it's basically work that requires human judgment. So computers can't automate engineering, which I thought was interesting by this definition. It's work that produces permanent improvements in a service and requires strategy. The design-driven approach. So wait a minute.

If my work that I do didn't have any strategy involved, am I like not engineering? Yeah. That's why I'm doing it wrong. engineering problem yeah that's what i didn't i mean really like if you could find you know something like uh you know unskilled laborer if you could train a dog to do it then it's not engineering right uh it's just going through the motions if you can show someone else how to do it if you provide them instructions then it's not engineering it's mechanics i don't know way to take away from my joke but yeah i mean let's, I was very excited about training dogs to do stuff. Get all serious about it.

Um, I do like this last one there that they mentioned is the more generic or general, the better it is because it can be applied to more than just your service. Right. So it's, it's thinking more holistic about, um,

the services that your group or your company or whatever has that you can add some good benefits to. Yes. So let's talk about some typical SRE activities. Software engineering, basically engineering using software, obviously writing or modifying code that aligns with your strategic vision. System engineering. Configuring systems, modifying configurations, documenting systems that provide long-term improvements. I kind of feel like there's a lot of blending of those two going on these days, but maybe that's just me. I don't know.

I think it's because we live in the Kubernetes world and it does seem to be intermingled. Yeah. I like the Kool-Aid, though. intermingled. Yeah. I like the quality, though. I like it. Yeah. It's the toil. Well, I was just going to say, like, the documenting part, though, is only included in the systems, not included in the software engineering.

Yeah. That was interesting. Weird distinction. Yeah. My code is self-documenting. Oh, the Uncle Bob stuff. Yeah, okay. I see where you're going with that.

Yeah. Totally. Of course, we talked about the percentage of that. Overhead. So this is things like administrative work, things you do in like JIRA or kind of maybe HR, paperwork, meetings, stuff like that. Also, I guess the retrospectives would be considered that. You know, things like that also i guess the retrospectives would be considered that um you know things like that uh 50 percent goal is over a few quarters or a year which is really nice because it gives you a chance to have some long periods of focusing on one type of activity and kind of averaging that

out over time yeah so it means that like you know you could definitely have a month where you spent doing more toil than you would have liked. But over the course of the quarter or the year, it should level out to be the 50% goal. Right? Yep. And here's that question we talked about earlier. Is toil always bad? I talked a little bit about it.

And it's probably because it was the last thing I listened to before. I was sitting in the drive-thru line listening to this on audiobook thinking about this. But the fact that some amounts of toil is predictable and repeatable makes some people feel like they're accomplishing something. It really feels like work. And there's something nice about saying like, I go here,

I shell into this, I get the ticket, I get it approved. There's some comfort and process I think for sometimes some people. Well, it's also like it's kind of it can i i was thinking of this stuff is kind of like that brain dead kind of stuff that you don't have to really like think about like you know like you just you know what you got to do you could do it and as long as you don't like get distracted like muscle memory will just kick in and you'll just you you'll just do it. You know? So you felt like you accomplished something now, you know,

but it's toil, but. Yeah. We ever thought or said like, I hope I just get some stupid easy tickets to the sprint. You know, like you want to get some wins. You get tired of punting on,

you get tired of being stuck and be tired of being confused and having meetings about stuff you're supposed to be working on. You just want to be able to like close your eyes and get some work done but yeah that's when i was on the whole big deliberate practice uh kick that was something i kept running into too is um you'll see a lot of programmers back then the book flow was like popular at the time this is a couple years ago and so people were kind of talking about how to get into flow state with programming and uh the deliberate practice stuff that i read at the time was like that's not really great because uh flow is great for manual tasks it's great when you're exercising or maybe playing a sport or you know doing um drills but it's not great for strategic decision making right if

you're in the flow state while you're making you know those decisions and it's kind of implying that you're on kind of autopilot and you aren't really thinking about what you're doing, where you want to go. It's kind of like when you're doing engineering, you want to be awake, if that makes sense. You want to be considering everything. You don't want it to be routine. That makes sense. That actually makes a lot of sense. That's not fun because the flow feels good, right?

Yeah. There's days when you clock in at 9, it's like 5 five or six and you're like, hey, I feel great. Let's go. Let's go mountain biking. Yeah. All right. So, but some of the amount of toil is expected and unavoidable, right? So there's going to be some things that I was trying to think of like an example though, of like, uh,

in our day to day is like, what's some examples of like toil that you can't get around. Right. I think alerts like, like, um, if, if there's an alert that,

you know, there's some, one of your systems is running too high on CPU or, or there was a, that wouldn't be an example, right? Because you could automate that to where you just scale out to a new pod. You could, what if there's a 500 error or something

that pops up, right? Like the toil might be investigating that thing until you get a chance to automate whatever or fix whatever that problem was. I mean, there's, there's all kinds of interrupts like that. So let's say new errors that new errors that pop up when you get an alert for a new error that you hadn't seen before. That would be an example of unavoidable toll. I can't say the word. Because it's going to require digging into some logs and going through some stuff. You haven't made any state changes to anything at that point, right? And maybe you have to go do something. you haven't made any state changes to anything at that point. Right. So, um,

and maybe you have to go do something like me. I don't know. Maybe, maybe something was triggering that 500. You have to go manually clean it up. Right. Like that's toil. So that's, but,

but seeing that error for the first time and dealing with it though, that's an, that's an example of where toil isn't bad. Right. You see that error a second time. Cause you didn't bother to fix it the first time after you went and investigated it. Then maybe it is bad because if you spend time looking into it a second time.

I'll give you an example. If there's manual steps that you have to run in your deployment process, right? That's toil. If you can't just say, hey, go deploy this thing and, and be off it and just come back when it's done, then that's toil, right? Like if you have to go in there and manually adjust some things or copy and paste some values or, or,

or do any number of steps, maybe you can even argue the fact that you had to like go in there and start the process? Possibly. Well, I mean, depending on what thing is, but like, let's say you're deploying to a new environment. Maybe, you know, that requires some judgment, right? Like I'm not just trying to update an existing dev site or a QA site or something like that. But yeah, like that kind of stuff that you have to do every single time

you go in to do a task, that's toil, right? Because chances are it's the same steps you've got to take over and over and over. Yeah, they do use the word unavoidable, which means that it's not automatable for whatever reason. So maybe that would be getting sign-off from QA or something. But then you're kind of bleeding into administrative-type tasks. If you need to go get sign-off on people in order to get released to a staging environment, then you've got to go get sign off on people in order to get released, released to a staging environment, then you got to go get signed off and then get released in production. And some environments you just have to do that.

You know, like of course we all want that multiple releases a day, but there are some industries and governments and stuff that you just can't be releasing like that and require just more regulation. But the whole like require sign off thing though, sounds like administrative overhead. Like that sounds like toil because that could be in my mind. I'm thing, though, sounds like administrative overhead. That sounds like toil because in my mind, I'm like,

well, why couldn't you just put in some tests then to check to verify that whatever they're going to do to sign off, quote, sign off on it, why couldn't that be a test or something that was automated to say, yep, it passed. It's good enough.

If it passes all these, go back to the DevOps handbook. If it passes all these things, then fine, deploy it automatically. Well, you could take a step back, right? Like maybe, I don't know, maybe there's something that somebody has to check, but even then it shouldn't be anything where, where you as a developer or an SRE has to go in and do anything. You know, once it enters the state, all tests are passed. QA goes and just clicks a button because they're notified of it, right? Like you shouldn't have to

go tap QA on the shoulder and be like, Hey, um, that finished, you know, can you go take a look at it now? That's toil, right? I don't know. It's, I'm sure there are situations that I can't think of off the top of my head of why you can't just auto-release something just because it passes all the tests. I can think of times when it's not unavoidable, but it's not worth it. Like if you're doing a major upgrade of a database or something, that might be something you could script, but it's something that you do once every couple of years. It's better to just do it on a weekend and have a team around and just kind of budget some extra time for that. I don't see that as toil though. I mean, because every, every upgrade to a system introduces things that you can't anticipate, can't expect. So that, that to me is more of

that, that has long-term state benefits. Like that's one of the things they mentioned. That's not toil, right? Like you upgraded the state of your database, which means you probably got security improvements, all that kind of stuff. So the end state is better than what you started in. So to me, that's not toil. The toil would be if every time that you go to run,

um, your deploy, you have to go touch the database for some reason, then that should be automated, right? Like the, I don't know. Well,

here's, here's, this is going to be controversial, uh, should be automated, right? Like, I don't know. Well, here's, this is going to be controversial. Prepare. So, but I mean, Jay-Z just said something that to me was kind of like a trigger because here goes, I'm going to get some hate for this. But like, he was like, well, you know, qa has to like pass judgment on or anything and you know immediately in my mind i was like well man like if qa is like having them go and

manually do things like that is definitely toil like that's you know in the in that the way he described that there was like a whole group that's nothing but toil like in my in my mind i was thinking like well the utopia would be that the QA team is writing tests that, you know, that they want to see happen, you know, independent of what you've maybe developed, right? Like, Hey, this is the way it's supposed to work. And I'm going to write code to verify that it does that kind of thing. Almost kind of like a test driven development, but you know,

you're kind of separating out the developer from the testing or the tester part of that but uh you know and then and then that way there's no like oh well let's get qa to sign off on it because they've written test for it so you either saw the test pass or you didn't i think that's perfect like i think that's the goal but like if you think about like some industries and stuff like uh i was just thinking like banking or like taxes you know like uh intuit like when they publish h&r block or whatever turbo tax every year like you know it's kind of a big deal they're not they're not releasing that 10 times a day or 100 times a day you know it's like big releases that have to be communicated they have to work with government regulators on changes you know they have dates that they commit to in order to kind of keep up with contracts and stuff in order to be able to like submit stuff to,

you know, these federal institutions that don't move that fast. And these things take a long time. And so there are these kind of slower gates. And so, you know, I think some industries have some slower processes and, and sometimes,

you know, like more traditional kind of QA type roles might work out better for those kind of cases where it is more kind of waterfall and process and uh you know so in a perfect world like qa i think would be automating 100 of the time like that would be their jobs all day they're automating and coming up with test cases and so the strategy part is coming up with the you know things that they're going to automate and go after and you know how they measure and all that stuff but um we should probably do some stuff on testing but uh anyway uh yeah we did do an episode on unit testing by the way in a minute well i just mean i mean more like just around the whole like uh you know integration tests the strategies around testing like coming up with test plans

managing environments like this probably really good books on it, but more than just the technical side of it. Right. QA engineering. Is that a thing? Probably. Alright, so back to this. The next thing that they bring up here is when the amount of time on toil becomes too large, like you're spending too much time on it, they said that you should actually, as a person, as an SRE, you should be concerned and complain loudly, which I found really interesting, but it makes sense. So there

are potential issues when there are large amounts of toil for you individually and for the company as well. So one is career stagnation. And I thought this was really interesting. Like they don't want you to be stagnated in your position at Google. If you're spending if you're not spending enough time on projects and you're spending too much time on toil, your your career progression will actually suffer because you're not strengthening your engineering capabilities. You're not adding that value to the company, right? Or even to your own role. But then also, and you've probably seen this for teams that deal with constant toil type tasks is low morale. You know, you get, you get burnout. It typically becomes boring after some amount of time, right? Like when you're doing the same thing every day, like, Oh, the thing

crashed again. Let me go restart. I S let me go do this. Like it gets old and then you become discontent. Right? So it's not good for you personally. And it's not good for the team. And it's not good for the company. You know, good for the team and it's not good for the company. You know, like I said, I took a blue example here is like the old it help desk. Like, did you turn it on and off again?

Like you imagine like, you know, like the it crowd, like type shows where you did the same types of crest every day is like, can you fix my printer? I'm locked out of my account. You know, that sort of thing.

Um, firewall changes. Can't imagine. So I was, I was curious. I went back and looked to find when we talked about testing, unit testing.

It's been a minute. Episode 20. Okay, so I know the book, The Art of Unit Testing. I don't even know if that was the book that we talked about at the time. I don't think know if that was the book that we talked about at the time. I don't think we talked about the book.

Yeah, well, kind of. So, episode 20, coincidentally, it was just coincidence that Jay-Z was right. That's all it was. But that was the first one. And The Art of Unit Testing was mentioned in there, as well as some others. Code Complete was another book that was referenced during that episode. We talked about it again, though, in episode 54, as it relates to the Clean Code book. So 54 and 20.

Wow. So I'm kind of, because episode 54, I mean, that's the newest of that pair. That was January of 2017. So I don't know. Maybe we've got a difference of opinion from then till now. Clearly, testing's dead. I mean, if we haven't talked about it in five years, it's just not even a thing anymore. And episode

20 was from 2014, December of 2014. Wow. It's been a few episodes since we talked about it, but only by a few. Wow. That's pretty awesome. Alright, so

too much toil also hurts the SRE teamre team right because they're not able to focus on the projects this one this next one was really interesting to me i think their wording is really what's interesting because they say it creates confusion um if the sre team is supposed to be an engineering team and they're not getting to do engineering, then that means that the goal of the team is not matching what they're actually doing. So confusing seems to be the wrong term to me. It's more like they're probably annoyed. They're not confused. They're irritated.

It's bad to have a directive that's kind of ignored, too, because there's going to be some people up on the team. They're like, hey, this isn't right. We need to change. There's some people that probably don't want to. And yeah, it shouldn't be discussion, right? You like deciding the rules ahead of time. Slows progress as well. So the team will be less productive if they're focused on toil. So it's a bad precedent.

If you take on too much regularly, then it tends to kind of accrue and you find yourself doing more similar type tasks and it kind of, yeah, it sets a precedent, right? Yeah. Cause of their tone. They'll be like, Oh, that's the toil team. Let's just give it to them to go with. Totally. Yep. Like they're already doing this and that it's not too far off, uh, promotes, uh, attrition. So people will leave and, uh, yeah. Yeah. The interesting thing here is they say you lose the talented engineers, right? Because they'll get fed up. Hey, if I'm

not going to actually get to write any software or update these things, then why am I here? I'm not trying to be tech support all day every day. Yep. I like this last one too. It causes a breach of faith. So if someone joins the team and they don't get to do any of that engineering, they're going to feel like they were sold a bill of goods. You hear a lot of a lot about that where, you know,

companies will have the interesting interview where you're kind of doing the whiteboard stuff and then you get an actual job. Same thing goes for, you know, SRE stuff. You join, take a job at a company expecting to do human tasks and they got you, I don't know, moving backups or something.

Well, the joke that we, you know, our, our joke before or historically on the show has been like you have this like really complicated uh interview like how many golf balls can fit inside of a 747 uh you know or something like whatever you know like all these weird like abstract math problems that you get asked for you know like here's an array find the missing number without sorting it or whatever, you know? And, and yeah, you're like, okay, well they're like, they asked me some challenging things. They must want me to do some challenging work. Like, that's great. And then you start day

one. They're like, okay, we need you to move the logo three pixels to the left. Right? Like, so yeah, I mean, it's not exactly toil, but you know, probably. I mean, in fairness, if you are somebody that's doing hiring and you're speaking to people, like, seriously, be honest with them and let them know the type of work that they're going to be expected to do. Because the worst thing that you could do is give them this pie in the sky type thing. Then they get in and they're like, yeah, I'm totally not doing that kind of stuff. Right. Like, yeah, that's cool to see. Well, I was just gonna say the technical interview has,

has evolved into this thing where this total rant, where it is not necessarily representative of the job, but it's in the hopes that like, well, what if everybody we hired was just amazing at whatever, blah, blah, blah, blah. And we don't have to do that yet. But until then, we'll just have them doing like, you know, this type of work, like move the logo three pixels to the left. But, you know, something great is going to come along. And, you know, if we go ahead and stack the team with it, then, you know, we'll be there. So like, you know, you have this like really incredible, incredibly difficult interview.

Yeah, it's tough. Because you also want interviews to be fair, right? So it's like one thing that's nice about it is you can be a little bit more objective when you ask the same type of people or the same people the same types of questions and kind of try to deal with them in an unbiased way. So it's like I understand why big companies want that sort of thing. They want repeatable processes even if they're flawed. Yeah, but on the flip side though, it's also difficult too because it's like, well, what if the interviews were honest?

Imagine honest interviews, right? Like, so tell me, how much time do you spend getting coffee? Just, you know. And your typical lunch is two hours? Yeah. How many meetings can you handle in a day before you just freak out and stop caring about everything? Wait, do they have to be consecutive or can they be broken up? Because my ability to care definitely goes down as the contiguous number of meetings go up.

Stop me when you think the number is too many meetings per day. 12, 13. And you're like, wait, why did you start at 12? Yeah, wait, hold on. up stop me stop me when you when you think the number is too many meetings per day 12 13 and you're like wait why did you start at 12 yeah wait hold on we could just interview in the interview right here we could tell that this is gonna go very well the lower bound wait wait wait i thought that was the upper bound no no no you didn't read it right wait a minute oh wait wait all right so the last two things we have here oh the sre like slogan that they had sort of at the end of this was let's invent more and toil less which i think that's awesome that's that's really cool and then commit to cleaning up a bit more toil each week with engineering activities,

right? So solve it in an automated, um, you know, machine driven way. And I promise we will never say the word toil again. We are done with it is dead to us. We don't even know how to spell it because it hurts.

Did you see how I tweeted? No, I control F our notes for that word, uh, 40 times in the notes. And we said it a whole lot more. Oh, that's awesome.

You know, Joe, well, uh, yeah, you said the last part about committing. Um, what type of music do balloons hate pop pop yes pop music wow joe's doing good today like he's he's on a roll man yeah are you gonna

play the lottery after this like you probably should never uh okay so we'll have some links to resources we like as part of this episode. And with that, we head into Alan's favorite portion of the show. It's the tip of the week. Yeah. So, so I am probably extremely late to the game.

I'll admit here on Python has been around for a minute or two, I think. Um, and there's this thing, it's not a trash panda, but it's called pandas. And I kind of fell in love with it this past week. So here's the situation. I'll tell you what it is. And I'm going to, I don't have my code in here right now. It's literally four lines of code to do what I needed to do here. But I'll get that into the show notes so that you

can see it. But here was the gist of what I had. I had a big GCS bucket. So a Google storage bucket, similar to AWS S3 or whatever, if you're familiar with that or Azure blob storage, whatever. This thing had millions of objects in one directory. And if you know anything about blob storage on most of these platforms, they don't give you the ability to sort by anything except the name of the file. And they do that on purpose because it's just a ton of blobs. Well, I needed to find out what my biggest blobs were and what my smallest ones were so that I could have an idea of what was going on. And so I started writing a program and I was like, man, this stinks because the only way to get it by the way is to list the

entire bucket, right? Like you have to list it all out and get the stuff. So in, in my world, that's like a GS util, um, LS dash L to get more detail and to list that bucket out. So what I ended up doing instead of writing a program, because I started doing, I was like, man, there's gotta be a better way. So I essentially did the gsutil ls-l and then my bucket name. And then I did a redirect to a file, right? So that I could, um, so a greater than, and then, you know, my list of files.txt. So after gsutil finished running, which was probably 20 minutes on millions of items in that bucket, in that particular folder, then what I was able to do is load up Python, which

by the way, this should be my tip of the week as well. I might've used it before. I Python, I think I might've mentioned this before. Um, loaded up. I Python, I imported pandas after I did a pip install pandas. And it was literally, literally two lines of code, one to import the file from its TXT format into a data frame. And then one more command that basically did a group by of the, um, the units, which I had done an LS style dash LH to give me human readable. So it would tell me whether it was in megabytes or gigabytes or

whatever. And actually it was medibytes and gibibytes. And if you don't know what that is, look it up. Um, wow. It'll be a fun exercise. You can't give away everything, right? How can the tip of the week be like, go figure it out? That's right. That's right. RTFM. So at any rate, what was so amazing is I could basically do a group by of this text file and say, hey, group by the units and show me the max man average of the sizes in there. So I could easily look over millions of files. What my, what my biggest file was,

what my smallest was and what the average size was. And I honestly crap though. I mean, we shouldn't averages are crap. They're completely useless. I don't even know why they're a part of math. Um seriously, I think this entire operation, loading the millions of records and doing the stats and all that, was like 15 seconds or less. Four lines of code using pandas. So, I mean, look, I've done a lot of data imports over the years using SQL Server and other tools, and I swear to you,

I've never done anything that works so painlessly as loading this data into the Python Pandas app and doing this stuff. Like, it was mind-blowing. And I believe behind the scenes, Mike, I think you know this better than I do. Doesn't Jupyter Notebooks, they use Pandas behind the scenes maybe? Well, it's not that Jupiter notebooks is necessarily using it. It's just a library that you can use inside of Jupiter. So I've got plenty of notebooks where I've written using like pandas and

numpy, for example, or like, you know, like two of the like that and a matplotlib would be like, you know, those three are like the common go-tos for any kind of like a data analysis that, you know, I mean, your mileage may vary, but you know, that I, that I've done like those three are common. Well, I was in love. So it's so super powerful. I mean, it's, it's the ease of getting to use the power that is, that is the part that's so beautiful. Right. So, um, yeah, man, go check that out. Um, like I said, there'll be code and a link to

pandas up here and, and I ran mine in Docker, right? Like I didn't want to put all that mess on my system. I mean, we've talked about these virtual environment things in Python before and how they're trash. Save the hate mail. I'm kidding, but I ran it in a Docker container, super easy. And it was, it was it was amazing so yeah that's all all right well i have one i think is pretty cool uh you remember that book many years ago was popular called seven languages in seven weeks yes yeah so it's cool like each chapter was a different language and it had to kind of do different problems and there's been a couple things that've kind of spun off since that there was a seven databases seven weeks book there's

you know people write blog posts and stuff and kind of arrange it that way. And this is one of those. This person put together a blog post and detailed seven user interfaces, some graphical user interfaces that you could build in order to, the goal is basically kind of give you a nice representation of what kinds of things you do in UIs. And if you think about that, just kind of step back and think like what seven uis you would build or ask people to build in order to kind of do that uh it's it's kind of a tough challenge especially to come you know come up with seven and so if you look at the first one is just building

a counter it gives you the the understanding of basic ideas of kind of like language and in tokyo it's basically the most simple thing you do you push a button and number is up the next one temperature converter goes into bi-directional data flows like two two-way binding um you know where you can change the fahrenheit on one side of the celsius on the other like okay you can see how that's kind of like a different yet common challenge uh flight booker is the third one constraints there so maybe you drop change change something in a dropdown and it affects other things on the form, right? That's kind of the dual here. And so each one you're kind of leveling up. A timer, you get into concurrency, CRUD apps, you know, just different things around kind of creating and showing and things like that.

It gets into circle drawer, which is pretty cool. Ideas there being things like undo and redo so if you're interested at all in ui design and not i don't really mean pretty graphics but just actual like kind of user interface the very you know most basic level this is a really cool one to check out i have to mention the last one of course which is basically uh excel you're going to be dealing some stuff with cells where you make a change in one cell maybe it changes the sum and so it has these kind of cascading effects which is a cool challenge. So I like the idea of like, you know, if you are interested in human usable user

interfaces, then this is like kind of a cool thing to work through. I'll have that link in the show notes. So, Alan, you're already shaking your head. Why are you shaking your head? Dude, that sales challenge. Like,

no, yeah, no, I haven't looked at anything else on the show notes, but no, that, that sales channel challenge, that one where he's talking about building like an Excel sheet.

I can't even imagine the pain that that's gotta be. So yeah, yeah. You gotta work to that one. The reasons last. Yeah. Yeah. Imagine when you like have to like write other worksheets to reference previous worksheets i mean now writing the first one is like bad enough oh what would you read oh yeah now i read the line on 173 we

got here oh what does that say alan it says trolling to alan i'm curious oh oh oh well so episode 21 yeah yeah well at the start of this episode uh we were talking about or even before we we started recording we were talking about like how my world here lately has been crazy writing like, you know, just in a, you know, Linux type of world, but just, you know, writing one command, piping it to the next, piping it to the next, piping it to the next, you know, or just writing like, you know, what should probably be a bash script, but, you know, in one line and like looping through a bunch of data or whatever, doing whatever you have to do. So when you were describing this problem of the GS util LS minus LH, blah, blah,

blah, blah, blah. And like how you're like, okay, well let me go write some Python to deal with that. I'm like, really?

Like, why didn't you just pipe it to sort? But then you did start getting into some like other legitimately, like you had some other use cases there where you're doing like using like the stats from pandas to get like this, you know, min max average standard deviations, whatever of your sizes and everything. So that was my first attempt, just so you know. Oh, not was to use the sort command.

You use the sort. Yeah, I was going to try it, but then I was like, man, this is so big and it's going to take so long. Um, that, and it wasn't going to give me all, all the information I needed. So, yeah,

but that was my first attempt. Yeah. But that was immediately what came to mind was like, why don't you just use a sort command? But then you sort, sort dash K two. Yeah. That's,

that was what I was. Yeah. K being the column that you wanted to sort by and then dash n for numeric. And then you just like pass in your file and whatever. Or you can just pipe it. You could have just done the gsutil and then piped that into, or like, you know, into, yeah, pipe that into sort

and then blah, blah, blah, blah, blah, whatever. So it was just, you know, kind of a funny to do. But that wasn't my real tip of the week though. So the real tip of the week that I have one was that there was pretty cool. That was shared with us earlier today by a coworker was that if you use mini cube, like we love mini cube and you should love mini cube. If you're doing any kind of Cuban Kubernetes work, the, here's the quick elevator

pitch on Minikube. If you're not already using Minikube, Minikube makes it super simple, like stupid simple to use a Kubernetes-like environment locally that you can specify the version of Kubernetes that you want to use so that you can match your production environment or your test environment, whatever environment you need to do. They make it super simple, easy to specify that version. Unlike, sadly, Docker Desktop, for example, the version of Kubernetes is tied to the version of Docker Desktop. So if you've already installed the latest and greatest Docker Desktop, well, guess what version of Kubernetes is tied to the version of Docker desktop. So, you know, if you've already

installed the latest and greatest Docker desktop, well, guess what version of Kubernetes you're getting? That one. So there's the sales pitch on Minikube. And the problem with that, though, is that if you use Minikube's Docker environment as your Docker environment, which is the way I work, there's, you know, you can do an eval statement. Like there's a Minikube command. minikubes docker environment as your docker environment which is what the way i work there's you know you can do an eval statement like there's a minikube command i think it's like minikube minus p and then you specify the profile name and then you would say something like docker dash env and if you were to do an eval of that you could actually like set your host docker commands to use many cubes,

Docker Damon to text cute all your stuff. Right. And there's like, you know, if you want to use, I think pod man, there's support for pod man or rancher as well. But you know,

in this case I mentioned Docker. The thing though, is that now what's technically happening there is many cubes running in a VM as your Kubernetes cluster and any of your Docker builds, technically those images live inside of that VM. And the issue that Alan ran into was he wanted to be able to Docker run one of those images, but pass in a volume from his host operating system into that thing and minikube will like happily act as like a facade you know for that

and and just give you a false positive of like yeah i'm going to mount that sure it'd be fine but really it's not it's not passing through your, your host operating systems. Volume is not passing through to the guest VM that is mini cube that is then passing it on to another guest. That is the Docker image inside of mini cube that you're going to run as a container. And what you can do, if you wanted to bring that out externally to the host so that you can run it is you can use the Minikube image save command to bring that image out from the Minikube VM to your host operating system. And now your host operating system has the image. You could Docker run and mount volumes and everything will work just fine. Hey, real quick. I do want to say what you said very simply on the Minikube thing. Don't think that you can use, just say, make Minikube your Docker demon and everything will

be good. Because what Outlaw was saying is if you Docker run an image that Minikube has and you try to mount a volume, it will act like it does it. But if you go in and you look at that volume inside that running container, it'll be empty because it does not actually mount the volume. It looks like it does and it acts like it does. But when you get in there, there's nothing in it. So it has been the cause of probably hours lost between multiple people dealing with that kind of stuff. I've even forgotten about it and then gone back looking like, why is this not working? Yeah, it's frustrating because it's truly the Docker daemon that will run inside Minikube looks like it's perfect until you realize that it's just not working the way that it should. Now, that said, I do take issue with what you said about just use the Minikube Docker daemon. It's fine.

It's great. No, no. Don't do it. No, it's fine. Because you didn't need the Docker daemon to do the run, right? Or no, I guess you did. No, you did yeah whatever if you're gonna yeah no it's not good it's no boy you could uh i'm like those few times that you need it whatever you could do what joe skip it all of it i've been messing around with uh at home with uh rancher desktop yeah uh which is pretty nice so you can choose to do either docker d for the daemon or container d uh you

still need something to build, so like Podman or Docker is going to be fine for the actual building. But it's pretty cool. So it runs on K3S, which you can kind of think of as being somewhat of a competitor to MicroKits, or sorry, Minikube. But the way they run Kubernetes is kind of different. They strip out like older APIs and a few other things.

They just don't support it. They use the SQLite instead of etcd. But because of that, it's, you know, it's not high availability. It's definitely not as resilient. But what I like about it over Minikube is you can run it on multiple nodes. Not something you want to do for production, probably, especially running on SQLite. But it's nice for local. And it's just a lower memory footprint overall

probably because of the stuff that they just decided flat out they're not going to support. So I don't even know if K3S is technically Kubernetes because it supports a subset. The subset that it does support is like the most common stuff that you're probably using. It's a certified distribution, Kubernetes distribution. Okay, cool.

I thought it was supposed to run on arm. Wasn't K three is supposed to be an arm specific arm. Yeah. Arm 64 arm V seven. Uh, yeah. I mean, it's like for I edge IOT devices are,

and it looks and feels like a docking desktop. So if you're used to docking desktop, it's basically a drop in. It's almost like tab per tab. But when you when you uh enable kubernetes you can pick your version i was trying to remember like the reason for the name for it though like there was something clever about the reason why it was called k3 wasn't it because they dropped like five somethings of it yeah it's koonities or something. Who knows? So, uh,

all right. So one more tip of the week though. Um, so I've been doing a lot of like, uh, trying to optimize things like related to Docker and whatnot and Docker caching and things like that here in, uh,

you know, my past days. And, uh, one option that if you haven't already used, so if you, uh, you know, my past days. And, uh, one option that if you haven't already used, so if okay, let me just back up. So with Docker, your typical Docker command or Docker file, you're going to have like from, and some base image, and then you're going to do some other things. You might copy some stuff in, you might run some things, whatever. Right. And that's like your, you know, that's, it's almost kind of like a vanilla Docker file, but I kind of want to think of it as more

like the happy path because it's so much easier if that's all you have to do. But there's also other things that you can do, like they're multi-stage Docker files where you can have like from some image as, and you can name it as, you know, base and then another from later on where you say from some other image and you could call that like as dev or, you know, whatever, like you can give these things, whatever names are. So you could have like multiples of these things that are all stacking on top of one another that, you know, you can build in one of those stages and copy from that stage into a later stage or whatever. The problem with it though, is that from a caching perspective, it's awful. It's awful from a caching perspective because

the, the straight up Docker builds just don't like them from a caching perspective as well. Because what's going to happen is, let's say, okay, first of all, let me clear some things because somebody's probably like, wait a minute. So if you built, let's say you had a Docker file that had five stages in it and you built it locally, then yeah, you have all five of those stages cached and a subsequent rebuild of it's going to be fine, right? But let's pretend you don't live in that world where you can like maintain the cache locally on your computer. And instead you're like pulling from a repository or some kind of shared cache, right? The only thing in that case that is typically going to be cached is the final stage. So when you do your Docker build of that five-stage Docker file, the first four, it's going to be like, well, I guess we've got to rebuild them, only to then get to the end and say, oh, I already have this cached.

I'm fine. Right? Well, you can use the Docker build dash dash target option. And what that'll do is whatever your names are inside of those, uh, that Docker file. Like, so I said, like as base and as dev, uh, for example, you can set, you can specify that as the target you want to build up to. And when you do that, what you can do is then Docker push those stages. You know, you could tag those stages like any other Docker image. You can, you can tag those, push those up to whatever your, your repository is that you're using for the cache. And then in subsequent builds later on, you could pull those individual

stages down. And then in your next Docker build, like, you know, take advantage of the cache. So, cool. Excellent. Or not. I looked up K3S, the joke,

with it. So Kubernetes is a 10-letter word frequently abbreviated as K-A-S. up K3S, the joke with it. So Kubernetes is a 10-letter word, frequently abbreviated as K-A-S. So K3S, you add up the total number of letters, it's 10. Or sorry, it's five, half of 10. So the idea is, and that ties in with our original goal of being basically half the memory footprint as other Kubernetes distributions. Wait, how is it five? K3S. Yeah, well, the three it five? K3S.

Yeah, well, the three plus the K and the S gets you to K3S, you know, five. But Kubernetes, K-A-S is a ten-letter word with K, eight letters in between, and then an S. If I were to count on my fingers, Joe, math in my chicken, hold on, watch me.

K is one, three is two, S is three three how is that not the three it should be k plus three plus s yeah because kubernetes the eight represents the eight letters that aren't shown there okay okay and what are the three letters that aren't being shown in there aren't any it's literally the joke joke because they wanted half the size of Kubernetes. See, what you should have said is it's UBE. It's UBE?

Because then it'd be cubes. Oh, nice. Well, I like it too. It's funny because three is also, you know, if you were to cut the number eight kind of in half, it kind of looks like a three. So it's just kind of clever all around. But no, there's no word behind it. See, when you were telling us a minute ago, I thought you were trying to say that, like, well, if you divide eight in half, it's three. And I'm like, oh, it strikes again.

It's so good. Cut it in half. Yeah, if you slice it in half, the actual character. Yeah. Okay. Good stuff. How about if I end with a uh a bit of advice because you know how you've heard that an apple a day keeps the doctor away but an apple a day also keeps the

bullies away if you throw it hard enough so with that subscribe to us on itunes spotify stitcher wherever you like to find your podcasts. I hope we're there. If not, hey, let us know, and we can figure that out, sort that out. I'll open up a Jira ticket for Jay-Z, and he can submit a pull request to Alan, and we'll get it all sorted out.

That's right. Like I said earlier, definitely use my recommendation, not Jay-Z's. But if you haven't already left us a review, we would greatly appreciate it. You can find some links at www.codingblocks.net slash review.

Hey, and don't toil about it so much. Head up to codingblocks.net and check out our show notes, examples, and discussions that we're... We said we wouldn't say it again! Oh, did we say that? I missed it. Yeah, we have to beep it out. Yeah, you can also send your questions, feedback, and rants to jayzee

at the Slack channel. Any negative reviews can go directly there. Yep, yep. So I'd love to hear that negativity. Just send it to me. And we also have a Twitter account at CodingBlocks if you want to, I don't know, send us a cat picture or whatever we'd love to see that

sounds good right now and cuttingblocks.net has all our social links at the top of the page