Coding Blocks - Site Reliability Engineering – Evolution of Automation

Episode Date: June 20, 2022

We explore the evolution of automation as we continue studying Google's Site Reliability Engineering, while Michael, ah, forget it, Joe almost said it correctly, and Allen fell for it.

you're listening to oh wait no i gotta start over with like a bunch of like pizzazz and like wow and you know hey you're listening to coding blocks 187 because you know i can't can't do it the same you know i gotta like trip it up you know spice it up right i think you know thank you thank you for recognizing so uh yep this is coding blocks that's our new intro. Yep, this is Coding Blocks. Here we are. No, this is 187. Subscribe on iTunes, Spotify, and forget it.

Wherever you can. Hey, and while you're forgetting it, go ahead and visit us at codingblocks.net. Forget it. Well. If you haven't unsubscribed and left yet uh you can find us on web5 at codingblocks.net uh follow us on twitter at codingblocks and uh yeah we got a bunch of social uh dillies at the top of the page i'm joe zach i'm michael outlaw And I'm Alan Underwood. All right. So what are we talking about here?

So we are talking about in the book, site reliability engineering. Oh, I said it almost right. Almost. The book we've been talking about for a while. We're talking about chapter seven tonight, the evolution of automation at Google. And we skipped a chapter earlier in the book that was kind of focused on Google,

but we decided to go ahead with this one because we thought it had a lot of just kind of good general stuff about automation and some interesting case studies, which, you know, I'm typically lukewarm on, but I liked here. So we'll see how far we get tonight, but I'm pumped.

I like it. Yep. So from iTunes, we'd like to say thank you to everybody who left it. Yep. So from iTunes, we'd like to say thank you to everybody who left us a review. So from iTunes, we have a sobering. Pushbendy? Pushbend, I would think. Oh, that's probably better.

I like the way you said it better. I'm going to pretend I said it that way. Membrane, Angry Little Hamster, and John Smith's 1982. Like M&M Brain. I do too. Oh, is it supposed to be? Yeah, okay. M&M Brain.

I was thinking like Membrane, but like where there'd be like a silent M in the beginning of the word or something, you know, the way he, with the way he spelled it. Yeah. That's what, that's what made me think of it. But yeah, M and M brain that works too. Um, so say that name again, Alan. Rupesh. Rupesh. Bindi. Okay. So Rupesh asked, uh, along with his, his comment, like, how do you find the time along with your day jobs to, and hobbies to, to keep doing this and everything? I mean, we do this to study. I, at least that's the way I kind of view part of it. You know, like, I mean, this, this definitely over the years has been, uh, a great vehicle to vehicle to force me to continue learning on topics that I might have not bothered to dive into. I will say, so for me, I still get like a fair bit of just kind of couch coding time.

And my trick there is to just code on things that I'm excited about and that I'm having fun with. And as soon as it stops being fun, I quit. That's not a great way to get projects done but it is a nice way to just enjoy life and also learn some things along at the same time. So I am fond of doing fun projects and doing it when I feel like it.

But what does that have to do with CodingBox? Because he was asking how we find time to do CodingBox. Is CodingBox fun? For some reason I read this is programming along with your day job with hobbies yeah no so yeah i'm gonna just make up another question and answer that one randomly yeah well along those lines i just let these guys study it and then i just show up for the show right like that's that's

and i just like to call out Joe on the things he says. Yeah, no, I mean, I'd say the safety outlaw said though, I mean, for real, this is, this is kind of the way that, that we study, but you know, ironically, a lot of times, you know, we'll pick subjects that we feel will positively impact the things that we're doing in our day to days too. Right. So, so it's a good way for us to learn and share at the same time. And we also get feedback, right? Like we've had amazing people like, um, Merle who will hit us up and be like, Hey, you know,

here's some additional context to what you're talking about. So, you know, we study for this and then we share, and then we also get killer feedback from our Slack group, from people just emailing us or, or adding us somewhere like it. So, yeah, I mean, and that's the other thing.

And I know that we say it when we're, when we're begging for reviews and all that stuff, but it truly is like, I know for me, anytime I get something that's like, Hey man, you guys don't even realize, but you've, you've changed my life. Right, right? Or you've made things so much better or I'm happier with what I'm doing now.

Like that kind of stuff is so motivational that it makes it to where it's exciting. Because there are times like, I mean, we could all attest to it. There are times like, man, we got to crank out another episode. And man, that means we got to study and we got to get show notes done. And there are just times when you're busy with work and everything. You're like, I don't want to do this, man. Like I really don't want to do this.

But when you get- I don't feel like that ever, Joe, do you? Like it's just Alan, right? You're just going to leave me hanging? That's good. So I will say, you know, like what I've gotten this question before, my first answer is typically having good partners. Because if I was doing this by myself, just like you said, with the couch programming, I quit as soon as it got tough or, you know, the first time I got tired, I just would have backed away.

But when you've got other people that you're kind of involved with, it makes it easier to kind of, you know, show up and do a poor job, which is what I usually do. You know, the other thing that does help too, and it's funny coming from me, because if you ever saw my inbox, I know we've talked about inbox zero on here, which I think is insanity. Um, like my inbox is a search box. It's got, you know, hundreds of thousands of emails in it. Having a schedule like we do, like we typically have a night that we try and record on knowing that that's coming up definitely helps, right?

Like it's like knowing that you have a test coming in and you've got to prep for it. Remember in the first year though, when we didn't, Oh man, Hey, can we get together tonight? Nah,

man, I got things to do tonight. What about tomorrow night? Nah, I got things to do tomorrow night too. One month. We might only get an episode out. Oh yeah. Oh yeah yeah there were times when it'd be like six weeks or whatever yeah next month we're like here's three episodes maybe we should slide back to that that was kind

of good i wasn't going no i wasn't like you know going down memory lane trying to encourage bad habits again. We have moved on from that for a good reason. I like bad habits. Yeah, me too. But I think there have definitely been some – I think that the community and you guys, there have definitely been some books that, for example, or topics that I might not have found or gone after on my own. But because either one of you or gone after on my own, but because you know,

either one of you or somebody in the community said like, Hey, what about this? And I'm like, Oh yeah. Or even topics that we, we do already, you know,

like user, like we think we know. And then somebody would be like, Hey, have you ever thought about doing an episode on this? And then like, we'll do a deep dive on that thing. And it'd be like, Oh, okay, that's why that was like that.

I mean, this has been a way to, I view it as a way to stay engaged in my career and to continue learning and just kind of advancing myself. Does that make sense? To keep myself relevant in this ever-changing world of technology. Why did I not just become a plumber where everything stays the same? Right. And you make a good amount of money doing it too.

Yeah, right? There's so many other jobs out there that you make seriously good money and the things don't change you know but uh i don't think you've talked me into it so we're gonna do an episode every six weeks and i'm gonna be a plumber we're good that's it real estate you know i mean like that it's it's the same you're just selling stuff like it's but yeah i decided to change i decided to go for this one so yeah i mean that so i mean i don't know if i if i if we answered rupesh's question but uh yeah i mean this is part of the i kind of view it as like part of the job you know so yeah so with that um this isn't a survey, but we ask, why do we automate things? Right, yeah.

So diving into Chapter 7, all about the evolution of automation in Google. They start off just by kind of looking at basically the reasons why you would automate things. And these are all kind of obvious things, but it's kind of nice to just split it up a little bit to make sure we're all kind of on the same page. The first one is kind of on the same page the first one is kind of obviously uh it's consistency so people make mistakes you know even on simple tasks maybe even especially on simple tasks especially if you have to do that simple task a million times like how many times you uh brush your teeth and realize you're using the wrong end of the brush you know something i do i don't, once a month or so. For real? No.

Wait, I don't know that you're lying about that. I like that Joe trolled Alan and Alan totally fell for it. And he's still not convinced. Well, what's it say about being that it's believable that I could have done it? Well, I was thinking like a better example is like um how many times have you just like misspelled common words right like i mean oh yeah that that's that's a simple simple task you know like you're typing in an email with just you know the language you speak every day and for some of us it might be the only language we know and we still like mess it up yeah i try to not even type variables at all copy paste them

and like outlaw bust me today on like having somehow dropped a letter on a word this is crazy yeah you know you know what's insane that they brought up here and i actually remember this from from working at a college campus is they brought up like school campuses and whatnot how they typically have people that manually do a bunch of steps all the time, right? Like a new person comes in, uh, I need you to set up this account.

I need you to log into this. I need you to do that. I need you to install this, right? It's a bunch of manual things and somebody is going to miss something along the way, right? There's just too many things.

And so the consistency really is key. I think more than even some of the other ones that come up here. Yeah. You know, there was, um, it reminds me of a story. So, um, one time I was working on the company and had filled out the, uh, the paperwork, you know, when you start the company and one of the things you had to fill out was the, the match for, um, for 401k or not the match, sorry, but how much of your, your paychecks you want to put into 401k, which is like a uh american thing for basically retirement savings and i typed in a number and

got my first check and it was wrong i was like that's like doing the math so i contacted hr and i put this number in it looks like this number is coming out and they're like oh yeah i must have typed it wrong like Like, wait, what? Yeah, right. I typed it into the form. What do you mean you typed it? They're like, oh, no, you typed stuff into the form. Then I get an email with everything you typed.

And then I go type it into all these other systems. And so I must have missed somewhere. And that was just amazing to me that there was someone actually going through for doing that and basically doing data entry and HR and like a modern day company, you know, but. And you know, the insane part is that was probably several years ago.

Um, I remember just a handful of years ago through a very popular HR company that was out there that did the same thing. You'd type in stuff and behind the scenes, they had a bunch of people going into other systems and entering it into the other system. So they were selling a platform that they integrated with a bunch of other things.

The reality was you'd fill stuff out through a nice front end and then they had a bunch of people going out and logging into the other systems, setting things up. And so you'd have mistakes all the time. I bet we're talking about the same company. Oh really? Yeah. They had a really nice website.

I was like, no, this is the best hr experience i've ever had and then beautiful i can't was it called uh does have a z or something are you gonna you're gonna throw i wasn't gonna throw them under but i remember i had a similar thing though because and i remember i remember too uh like it did but not not to beat up on them, though, specifically. But I mean, I remember I ran into a similar thing related to setting up your insurance options. And part of the reason that they said that they did have it as a person entering it in is because all these different systems that they were, quote, integrating with, not all of the options were the same or supported across. And so that's why they didn't have anything automated on their end. But I don't know, like that seems whatever.

I'm not going to like, you know, judge your business. definitely if there's anything that we've learned through this, this journey on coding boxes, that like whatever you can do to do things in a repeatable way, the better you are. Right. Especially like, like, you know, as our CICD type conversations,

which is where this like automation kind of comes from. And specifically the consistency, right. Is coming from is like the more things that you can do in a repeatable fashion, then, you can do in a repeatable fashion, then, you know, in repeatable, having a person type something in is not something repeatable. It's a task. It's, it could even be toil. It might not be, but it's definitely not, it should not be considered repeatable, even if they do the same thing three times in a row.

Yep. So I think anyone who's written software that's been used by people know that consistency is a you know a nice dream um now the next one is not as obvious at least it wasn't to me the reason that google says they automate things is because they're building a platform and automation begets automation, which means basically you can extend and grow and fix bugs in and combine automated tasks into bigger and bigger automated tasks. And so it's kind of like the idea of like almost like starting a garden or something, you know, the idea being that the work that you do to automate things pays dividends. Every time you run that automation or the automation gets run, you're saving time that you would have had a person doing that. And so it's providing value as opposed to somebody going through and doing something manually, which is literally a cost center. And also has the higher chance of going wrong and costing you more money to kind of fix it. I mean, we have a, you know, a recent example of that in our day job where like

we automated a, uh, you know, uh, one of our processes in, and, you know, somebody on the team bothered to do like some back of the napkin math and came up with like, here's, here's like a ridiculous amount of money that was saved per month just because we don't have to have an engineer go and do that task manually that was being done for a while like somebody would you know go and do uh you know like a build and deploy kind of operation of some code uh you know out to a specific environment and and now they didn't have to do that and you know so to your point like you there can definitely be like real savings in in doing these things.

Yeah, totally. And centralizing is important, too. So back to this notion of platform, this notion of platform comes up over and over again throughout this whole chapter. Platform centralizes the logic, too. So you think about kind of having a centralized kind of code base or automations, your configurations all in one spot. So you don't have to jump around to computers or go ask someone how to do something you've got it all in one say you know repository or one centralized location doesn't necessarily have to be one repo or anything but uh basically a centralized kind of i don't know uh was a nucleus of control i don't

know well that's what they called out here, right? Like if there's a problem in that centralized system, that's doing things, that's one place you got to fix it. Right. And then it fixes it for everything. As opposed to if you have a bunch of people out there doing things and you find out that you had a bug in your, in the process to onboard somebody, right. You now got to go tell everybody that does any onboarding, like, hey, don't do this thing here. And it just, it doesn't scale well when you're dealing with people or anything like that, right?

So if you have a centralized system, not only does it make it easier to do, but it also makes it easier to fix things. Yeah, totally. And things drift. You imagine if like, there's some tasks that everyone does and everyone does a little bit differently.

Like Alan's got a script, alan's got a different script and i was too lazy to do a script but maybe i found some way that kind of shortcuts something so i you know i've just reset it or something every a couple hours um but and then all of a sudden you figure something out wrong you know with the process say okay we all need to do this uh additional step somewhere in the middle then you've got to communicate that to all these different people. They need to figure out how to integrate with the processes that they're already doing and adapt. And it's just painful. But it's centralized.

You fix it in one spot and you're done. Also, if you've got a platform and then you start doing these things more and more often, the more you do it, you start getting kind of like economies of scale. So you can start getting metrics and like economies of scales uh scale so you can um start getting metrics and using these measurements to make better informed decisions about you know things kind of going forward and how you want to spend your time and resources and kind of tying into that faster repairs is the idea that the more often automation runs you hit the same problems with it so you imagine you've got a task that runs a thousand times a day.

It's going to run into network problems or it's going to run into timeouts or it's just going to have things go wrong in it. Compilation errors. Compilation errors. If you're running it a thousand times a day, you know, after one month, two months, two years, four years, you're going to see a lot of the same problems over and over again. And you're going to figure out, you know, how to solve those problems quickly either by scripting around it or just, you know, having some kind of manual fix.

But the idea is that the more often you run it, the more often you hit the problems, the faster you get at repairing it. So over time, the average cost of fixing those errors drifts down lower lower lower until it basically gets as low as it possibly can go well and also too like what my experience has been like what i've seen happen is that like the things that you start fixing the improvements you start making they start becoming more and more like granular as you get like like in the beginning, it might be like the compilation errors, right? Like you're just trying to get the thing to build right in a consistent and repeatable pattern. And then, you know, you might have that as part of your PR process, like it's automatically kicked in and eventually like, it'll get to the point of like, you know,

Oh, the, Hey, there's these network timeouts. Maybe we should use something else as a, as a proxy, uh, you know, to cat like. Maybe we should use something else as a proxy, you know, like an artifactory or something to cache external dependencies so that we can kind of like reduce our dependency on the external internet, you know, or connectivity to it. And then it might be like, hey, you know what? These build times, like let's improve these build times and get those like, you know, I mean, two minutes was okay,

but you know, hey, what if we were able to cash a bunch of things and get a two second? And like, you just start making like these improvements over time, you know, to where like, that's not the thing that you, you know, you didn't start with trying to get your build time down to two seconds, but eventually you were able to iterate to that point to where now, because your automation keeps getting mature and the things that you work on the improvements that you're making like what becomes a problem today you know is so much more minute than what was a problem yesterday right and so like that that's one of the uh advantages to like setting these these things up yeah totally yeah totally um and i'm using build as a sorry but i'm using build but obviously like you know google isn't necessarily only talking about building things yeah i was just using that as an example yeah totally and i think i think it's a really good example and we'll get they have an interesting uh i don't know how to see the um i

don't know if i want to say definition of the word automation, but they have an interesting take on automation where they say, I'm sure this is somewhere up ahead in the notes. It's going to trip me up later. But they say that they think of automation as being these things that kind of like sit outside their systems and like orchestrate them. Um, thought that was pretty cool. Uh, faster action.

So obviously there are things that a computer can do much faster than human. You know, there's only so fast that you can kind of click buttons or enter commands or run scripts. And if you've got a, uh, or wake up in the middle of the night to create a new pod. Exactly.

Yeah. Can you imagine if... Yeah. Can you imagine if Kubernetes worked that way? Like every time it needed to auto scale, it would be like, hey, let me page Joe or Alan and get one of them to create the new pod for me. Yeah. What's the turnaround time on that?

But I mean, so that's an example of like I'm being kind of extreme. But, you know, in faster action, it doesn't necessarily need to be that you couldn't like click the button fast enough. It's just you might not a person. I'm sorry, Alan. A human might not even be available, you know, in a reasonable amount of time, especially like in off hours. Yeah. And another good point there is that there are things that as you scale would be prohibitively expensive to just have a human sitting there doing. So, you know, maybe when you've got, you know, 100 different servers running or, you know,

you've got one data center running, you can have a team of people that can manage that. When you get to a hundred data centers, you just cannot hire enough people and kind of keep that many people active and solve that problem. You have to automate. Otherwise you just can't do it. You can't grow to that size. So you can't scale people that well.

Hey, one thing that they actually called out along those lines, and it might've been further in the chapter. I can't remember either, but they said, you know, a lot of these things that they talk about here are specific to Google and the fact that Google has these scale issues, right? Like if you work at a small company, that's got four or five people on a developer team, chances are you handle a lot of that stuff manually because you sort of can, right. But they've grown to a point to where they can't.

They just, they couldn't hire enough people to do all that. And it wouldn't even be cost effective to do that, right? So there's definitely this, you've got to keep in mind that there's a scale of a company like Google or an Amazon or a Microsoft or whatever that may not match for a small company, but it doesn't mean you couldn't get these same type benefits if you started pushing for some of that automation in your smaller company, right? Like how awesome would it be if when your boss asked for a build, you didn't have to

go in there and do something yourself to get a build out and get it over to him. What if there was a push button, you know, package and ship type thing? So, um, you know, they call it out and it is worth knowing push button you know package and and ship type thing so um you know they call it out and it is worth knowing that you know they are aware that they are sort of special because they have a lot of huge systems yeah and i think um originally i thought about maybe we should skip this chapter for the show whatever because it's so google centric but part of what i thought was so fascinating about the idea of this kind of platform is that it seems like Google thinks of themselves as a platform. Google is a platform.

Google is building and investing in these systems, not just for their search and their Gmail and all their other products, but also their HR systems, their employee systems, their ability to kind of spin up clusters and sell Google Cloud. If you think about it that way, it's like they really think about this almost as like kind of investing in themselves. They want to own the source code for their business. And that's not just the products. It's also the business itself. And I think that's a really cool idea. Yeah. And they even called that out explicitly like you're talking about. One of the reasons why they said that they like investing in building their own software is because when you buy off the shelf products or whatever, they don't have the APIs and things built in that allow them to get the visibility into how the thing's running, right? Like the metrics that we mentioned

or APIs to see how things are running or getting data out of it. Like they want it because they know, because they've done it so well for so long now, they know what they need out of that software to know if it's running effectively and doing what it needs to be doing and all that. So it was a really interesting take

from their perspective on why they prefer to build versus buy. Yeah. Yeah. I thought that was really good. And if you think about like, you know, if you're a company that large, like why would you want to buy something that you're not going to be able to integrate with the other systems? So why buy these black boxes? Like you really need to be able to kind of own that stuff. And that's, that's something as a large company, again, like you, like you said, like you have to think about things differently than if you're a scrappy startup. Right. Yeah. I mean, if your core competency as a small startup is to sell widgets, everything you do needs to be geared towards getting those widgets sold, right? You can't

be investing in writing your own logging library or your own metrics gathering library. Like there's reasons those tools are out there and you should leverage them, right? Yep. Google, I'm sure, runs their own SMTP servers. They run their own mail servers. Yeah, totally. They run millions of billions of emails through every day.

If you're a startup, you should not be trying to run your own mail server. It seems like, though, it seems like though, like, I don't know if it was from a book or, or where, but it seems like we've ran across something at some point where it was like,

you know, you should use the tools that are available, whether you like they're commonly available or you had to buy them or whatever. And until you grow big enough to where, you know, it no longer meets your needs. Right. So if, if using like Alan mentioned a logging library,

so like if using a log for J is good enough to get you going, like focus on your core business until that thing becomes no longer, you know, doesn't do it for, it doesn't do what you need it to do. And now, you know, there's some kind of business need why you need to change it or whatever. Or even like, you know, you mentioned, um, as it relates to like, uh, using other services for like, you know, HRs or whatnot, you know? Um, yeah, I mean, focus on, focus on what your core business, the problems are at that time until it, until you outgrow it. But I don't remember where we came up with, where that came from. I remember hearing that too at some

point in the past, but I couldn't tell you. I'm going to go ahead and credit it with Uncle Bob because it's probably in one of the Uncle Bob books. That sounds like something he would say. So the last item here is time-saving. When they talk about time-saving, we already mentioned how it's faster to action, faster to actually do these

automations and faster to human doing it. It's faster to fix. But whenations and faster than a human doing it. It's faster to fix. But when they say time-saving, they're actually talking about the human cost, meaning if something's automated, then it can just be run by a simple script and anyone on the team can do it. We don't have to find the specialized expert

who has the knowledge and maybe the shell script and their home directory or whatever, right? So the time to actually do this thing is really fast. It's whoever gets to it first or whoever gets the page and, you know, you've got it as opposed to kind of having to scramble whenever there's a problem. So it just a kind of different take on saving time there.

It's interesting. But they also called out, right. That in order to get those time savings, you have to invest the time in creating whatever it is that will do that. So if you have something that takes away 15 minutes from a person every day, it might take you a day to automate that. So there'll be a payback period before that thing breaks even and ultimately ends up paying for itself over time. But you've got to bite the bullet and get that, you know, get somebody working on the

problem to get it done. Imagine if you had to manually take time out of your day, QA comes up to you, they tap you on the shoulder, tap, tap, tap. Hey, I want to test this version of the code. And you're like, oh, let me go check out that specific version of the code. I'll compile it. I'll build it. Okay, now i'll go deploy it for you somewhere and there you go there's your there's your thing right versus if you've actually like automated all that then they can just click a button and away they go right right i mean that that's you know clear example of this time saving

idea where like now anybody and it doesn't even have to be QA in this case, it could be like a salesman who, who wants to like be able to demo the latest thing to a potential customer, right. Or, or anyone like, you know, C-level executives that want to be able to show shareholders or board or whatever, like, Hey, here's the thing right now, you've like literally automated the thing to where anybody can do it. Yeah, absolutely. Um, we have a fun little quote that relates to that, uh, coming out. Like you want to try reading this? Okay. If we are engineering processes and solutions that are not automatable, we continue having to staff humans to maintain the system. If we have to staff humans to do the work,

we are feeding the machines with the blood, sweat, and tears of human beings. Think The Matrix with less special effects and more pissed off system administrators. Yeah, so it's kind of a funny idea of like the humans kind of powering these machines and powering this process, which is kind of funny idea of like the humans kind of powering these machines empowering this process which is kind of the reverse of what you want to be doing with computers right you have

computers to to replace the the human need to do those repetitive tasks so yeah and that quote was from joseph baronis yep yeah i'm so i wasn't gonna be able to pronounce that name so i literally didn't say the name because i was like oh oh, I'm going to butcher this name. And I was going to call out, I was going to make a joke to call out to have you say the name, Alan. So you saved me that embarrassment. Oh, wait a minute. I don't think it happened that way. I didn't save the embarrassment at all.

Yeah, nice. And the next section was they had a couple paragraphs on the value of sre google and this is all stuff that we already kind of hit on like google has a strong bias for automation because of their scale and google is a massive company do we know how many developers they have um just like i think it's nine because they've automated everything so um wouldn't it be like super crazy if it was like a much smaller number than you expected like automated everything. Wouldn't it be super crazy if it was a much smaller number than you expected?

What if it was like, they have 57 developers around the world. I have a number from last year. Do I want to take a guess? I'm going to say it's going to be a strong five-digit number would be my guess. Of developers, not all employees, right? Software engineers. What a strong five digit number would be my guess of developers. Not in, not all employees,

right? Software engineers, software engineers. What's strong five digit mean? That's very vague, sir. Um, I'm probably going to be like super low,

but I'm going to say 25,000. Wow. Wow. That was really high. I don't know. You were real close. 25,000. Wow. Wow, that was really high. No, no. You were real close. Yeah. That's good.

27,000. Isn't that what you're saying? Yeah. Yeah. Okay. Now, how many employees do they have? That's the number I looked at, 150,000. Yeah.

I remember for the longest time, IBM was one of the largest uh employers because they were i think like around 150 but then there was like a there was a german company i can't remember that made like medical equipment and they were closer to 300 000 um wow so yeah that's a few. Yeah. Yeah. Just a few. Just a couple people. So, yeah. Sorry, I was just reading.

So, Google's core is software, right? I mean, that just makes sense. If you've read, if you've even spent any five seconds learning anything about Google, you could know that it was about software and creating software. So, they don't want to use software. They were, this is kind of what we've said before, where they don't own the code and they don't want processes in place that aren't automated. You can't scale tribal knowledge. So, and that's the worst thing, by the way, like you, have you ever been that person that knows something and like you get hammered, like everybody will keep coming to you for like that thing. And it's like, why don't you just write it down or automate it or do something like here? I never want to talk about this thing again. Right. I think we even

referenced something similar to like, um, I think it was Scott Hanselman that like every time somebody would ask him a question, he would never respond by email. He would just write a blog article. And then that way, if a second person ever asked him the same question, he could just be like, there you go. Go away. It's brilliant. Yeah, I mean, that's brilliant. With 150,000 employees, how often people are getting locked out of their accounts or needing to create a new account or quitting, transferring, changing their 401k contribution rates. I mean, how many employees would they just be dedicated to doing that?

You know, it's crazy. I had a friend who worked for kind of a hospital complex, a couple of local hospitals. And a lot of his job was like resetting passwords and stuff because there were like nurses and doctors on the floor that don't want to be filling in the password. So they needed to be able to call a number quickly and have someone kind of do that. But there was a pretty large staff for, you know, a couple hundred employees. And that's all they did all day long was kind of deal with like sysadmin type stuff like that.

And yeah, that doesn't it doesn't scale very well. I mean, it doesn't even have to be a Google scale. Just imagine like you're having like a Thanksgiving or Christmas where all of the family and extended family are coming over to your house for the holidays. Right. And immediately everybody comes in and like, Hey, what's the wifi password.

Right. When you just like to have something just like automated, like boom, they came in and like, okay. And you, you now have the credential. You're good.

As long as you're in this radius and then you're out, you're gone. Yeah. You're good. As long as you're in this radius and then you're out, you're gone. You're off my network. So the last little bit that they have here on, on Google's value of the SRE is Google wants to invest in platforms, right? Because when they build those platforms, then there are things that they can improve and extend over time. And that goes back to them not wanting to use other people's software, right? Because they have an idea of where they need the software to be, and it's sort of its end state because they've been doing it for so long. But really, in the grand scheme of things,

they haven't really been doing it, quote, that long. I mean, it's only been around since, like, what, the late 90s? I mean, you're talking about the heyday of all development though. Right. Since, since like software development became as huge as it has been. But my point is, is like, they're not as old as like an IBM or an AT&T or, you know, whatever. Right. Yeah.

Like even an HBO, for example, you know, like they're older and then you have this like new kid on the block called Netflix and it's like, hey, let me show you something. Right. So. Yeah. So I guess we got somewhat, you know, last time. It worked. It pains me, Joe.

It pains me. That's what it kills me. Because I dread what you're going to say. But I'm going to ask you to do the bag for the reviews, but I'm probably going to regret it. But then when the next episode comes in, people are going to make me not regret it. And I feel like that's worse because it only encourages the bad behavior. That's right. Right?

So here we go all right i'm gonna do it i'm just gonna i'm gonna shoot straight here uh i don't trust that that's already off to a bad start you're lying already whatever tells you something like that you just know know. It's going to be great. Like, wait for it. Imagine we're a bunch of vampires, right? Okay. But instead of needing blood, we need reviews to live. Right? Way better than blood, right? We can all agree.

And we even try to make it easy for you. So if you want to feed the vampires, we set up a URL, comingbox.net slash view. And even if it's a terrible view, it's still pretty good. See? That's where you took it too far.

You were doing okay. You're fine with vampires. It was weird. I'm not going to lie. It was weird. But I was like, okay, he's got this analogy. It's weird, but it's working. And then you took it off the cliff.

And then he sucked the life right out of us. Yeah. I've been watching a lot of Twilight lately. I don't know. You know, I thought you were kind of sparkling. Thank you. All right. So how about we start off with a joke?

How about that? Let's do. What do you call a group of programmers? How about we start off with a joke? How about that? Let's do. What do you call a group of programmers? I don't know. An assembly. Oh, nice. That's pretty good.

That's pretty good, right? Yeah. I saw the survey that's coming up. Jim Hummelstein, he sent us this link link i'll have this in the show notes but it's the ultimate list of programmer jokes and puns and other funny so yeah we'll have it in jim excellent thank you yeah one last one how you know how to you know how a hacker escapes the FBI no backslash FBI

okay I like it there's going to be more of them in there it's a good list I'm telling you this is this is there's some true gold in here but so now we head into my favorite portion of the show, Survey Says. All right. So a few episodes back, we asked, so, I mean, we're talking about this whole Google SRE thing, right?

So wait a minute. DevOps is a culture, but SRE is a title? And your choices were... Wait, what? Or... Yeah, I get it. Or... Eh.

All right. So this is an odd number episode. So according to Tateko's trademark rules of engagement, Alan, you are first. So I'm just going to go with the fact that I think every, Oh man, this, this is amazing.

This goes along with something micro G shared with me with a optimism, pessimism and whatever. But at any rate, I'm going to go with most developers are pessimistic and I'm going to say, meh. And we're going to go 50%. Meh. 50% okay.

Joe? Right away Joe before you say anything mathemachicken like this is already in your favor like I think from a mathemachicken point of view you're

I have faith that you can pull this off. I feel like he's giving you hints, man. That's not fair. Yeah, so I got this. No, I wasn't. That totally wasn't a hint. Yeah, I get it. I'm just saying it's more than two, so, you know. I get it. No, I mean, that's my answer.

Yeah, I get it. Oh. And for the percentage, alan did 50 uh so uh you know i'm gonna go with uh i'm gonna go with 90 bold spicy he failed so the hint didn't work i tried to help you there man i tried there tried. There's a hint? I tried. So, yeah. So, Alan says, meh, with 50%.

And Joe says, yeah, I get it. 90%. Alan wins. But he overshot, so he lost. Oh, doggone it. Yeah. Really? Yeah.

Meh was 40% of the vote. Oh, wow. It was close, though.one it. Really? Yeah, it was 40% of the vote. Oh, wow. It was close, though. All right. Yeah, I get it. It was second, 32%. So, you know, there's hope. There's hope in there.

So, yeah. How about if I ask you this, then? What do you call two monkeys who share an Amazon account? Primates? Yeah. Yes, primates. That's it. Primates. Yep. Primates.

Okay. Mates? Primates? Primates. Well, instead of a primate, because a monkey would be a primate. Yeah. Well, yeah. True. I like the additional pun on top of primates. I do, too. I think Joe's answer is better. I like it. He wins.

Fine, then. What is a... P-R-E-S. Okay, so a thesaurus is the only dinosaur that can read. What does a thesaurus eat for breakfast? I don't know. A cinnamon roll. All right. So for this episode's survey, we ask...

Synonym? Oh, man. I got it. Got it. Man, that one took longer than I expected. It sounded like you said cinnamon roll, but maybe that's what I was trying to hear. Yeah. Yeah.

Maybe I did. Okay. Sounds good. I'm sorry. All right. Well, then for this survey, I asked, or we asked, what's your, you know, because there's this new Tom, everybody's making a big deal about this new Top Gun movie, right? I don't know if you've heard, but there's a new Top Gun movie. So everybody's making a big to about this new Top Gun movie, right? I don't know if you've heard, but there's a new Top Gun movie.

So everybody's making a big to-do about it, right? So we ask, what's your favorite Tom Cruise movie? And your choices are Mission probably something, something. Or Top Secret Gun. Or Risky Company. Or Risky Company. Or Edge of Today. Or Majority Report. Or Rainy Men.

Or Tom and Jerry Maguire. Or A Few Okay People. Or Interview with a dead guy. These hurt so bad. Oh, it hurts so bad. It's so hard to choose just one of those delicious titles. Well, you know, like some of those, some of those, like, I think I was putting this together.

I think I might do this for like all of them from now on. Like they might all be silly stuff like this. because this one I had fun just trying to put together the titles. But the Tom and Jerry Maguire, that was among my favorites on there, especially since it's Tom Cruise, so it had an extra little oomph to it. Ah, yeah. But then also Top Secret Gun made me feel a little sad, because that like one of Val Kilmer's first movies, if not his first movie, Top Secret. And they were both in Top Gun.

I did like it that you had an interview with a dead guy in here after Joe's take on the review as well. It kind of fit, right? Right. I thought that that's where that came fit, right? Right. I thought, I thought that that's where that came from when, when he was talking, I don't know. Did I,

did I influence your review there with that? Did you cheat? No, I just play a lot of video games. Oh, right. Okay. That makes sense.

That checks out. All right. Well, a lot of novels I think is what it is. Let's keep talking about, uh, automation and specifically Google's use cases for automation. Yeah, so much of Google's automation is around managing the life cycles of systems, not actually

the data, which I guess is kind of obvious, but it was just kind of interesting to think that they have the distinction. We talked about how they kind of view automation as, I think I jumped ahead to the spot, where they kind of think of it as being software for software, like meta software. They did mention a couple of tools that they use, including Chef Puppet, which if you're not familiar with, I think of those as kind of centralized tools that like manage things like installations and kind of, I've seen it for desktop software. I'm sure it's,

you know, all sorts of stuff. It's like Kubernetes, but for like things that you can go poke at, like would, it's another way of like building automate of like defining how you would install a system. Right.

Except yeah, it's not a Docker file. It's like a real machine that's there. I know when we've seen Chef for Puppet, they're kind of similar. But what I've seen in the past, it would be do things like every workstation that comes up needs to have a security client installed and the certificates installed needs to be running this version of Windows. And the same thing works for servers, though.

So you could like spin up a data center maybe and say like every data center needs to have early. Every DNS server needs to have these things and every load balancer needs to have this stuff or uh it can kind of go crazy and uh just kind of gives you ways to basically manage uh these kind of more complicated distributed systems you know and can reach like really far out you know like i mentioned workstations and um like load balancers and stuff like things like into the real world like i would be surprised if there's somebody out there running a farm with like chef or puppet real world that's that's be surprised if there's somebody out there running a farm with Chef or Puppet. Real world. That's a good way of saying it. Reaching out into the real world. That's what I meant by you go poke at it. It wasn't

just something like a container running on your machine or a Kubernetes cluster full of mini containers. Yep. Totally. CF Engine, which is something I've not heard of before, but it's basically for change management. So it looks pretty cool.

Here it comes. Best one, Perl. Specifically mentioned Perl. So obviously these are tools that kind of work at different levels of distractions. Like Chef, maybe you'll have a recipe or whatever they call it that defines a data center

or some sort of storage block or whatever. Perl is going to be something that you're going to do for different kinds of things, you know, different kinds of tasks, like much smaller things. Like writing your websites in Perl. Hey, people do it. They certainly used to do it a lot. It used to be super popular.

I know. It's crazy. I mean, it's not crazy. It's like, it's perfectly fine and did a great job. It's just, it's kind of fallen out of favor.

Well, that's why I found like, you know, consider it. Cause I think the date of this book was sick. 2016. Is that right? And,

and Pearl still made it, it's way into this book at 2016, which was – I think it's fair to say that Pearl had fallen out of favor of mainstream by 2016. Oh, for sure. Well, you know what they said about it, though, was it has access to low-level system type things. And that's why, right? All the other ones were sort of higher-level abstractions. And they said Perl got you down to dealing with system-level APIs. And so you had more finer-grained control,

but that comes at a cost, right? That's basically what they were getting at. I mean, they could have picked any language, though, for that. Yeah, they could have picked any language there for that. Yeah, they could have gone to C or anything. But I guess Perl's more scripty, right? So it's a little bit easier to deal with, I guess. So, I don't know.

So yeah, confirmed 2016. Some of the authors or some of the people thanked the mentions were all like, SREs until 2013. So let's, you know, just because the book came out in 2016, it was going on before that. Yeah. They moved on to something else, you mean? Yep.

But just the fact that they were SRE in 2013, before the world has heard that term, was interesting to me. So, we mentioned Perl being a very low-level tool. so uh you know we mentioned pearl being a very low level tool you could do a lot of things a very granular system level uh you literally write to standard out and you know things like that um higher level abstractions like things like chef or puppet are easier to work with and reason about but they're leaky uh so like when we talk about kubernetes like it's a i guess i would call it a high level tool, but there's a lot of stuff that goes on underneath. And there's a lot of layers of indirection before you get to something actually running on a VM somewhere and getting traffic started. Like the way you, the concepts you talk about and the terms that you use are just pretty far away from like what's actually happening on your cloud provider of course, or on-prem or, you know, wherever you're running it.

And, you know, high level obstructions are great because they're easy to work with reason about and you know you can say with um programming languages like the higher level it is the more stuff you could be the more expressive the languages are you can do more with less but they're leaky so things that can go wrong include like a partial failure, a partial rollback, or a timeout where you're not sure if the thing happened or not. And so the higher level the tool is, the less it's able to handle these little kind of intermittent props because it doesn't know what you want to do. If it is a partial failure, what do you want to do about it? The further back, the less you have to do of set this stuff up and get it to run. It's basically like saying another, another way of saying it is that like you have,

when you get to higher levels of abstraction, then you have less control. Yeah. And the reason you have less control is because it is a higher level of abstraction. So there are cases where like, you know, there could be something very specific,

low level that happens. Like it failed to open a socket and you're not going to be able to know. You might not be able to catch that at the top level, you know, whatever you're working in, you know, so you're you know, you you don't know why it failed or just that it failed or, you know, kind of thing like that type of thing. But what's weird about that is typically when you hear a leaky abstraction, that usually means that it's abstracted, but it's giving you a lot of visibility into the layer that it's touching. And their example that they mentioned here doesn't really, doesn't follow that. So I don't really know what they meant by the leaky abstraction but they did give an example of like let's say that you had something that was doing a rollout and it was supposed to be pushing binaries out to to a bunch of different clusters right or whatever and things could fail in different parts of that binary being pushed out

like maybe the binary didn't make it to a system Maybe it did make it to a system and it didn't stop, stop the old one and start the new one. Or maybe it did get there and it did run. So, so they were basically saying these higher level abstractions have problems and that they may not be able to deal with things that happen further downstream because they're not even aware that they happened, right? Because they expect things to happen in a particular way, right? Like release the binary, restart the thing. And if something happens somewhere in between there, then it's not going to know about it. But again, that's why their term leaky abstraction is sort of confusing here.

Yeah, because normally you would think of it as like, you know, you're going to have an interface like how to create a user but that generic interface that could like go to any number of providers also has something very specific to like one or more of those providers and it's like well wait a minute like you're actually leaking details of how you how the underlying implementation is is working and i don't want people to know that yeah or care about it. Yeah, and that's where the leaky abstraction at least when it comes to code and stuff is how that's communicated. But it sounds like all they're saying here is your lower level stuff like Perl, awesome because you can do things, but it's more complicated. Your higher level abstractions are great until they're

not, right? And that's kind of the trade-off you run all the time with layers of abstraction right so yep i'll say it's like a script that calls the script that calls scripts calls script and something goes wrong there and then somehow it's got to bubble that information up to a spot where you know someone can make an informed decision about what to do but the the options and just everything that might need to all the information you need to to have in order to kind of fix that are not necessarily present in the machine so i think of that that is kind of being almost like the leak that's where things kind of go wrong and uh you've got to actually

get in there and kind of diagnose it and look at logs and do whatever you need to do uh get past it uh i got a couple examples here of uh automation um so So account creation, termination, that's something we talked about. Cluster setup and shutdown. So when they say cluster here, they think about just a group of computers. I mentioned DNS, load balancers, things like that.

Authorization servers, whatever. Certificate issuers. Software install and removal removal uh kind of mentioned that too with the security agents that sort of thing chef um a bit yep this is definitely what i think of their uh software upgrades so not only your software aware but also system upgrades so you want to upgrade your linux kernel or something like that windows uh windows uh configuration changes like you're making some networking changes or something, you know, changing your servers around.

Dependency changes. Now, this is a one where it's this is a little bit trickier because it's something that your system depends on change. But maybe maybe your code hasn't changed, but there was a patch deployed that fixes some security vulnerability or something so you need to not only look at the things that you have told your systems to deploy but also the things that they're built on so imagine like um i don't know the log for j uh cve whatever vulnerability that had a couple months ago now uh you have to know as a system administrator that your which of your systems are running code that's vulnerable to it.

And that may not be the code that you ran, but the framework that it runs in, the other libraries that it brings in, the company blog that is running somewhere, if any of those dependencies need to be upgraded, then it's going to have a cascading effect. Speaking of, I remember you've heard about, like, where GitHub has capabilities, like, where they will, in your repos,

they'll find, like, hey, you have this dependency on blah, blah, blah. That dependency has these bugs, blah, blah, blah. And I think I recently heard something. Tell me if you guys heard about this, too, where, like, Google is now going out after like projects and they're finding, you know, like, Hey,

this project has this bug, but also like, here's the PR. Does that sound? Did you guys hear about that? I'll see if I can find it. I hadn't heard about the bug. I knew about with dependencies,

like they started creating PRs and stuff or upgrading software. I think I've even had it happen you know what's interesting about this list that you had here like you just rattled off about seven or eight of them but they also said this list could just basically go on and on forever right like finitum yeah because any anything that you can think of that they could possibly automate, they will try to. Right. I found it. Google announced this is the beginning of last month.

So or well, basically about a month ago, Google announced that they were creating what they called the open source maintenance crew tasked with improving the security of critical open source projects. That's pretty cool, right? I mean, it's kind of off topic to what you're talking about, but what made me think about it though, was just the dependency changes though. Like,

uh, because now Google is doing it, Google scale. And they're like, Hey, it's not enough that we do this for us. We got to do it for your stuff too. But I mean,

the reality though, is that like how interdependent the world is on open source now. So it's not – I suspect it's not entirely altruistic that they're doing it. They benefit from it too because they use a lot of open source as well. So it only – it helps them to be more secure too. benefit from it too because they use a lot of open source as well. So, you know, it only helps them to be more secure too. But, you know, related to the configuration changes, the example I was thinking of there was like,

I mean, nowadays, especially with like Sarbanes, you know what I'm saying? Sarbanes-Oxley. Yeah. There's like controls where like you'd probably want to, like, you know, specific credentials for auditing systems, you know, here and there. But, you know, you can think back to a time in the not-too-distant past where, like, you might have had, like, a password for, you know, some credential. Like, oh, hey, here's the DB reader password.

Here's the DB writer password, right? That multiple systems might've been using. And, you know, if you wanted to like roll the password occasionally, uh, you know, depending on how many systems you had, that could be a big deal. Right. So, you know, that rolling passwords was kind of an example of like, uh, a configuration change that I was thinking of that, you know, automate that as much as you can. Right. Even though there are like, I know there,

there's like other, uh, services out there to where like, you know, the, the, the secrets can be shared, you know,

and, and retrieved as needed. Yeah. Totes. Totes. Is that what we're, is that how we talked now? That's where we're at.

Yeah. That's what the vampires. Okay. Well, I am the youngest. So just at, yeah. That's what the vampire says. Okay, well. I am the youngest, so just saying. Hey, wait, are you? You are. I think you are, yeah. I mean, I look it.

Wait a minute, I thought I was the youngest. You still got hair, for sure. Yeah, 21 still. Thank you, thank you. All right, so the final section we're going to cover tonight, it's the hierarchy of automation classes. And this is the

idea here that there's kind of a maturity model. We've talked about this with security. We've talked about this a few different times.

It's kind of a repeating pattern and kind of technology and probably just the universe. But the idea is that ideally you

wouldn't need to stitch any systems together because they would all kind of just handle their own problems and they would all have APIs and, you know't need to stitch any systems together because they would all kind of just handle their own problems. And they would all have APIs and, you know, components that work together and were designed to work together with all these different systems. And everything would just be perfect and you wouldn't have to do anything ever.

But that's not the real world we live in. We have systems that are separate and we have a bunch of glue code. And that glue code is especially susceptible to bit rot so basically you know one of the systems upgrades maybe it removes some old options add some new ones and changes some things how you know how it does and so your glue code is now maybe out of sync and then your other system that you're gluing to maybe it's got other you know configurations and things and options added. Now it's kind of out of sync with that original system.

And the way your glue code works doesn't really apply anymore. And those things kind of drift over time. And so the problems might be kind of subtle. And it's really hard to know. And it's expensive to go in there and kind of test this glue. Could you ever add code where you're like, you did something that integrated two systems like three years ago, and you gotta go figure out why it's not working anymore and you like don't remember anything about it and then you look at the apis now and they're like totally different

and the the api you're using it was apparently depredated like three years ago you can't get a key again sorry outlaw i was gonna it's funny you mentioned that because I think we talked about that as an example in a past episode where like the Slack changed how they create their tokens. You know, they're completely different now and, you know, from one to the next. So, yeah, I definitely am facing that. Yeah, even when Slack changed our APIs and so the plugin we used to let people join automatically and stop working one day. Yeah. There's another one. Yeah. And didn't even know until people started uh not being able to get in but we do have that up uh by the way if you go to your coinbox.net slash slack can uh and you can join uh so i mentioned those maturity levels uh so the idea here is kind of that the more rare and risky a task is the cheaper it is to basically to deal with, the less likely it is to be fully automated.

So in this example, we'll talk about a database failover, which we've talked about in prior episodes as being something that a lot of times a human is involved in because of all the different things that go wrong and severity of kind of how bad you can munch your data if things don't go well uh database rollovers are kind of a sorry failovers are kind of a famously difficult problem and a lot of databases are specifically written around the the kind of goal to make that a little bit easier like we mentioned uh cassandra well i was thinking like not all not all database systems are created the same. So could you imagine like failing over multi-region in like, you know, going back to like a cloud example, like multi-region failures, like, you know, especially like it's some of those problems are okay in like an active-active kind of situation, you know, or not even active-active, just like

the other thing is available, right? So you have like the hot, the hot region and then standby region. But, you know, if your strategy is I'll spin up the other region on an as needed basis, you know, like automating that would be pretty, you know, pretty tough, but, you know, and there might be like legit cost reasons why you only want to spin up your, your other region on an as need basis. So I can't imagine why you would take the time. It's rare and the truckload of risk in it. And so even if you did go through the expense of trying to automate that process today well things might change in a year when you actually need it to happen and so are you still going to trust that the automation is

going to work or would you still want a person there like checking that as it went right so yeah i mean that's an example that comes to mind yeah totally so that first level is basically no automation just like you said like you know you you've got a site making 150 profit a month uh you're probably not going to spend three months you know practicing and setting up the database failover you know automation because it's just not worth it it's probably never even going to happen at that scale uh second level is what they call externally maintained system specific automations so what we're talking about here is that a SRE, a developer, has a couple commands that they can run, and they basically got some information in the notes,

or maybe they saved a stack over a full link. And so maybe they've got a shell script, their home directory, something like that. The idea is that it's externally maintained, which means it's the person who is who did not write the system uh so in this case you know developer our company is supporting a database that is from some other vendor and they're maintaining it and they're just you know keeping it to themselves and they can they can run it if you need to and that's kind of the second level

uh next one up uh externally maintained so, this is us running it, not the database vendor. Generic, but still system-specific automation. The only difference is here that SRE adds that script to a playbook or throws it in the wiki or checks it in somewhere, and so now anyone can run it. So it's the evolution in that now anyone can run this thing, but it's still externally maintained. This is much better than having no automation for a failover. If I started a new company and saw they had a script for failing over databases, I would think it's something that they're probably pretty good at. They've done a few times.

I'd have a lot more confidence in that script than I would me Googling my way through a manual failover. That's pretty good. But here's where things start getting even better. What I'd like even more than having some wiki with a script for failing over would be if the database itself shipped with a script that will let you do it. And so you could go shell in a box and run database failover and that's it. So now there's nothing, no special knowledge.

It's responsible for doing that stuff. It's presumably been tested, you know, by a whole bunch of people. That's the trick is it's, it's, it's continuously tested and maintained as part of the product. Not like that first example that I gave where like,

you know, you do it a year later. Yeah. And this is the experts that, you know, you do it a year later. Yeah. And this is the experts that, you know, you've probably got some sort of support contract. And this also assumes though that like now the, like the, the rarity is, is probably becoming less, right? Yeah. And that's why you bothered to automate it in the first place.

And now like that automation is just maturing as you go. And we're kind of getting into bot territory so you know if you're buying it you know into a database or something like if i'm paying for a ten thousand dollar a month license for uh you know microsoft sql server they better have some tools for me to fail over uh you know because i'm paying good money so i'm sure like the the more expensive database solutions like this these are the things that they're able to add on and kind of get their value add uh over like you know other ones well well worth calling out though is sure on like paid for databases but but google here is even talking about their own things right like so yeah so this this could be that they're building in failover scripts into their own thing

right like uh what is it? SQL spanner and that kind of stuff, right? Like they, they probably have things that they've built into their own product to handle their own failovers as time goes on. So it could be a paid for off the shelf thing or something that you're doing yourself that you're,

that you're baking into your own product. Yep. Are you ever seeing like a woodworking equipment or something that's got like an emergency pull cord or something that you know so it's like if your shirt gets stuck in the saw pull this cord real fast and it'll shut the engine off that's kind of what these scripts are and you're glad that's there still pretty scary though right well it's kind of quick man there could be a problem that you have to do something very important we provided that important thing for you but it still doesn't really make you feel great about using the product right right well yeah along the same that same line of thought though have you seen the the blades or the the saws where it will automatically stop

as soon as the next one yeah that's exactly yeah i i don't know that i could ever like i'm glad it's there i'm glad it's a thing like i i look forward to the day that it becomes so commonplace that like every saw every circular saw you ever want to buy is just has that it's not even something you think of as a feature anymore but oh my i don't know that i want to trust it yeah yeah and if you don't know what he's talking about just google or go to youtube and search for table saw with um blade stop and they'll show people like running a hot dog and as soon as the blade makes contact with the thing it'll barely nick that hot dog but it'll destroy the actual innards of the table saw to keep it from cutting a finger off. It basically injects a break into the blade.

The center of the blade can break as it does it. And it works off of like, it sets like a current. It trips like a current signal. Like when it touches that other thing, you know, like the hot dog example or, you know, your finger.

And, and yeah, to Alan's point, like it barely puts a, a dent on your, on the hot dog or your finger. But you look at like how fast that thing is engaging. Cause it's literally like shooting a gun to like fire,

uh, uh, uh, like a pen to, to break that, uh, saw blade. And it is read.

Diculously quick. Would either of you stick your finger in there to test it out? No. Yeah. Me neither. Yeah. Nope. Yeah.

So, all right. No, because it would be my luck that it would be like, you know, my long flowing locks would get caught in it instead. And so like my skin would never touch it. But, you know, my heavy metal long hair would get caught in it and it would like totally rip, you know, all the hair off my head. Yeah, definitely. totally uh rip you know all the hair off my head yeah definitely could you are you picturing me now with like long flowing locks you know like the scolding lock yeah about that fabio furl jeez i would hate for that to get damaged yeah be like uh chaining tatum in that in that in lost city you know i didn't see that that was i didn't enjoy that movie actually so uh the last one here

like i took us way off tangent but basically you know the example with the blade that i was describing is this last point that the system doesn't need automation the database notices and automatically fails over yeah isn't that automation like how is that yeah automation yeah definitely automation but there's something about it being kind of owned so you know example here be like the the blades you mentioned are like anti-lock brakes or you ever see pressure washers like anything dealing with electricity and water a lot of times they'll have a self shut off switch so they'll throw a breaker if the energy starts moving too quickly or whatever gets overloaded so i feel much better

about those systems because I never have to think about it. You know, um, it just kind of take care of it and it doesn't scare you. You know, it doesn't come with that script that you know that one day you might have to run.

So don't forget about it. Um, so yeah, it's just, that's the kind of the ultimate dream is that your system doesn't need automation and also that all your systems are going to work together. None of it's going to need anything. Just going to take care of itself.

Self self-correcting. Yep. That's kind of, I think, the promise of the cloud. So when you use a managed service, something like S3 or whatever, that kind of idea is like you don't need to worry about the automation. You don't need to worry about data centers.

Your stuff is up there. You've got this many nines for it. And you're kind of delegating that part to them. And that's huge. It's huge. There's a big difference between number four and number five here. It kind of goes back to the beginning of our discussion of this book where there was the quote that I'm probably going to mess it up but it was something about like they didn't want um you know what was it they didn't want automate they

they wanted not automation but self-correcting or something like that like i don't remember now but do you know the phrase i'm talking about yeah we looked this up last episode too oh we didn't remember we forget every episode. Automation, they wanted autonomous systems or something. Self-correcting systems, something like that. They didn't want continuously deploy. Oh, man. I'm going to go look it up.

Think about the difference. Just how you feel about this statement. Think about the difference between seatbelts and cars and having cars that can't ever crash. That's the difference between number four and five here that we're talking about. It's a huge leap from a script on my home director to a script in the wiki. That's much smaller conceptually and much easier to get to than it is to go from number four to number five here. Yep. Oh,

then you have one last one here. Go ahead. They want systems that are automatic, not just automated. Yeah, that's what it is. Dang it. That's a great quote. We wish we could just remember it.

Yeah. Yeah. I won't even bother to put it in the notes. We'll remember it. Yeah. Right. So, uh, no, Scott, one last thing and uh so we're breaking before we get into any sort of case studies or anything but this is kind of a theme that they um kind of come back to a little bit in

and some of the kind of case studies and so i thought it was an interesting question to ask here. Can you imagine a scenario where your company, your group automates so much that the devs are unable to actually manually support something when a real non-automated problem actually occurs? And do we mean that because the dev doesn't have experience with it or because things are just so complex that you couldn't possibly hope to do it? I would say a mix of so complex and also you're just so far away from that. It's like the difference, like you think about like me using it as like a computer user, if a Photoshop crashes or something, I restarted, like I'm not about to go diving into system calls and like trying to debug like you know i'm probably not even gonna look for a log file or anything because i'm so far from that or even better like if i pick up um uh or here oh i got one uh i push uh i push the button on a

toaster right to toast the bread and there's no power like the things that i'm gonna do to fix that are very limited right uh because you know ultimately if i trace the problem back to the power station like what am i gonna do as a human like what i'm saying is like you can get so far away from the the problems that you can have that you're just unable to fix it. If my toaster died tomorrow, I wouldn't be able to fix it. You know? Well, I was thinking like cloud storage providers, be it Apple, Google, Microsoft, Amazon, whoever, you know, they can't,

if you can no longer read your file or, you know, read a line from your file, they can't help you with it anymore, right? Like they don't have, that engineer doesn't have access to it right they can't see it in the raw to like help you you know i've been struggling with how to say this um do you ever hear that story about the guy who tried to make a toaster from scratch that's why i use toaster a guy got this idea he's like i'm gonna make i'm gonna smelt the metal. I'm going to smelt the copper. I'm going to make the wire.

I'm going to coat it in a, you know, non-conductive material. And I'm going to make a toaster from scratch. And he actually went and like mined the material. It took like, I forget how long, like 10 years or something. And it actually ended up catching fires in the museum. It's like a famous story. I'll find out. Like with shout outs.

But it was just the idea that you're so far away from things like just basic things that you don't even think about like imagine if you uh got dropped in the desert somewhere and uh your phone got smashed by a rock when you fell you can't fix that phone right you're so far away it's so complicated it's so the tools that you need to even work on your phone to fix it it's not practical it's so the tools that you need to even work on your phone to fix it it's not practical it's not something that you in your lifetime could build up the tools to have so i'm saying is like can you be so far away from the underlying building blocks of your systems that if when they do go wrong you like can't really fix it you just have to hope it works when you restart it i think that would be a good thing to shoot for,

because if you get to that point, then you've done such a good job of insulating yourself from all that. Like, I think that, well, here's the example. Here's the example. Um,

first of all, I want to answer your desert question. I would just die because I wouldn't even know which direction to go. My phone is the compass. Like, how would I even know? Like what, where's North? What are you going to do? You just got to die. Yeah. You just die because I wouldn't even know which direction to go. My phone is the compass. Like, how would I even know? Like, where's north? What are you going to do?

You just got to die. Yeah, you just die. Just follow. No one ever lives in a desert. That's just a fact. Right? Pretty sure. You can check my math on that, but I'm pretty sure.

I saw it happen in Spaceballs. No, no. They only combed it. They did comb the car. So, but to your automation question, though, here's the better analogy, though. Cars. It used to be that, like, the average person could just, like, buy a car and maintain it. But now they've gotten so crazy that, like, it's unrealistic to expect you and I to be able to maintain a common car now. And that was before we even got into electrics.

Now with the electric vehicles, it's, it's, you know, there's even an extra step there because now, uh, you know, you might not even have access to the actual software that's being used to run

that car. Right. So you, I mean, this isn't exactly, I don't think what you, you were talking about, but this is the closest example to a real world that I could think of was like, you know, you, you don't, you no longer have the ability to maintain that thing yourself when, when things go wrong, it's difficult. You know, a random light comes on.

You're like, well, I don't know which sensor that was. You can go by, you can go by a reader to tell you what sensor it is but even if you knew what sensor that is that doesn't necessarily mean that you easily have access to even get to that sensor because it might you know where it's physically located within the car you'd be like oh forget it i can't yep i would i would have to disassemble like half the car and that would just take me like you know the rest of my year just to disassemble it and then i'd be like okay now where was that baggie of screws that i had yeah knowing that you kicked over the plate that the screws were all that's why yeah they're everywhere that's why you eventually i just put them all into a bag you're like i'll figure it out yeah yeah and so

wordpress's example uh whenever that we have a problem with like a WordPress, it's something that I know if I put time in, I could learn more about how it works and how to kind of diagnose it. But it's so much easier just to restart the service and, you know, it usually is fine. But it's kind of an example. You can imagine like a company that doesn't need any developers. They've got a WordPress blog or something with a Shopify plugin. That's how they make all their money if that goes down they don't have the skill set to to maintain it or to fix it but they were they've been reaping the benefits from the automation you know like they're able to run a successful business with no no it staff at all um but when they're when they're

in trouble they're in trouble right when you were going on about your, the toaster example though, where I, no, no, no, I didn't mean that in a bad way. Uh, where I thought you were going with that was similar to the story of,

and I know Alan, I bet has heard of this one. Have you heard about the guy who built the Lamborghini in his basement? No, he spent, no, Alan surprised me. There was a guy who spent 17 years building an a

lamborghini in his basement and then decided he wanted to sell it and if i remember the story correctly he had to like have a wall of his house taken out in order to get the car out of the basement. Like that's just, I mean, Hey, kudos. The guy built like a 1980 Euro spec Lamborghini Countach LP 5,000 S like it was everything, your real taillights, everything like it was awesome. And if you saw the pictures of it, you're like, that is incredible. But yeah, i don't know why he didn't like think ahead and it's like you know if i build this car in the basement i'm gonna want to get out like let me build an airplane in my dining room that's insane absolutely yeah so we'll dive into more about automations and we'll hear some stories

that are kind of similar to what you just said about the airplane Lamborghini but yeah we'll dive into that a little bit next episode and so with that we'll all have a bunch of resources

links to resources we like in this episode especially the Lamborghini one so that Alan can catch up on his reading. With that, we head into Alan's favorite portion of the show. It's the tip of the week. I've been doing some development again lately, which is nice. I ran into something that was extremely frustrating. So I'm using Spring in Kotlin slash Java Spring Boot. And I was using the Spring Mongo JPA, whatever it is.

I don't know. It's something special. But here's the deal. Go look it up, kids. The Spring Java JPA, something special, you know, whatever. Yes, yes. So here's the deal. Go look it up, kids. The Spring Java JPL, something special, you know, whatever. Yes, yes. So here's the deal, man. Like we've talked about Spring is, I'm convinced.

I think I said this to somebody the other day. I'm convinced that if you actually knew Spring inside and out, you could write an application faster than anybody else could in any other language. You totally could because it does so much for you. However, if you don't know a lot about the Spring framework, you'll spend 10 times as much time writing it because you've got to figure out how things are supposed to work. At any rate, I had something that was using Mongo templates in Spring to do some database calls and I wasn't getting data back. And I was like, well, what in the world, right? Like this, this is magically going off and doing something. And apparently you guys know how, like in SQL server, they sort of have a history table.

So you can see what queries ran. You can even go find out the performance of the queries that ran, like there, there are specific queries and prox and stuff that you can run in SQL server to find what the last 10 queries were, whatever Mongo doesn't keep track of that stuff. Um, now I did read that there's maybe a version of Mongo or an enterprise version to where you can get some auditing turned on and it will write that stuff out somewhere. But by default, from what I understood, Mongo does not do this. Now I'm probably going to get 9 million comments. If you know differently than what I read, please feel free to share it on this episode's

discussion, which would be codingblocks.net slash episode 187. But so with all that out of the way, I needed to see the query that was running because it wasn't getting anything back. There is a setting for spring that you can just put in your application.properties, if that's what you're using. It might be application.yaml, whatever. You know, one of your 50 ways of configuring spring, you can set this logging.level.org.springframework.data.mongo2b.core.mongoTemplate equal debug. Just rolls right off the tongue. It does, right? Right.

It will actually output the queries that are being sent over to Mongo. I didn't catch that the first time. Could you repeat that? No, I can't. What was that setting? My spit's trashed. We're going to have this in the show notes.

So you're like, well, I didn't, let me rewind. Don't worry. It's in the show notes. It's in the show notes. So you're like, well, I didn't, let me rewind it. Don't worry. It's in the show notes. It's in the show notes. But the cool part is when you turn this on, then literally every time a Mongo statement or a Mongo query is made using Mongo template, then it would output that to your debug logs and you could actually see it. And for me, I got to see that I was using the wrong database name.

I was apparently substituting in the collection name for the database name, and thus it was never coming back. So having that one piece of visibility saved me hours of frustration and rewriting code that was never going to work, right? So there you go. So Spring almost automated you out of the way to be able to manually support the system. I was hoping it would have. It almost did. That was the goal to be able to manually support the system. I was hoping it would have. It almost did. That was the goal. That was the goal.

Yeah. So that was my bonus points. Have we ever talked about actuators? I don't think we have. No. Yeah. So here's a suite of configurable and extendable APIs that you can add to your Spring application that let you actually control things about the Spring application. Like, for example, changing the debug level live without having to do a reload.

So if you actually wanted to turn on debug logging for Mongo template and you have the actuator enabled for logging, for example, then you can actually go hit that resend point and say, hey, something says debug, do your thing, and then turn it back off again without doing any sort of deployment or anything, which is really nice in some cases, but not something you're going to want to expose to your customers. Probably it's not, you're going to want to keep those actuators locked up somehow. Yeah. Regardless of who,

where it's deployed to or who has access to it, you probably don't want it for everything because there's probably going to be some security-minded things. Even with the Mongo example of outputting the queries, I was thinking there might be some things in those queries that you don't want persisted out to a log. Totally. This is for local development type stuff.

And also the actuators. So one other thing to tack onto it, they're amazing. If you are running a web service or something with the web API, right? Because I tried to turn this on for a console application. And because there was no web endpoint for this thing, it didn't exist. So this is perfect if you're setting up a web service or if you have a website or something like that um but if you have those this is an amazing way to be able to do do changes on the fly without having to restart your system and all that

yeah that's also how they do like health checks and a few other things like that are just kind of nice little meta things about your spring is so nice it rip man i swear to you especially spring boot like i think we've talked about this in the past i would not want to live in the xml world that spring was prior to what we've been dealing with which is spring boot spring boot is just glorious once you learn how it works it is it is a joy to use. You like how we just started talking about this after we got some news about Pivotal? Yeah.

Anyway, this is inside baseball, but we might be worried. Anyway. VMware owns Pivotal. VMware owns Pivotal. Yeah. Do some more work withmware who knows yeah okay happen well this is awkward um okay you want to just talk to us about your

tip of the week then it's not weird um leave it to joe to make it weird yeah i know it's so late it's like she's 10 anyway that's why um so check this out there's a new but non-open source product from grafana called on call that helps you manage production support so it looks almost yeah i gotta compare it to like PagerDuty, which kind of manages like who gets the alerts, what the schedule is, what it integrates with, how those alerts happen, all that sort of thing. But this is a new product that is built off of

and extends Grafana from the Grafana people. So if you're heavily invested in Grafana and you need to set up some sort of on-call schedule or you're starting to kind of dip into notifications and you're finding it becoming a lot of toil to manage with those kind of production support issues, then this might be something you want to check out because Grafana is insanely popular. So I don't know if this product is going to take off. I think it's only been out and really talked about for the last couple of months. So, you know, who knows, may just be a flash in the pan, but I'm really interested in it

because it's such a popular product and so many people have it installed already. It kind of seems like a no brainer to just kind of go with them to take it the extra mile. And then you can figure your alerts and who's getting in your schedules, all that stuff in one spot, because Grafana is the king of the hill right now in terms of, you know, if you're running your own kind of metrics and stuff. And so if you don't already have something like a Datadog, whatever, if you're doing your own stuff, like it seems like a no brainer to me. Yeah, it does say, though, that it's based on the on call open source project. OK, so so there's some sort of open core

to it. There is an open source version of it, yeah. So like you said, I mean, that's the key is that if you're running your own metrics, then this could be helpful if you're heavily into Grafana. Yep.

If you're rolling around. The pricing's not terrible. Wait, wait, wait. It's not what? It's not terrible? It's not terrible it's it's wait wait wait it's not what not terrible it's not terrible that's right this i mean it says so they've got a free plan but then they've also like their pro plan is eight bucks a month per active user yeah that's nice so if you've if you've got three people that are on call that's's not terrible. Or if you have a pager that you rotate around to people, you basically have one active user, right? Like it's, it's not terrible. So it's totally affordable. Yeah. Pretty cool, man. Yeah.

Not terrible. It's not terrible. Uh, okay. So for my tip of the week, I'm actually going to steal this one from Angry Little Hamster asked. As part of their comment, they said, it's understood that you should run with the lowest privilege possible, but no one really gives a good way to go about it. Could one of you have a discussion or give guidance on how to do this correctly? I think I missed the part there where they were talking specifically about

Docker. If I remember the full question correctly. So there actually is in Docker, a user command where you can specify either a user by name or ID, as well as you can specify the group that the user should be a part of user by name or ID, as well as you can specify the group that the user should be a part of either by name or ID. And in the Docker file documentation, they actually show an example of where you could first create the image off of a Windows core server and then use a run statement to add a user and then a subsequent user statement where you switch to that user. And that user command will then instruct that everything from that point forward, you know, if you never changed users again from that point forward, then when the Docker image runs, it's going to run as that user.

Or you're going to, you know, when inside of that command, inside of that container, things are running as that user. Wait, does this work for Windows containers? They actually give a Windows example. Do they really? Yeah. Yeah. I wonder how that's doing. I haven't looked at like a state of the state of the Octoverse, whatever

type server in a while, but I'm curious to see how many people are running Windows containers. I would imagine there's a lot because Azure probably is. Yeah, I'm so far removed from that ecosystem right now. Yeah. Interesting. Good stuff. is right yeah i'm so far i'm like removed from that ecosystem right now but uh yeah yeah interesting i wish it well but i i've been like here of late like really more like alpine all the things i guess like make it as small a core as possible and just add on what you need.

I don't know. But I guess that's what Windows Server Core is doing, too. The idea behind it is as small a possible Windows environment. But also, too, it's like, why do you even need Windows specifically to run your.NET? You could do that in Linux. That's what we do.

Yeah, nowadays you don't need it. Yeah. uh here's one other tip of the week a really silly one do you remember the days of like when uh you know you when you first got your ios devices and you connect it to itunes by a wire crazy and you know that's when it would start this sync process and you could, it would show you a screen of all the pages that you had. And you could rearrange the pages however you wanted. You know, like, oh, this screen of here's all my media apps. They should be like the first one.

And here's like, you could rearrange all those things, right? Here's my reading apps. They'll be the third page. You know, whatever. That was like my single life before I got married. Yeah. Yeah. That was like the weekend for me.

And, and you thought it was a big deal when they like, then you could sync to iTunes wirelessly and you could still do stuff like that. Right. Well, but then like eventually they kind of got rid of, you know,

using iTunes to manage your, your screens. And, and, and you're like, Oh, I guess why I can't manage what screen is what anymore, like where those were.

Turns out you still can. On iOS, there's like a bunch of little dots there at the bottom that like indicate what page you're on. And I remember like, you know, in one of the recent ones they added in this ability where like you could like spread, you could slide over the dots real quickly to navigate between the screens. If you didn't know that.

But if you just press and hold on those dots, you can rearrange the pages of your, of your icons. It'll take you into like that old school looking kind of view where it's like, nice, you could rearrange them all. I would love to get excited about that, but it's,

it's, it's not, but it's also, it was also like, it's not interesting and exciting, but hear me out on this one. It was like, hey, somebody might find that useful. And like, this is literally the tip of the week. So, hey, don't be knocking on my tip of the week, man.

That's totally fine. I mean, it's one step closer to Android, so it's only like nine million steps away i don't well whoa whoa whoa whoa whoa in fairness i have no idea how long that's been there it could have been there maybe that feature has been there for a decade i don't know i just stumbled across it by accident and i was like what the shut the front door i am going to share that with people because that sounds pretty neat. It's well-deserved because I can promise you where this is going to come in handy is somebody is going to accidentally have screwed up and dragged one of those dots over and be like, what happened? Right? So this is helping that person out.

The anxiety levels will go down because of this tip. I can't hate. I'm talking about vampires anyway. So interview with a dead man or something like that. Yeah. Something like that. All right.

Well, um, I need to, I need to leave so I can go watch the latest episode of top secret gun. So, uh, you know, with that,

if you haven't already subscribed to us, uh, you know, with that, if you haven't already subscribed to us, um, don't necessarily take all of Joe's advice, but, uh, definitely subscribe to us. Uh, you can find some helpful links,

uh, at the top of the page for iTunes, Spotify, Stitcher, wherever you'd like to find your podcast. And, you can leave us a review. We greatly appreciate it.

Uh, it always does put a smile on our face when we read how we've helped people throughout their career or in during their career. And it really does mean a lot to us. It's one of the little ways that you can give back that that really does matter to us. So you can find some helpful links there at www.codingblocks.net slash review. Hey, and while you're at www.codingblocks.net, you can check out our show notes. Forget it. Well played, sir. Well played. All right. Uh, well, uh, make sure to follow us on Twitter at CodingBlocks or Web5 over at CodingBlocks.net.

And you can find all our social dailies at the top of the page. Don't you need like a keyword? Like what's the keyword for this? I got to get on the internet and I need a keyword to find it. Yeah. Yeah. Keyword CodingBlocks.