Coding Blocks - The DevOps Handbook – The Technical Practices of Flow

Episode Date: July 6, 2020

We begin our journey into the repeatable world of DevOps by taking cues from The DevOps Handbook, while Allen loves all things propane, Joe debuts his "singing" career with his new music video, and Michael did a very bad, awful thing.

You're listening to Coding Blocks, episode 136. Subscribe to us and leave us a review on iTunes, Spotify, Stitcher, and more using your favorite podcast app. And check us out at CodingBlocks.net where you can find show notes, examples, discussion, and links to our music videos. Send your feedback, questions, and rants to comments at CodingBlocks.net. Follow us on Twitter at CodingBlocks or head to www.CodingBcks.net and find all our social links there at the top of the page. And with that, I'm Alan Underwood.

I'm Joe Zach. And I'm Michael Outlaw. This episode is sponsored by Datadog, the cloud-scale monitoring and analytics platform for end-to-end visibility into modern applications. All right. Well, let's get on with the show. And we are going to be talking about an exciting book, I think, the DevOps Handbook. And I expect to be trolled the entire time as to whether or not it's a role or a culture. So we'll see where we land on that by the time we finish this book. That should a, this should be exciting.

Yeah. So before we do that, uh, I guess we should go through some, some reviews here. Definitely. So I've got iTunes this time. So I've got gal, the Jewish hammer,

Ragnar, wreck, Ragnar, Roick, Marcel, seven, four, seven,

three, eight, two, six, and our Costco 31. And I got to say, Marcel, even though your friend hates us and thinks we're not funny, Marcel7473826 and Arcosco31 and I gotta say, Marcel even though your friend hates us and thinks we're not funny, thanks for listening

that's amazing yeah, I wanted to know which one of us sounds like Kermit oh, probably me sorry he said oh I've heard that before about myself and I hate it. Thanks for bringing it up.

Well, this is awkward. I wasn't prepared for that. All right. Was I the only one that just assumed like, oh, God, is he talking about me? Oh, no. Jim's even said it in the past. Oh, you have? Yeah.

Yeah. Oh, okay. Well, I guess i didn't remember that but uh tell us you're recording live hey dude hey so if it makes you feel any better and i'm gonna hate this because it's gonna come back after this episode now when i used to do my headphone reviews inevitably i would get a comment about being like, does he sell propane? So it was... I was like, man... I'm just really waiting on Joe to say something like, hello, this is

Kermit the Frog reporting live. All right. From Stitcher, we have Bicycle Repairman and Bruno All right. I'm going to start singing that. It's not easy being green. Yeah, right? From Stitcher, we have Bicycle Repairman and Bruno LC. All right. And speaking of reviews, so like we said last time, I asked for 17 reviews. I made a mistake, and we got them. So I really appreciate the reviews. It's a bit unfortunate for everyone that now I have to do some singing.

And as promised, I have prepared something. I actually wrote a song for this. So I hope you all enjoy. It's basically about the inevitability of writing bugs and the inevitability of writing really bad bugs. And so here we go um the scariest bug is the bug i haven't written hold on that's not right hold on let me uh let me have some reverb here um and oh let me uh auto-tune okay here we go. in Dyson hoping it's not the big one yet that's excellent the reverb helped.

I think the reverb definitely spiced it up a little bit. Yeah, it's amazing what you can do in the studio now. It's crazy, too, because with the auto-tune, it doesn't even sound like you. Right? Yeah, I know. It's crazy, huh? So, by the way, I made a music video for that. So, we've got a link here in the show notes. You can find that on our YouTube channel.

So, put a lot of time and effort into that. And that crosses off my bucket list. Well done, sir. Thank you. Also, hey, wanted to mention that Alan is still MVP. How many years has it been of you being a Microsoft MVP now, Alan? Yeah, it's pretty cool, right? Like, it's super flattering like honestly and obviously you

guys are a big part of this and coding blocks and forcing me to go out and talk and do videos and all that kind of stuff so um 2016 was the first year so this is officially my fourth um you know mvp thing which is it's super cool right like it's an amazing community so yeah really really honored to be a part of it and still be in it. So thank, thank you guys for helping me. And, uh, that's awesome. Yeah. It's a, it's congratulations. Thank you. And you have like a trip that you can kind of slide the rings on. Is that right? Yeah. Yeah. Yeah. I was about to ask, I would go grab it. I'll get a picture of it for next time, but yeah, it's got rings that stack up. I I'm hoping that I will eventually make it to the top of that thing. There are people that have needed an additional one.

Wow. I'm not quite there. What do they do? Is there like a process to reach back out and be like, hey, I've MVP'd too many times? No, I think they actually have it. They know how many rings fit it, and then they will actually send out another one of those little things to, to put them on. So did they never change that base then? I don't know if it's a different one, but there are people that have been in the program since like the beginning,

which was like, I guess 18, 19 years ago now, something like that. It's been around for a while. So yeah. So eventually they just know that you've MVP yourself and you need a new base to put it on. Which, you know, the funny part is I'm surprised that that's ever happened, that it's filled all the way up.

Because I would imagine that Microsoft just ends up hiring most of those people at some point, right? At which point the trophy no longer matters. Because you don't get MVP if you're a Microsoft employee. Yep. Right. Well, that's awesome, man. Congratulations. Thank you much. Thank you much.

Awesome. So I put in the notes here, so I'll kind of step up here. We're talking about the developer's handbook, or the DevOps handbook, rather. Sorry. And what we decided to do was actually skip a big portion of the beginning of the book and kind of hop into the meat.

And I'll tell you, so I've listened to this book once, and this is my second time going through now. And I really hate the first half. So I was pushing for us not doing that. You've listened to it? Yes. There's an audiobook version, which I should get a link for. Yeah, it's on Audible. Oh, snap. That's the way to do it. Yes. Uh, there's an audio book version, which I should get a link for. Um, yeah, it's on audible. That's the way to do it.

And so the part we're covering tonight, I think was about, um, I don't know, like an hour and a half or something on audio, maybe an hour. So we'll probably go, I don't know,

four or five. Uh, Hey, in fairness, I will say the reading I've been doing of this book, because you guys suggested it, this is absolutely a book that would be well consumed on an audio book. Because there's not a ton of technical bits in there that, you know, there's not code and that kind of stuff, right? There might be a few diagrams you miss out on, but we're really good at describing diagrams.

So you can get that part here. I don't know if you've heard, but imagine an upside-down triangle. I saw that comment in Slack the other day. But I purposely said triangle and not like a pyramid. Right. Because then you're like, well, wait, which way is up on a triangle? Which side is facing me on the pyramid? Can you tell us the angles?

That helps. Yeah, but no, seriously, if you can, and the audiobook is roughly the same price, because I bought the Kindle one basically because I wanted to be able to take notes and read it and all that. But if I was listening to this thing just to consume it,

I would absolutely get the audiobook instead because it would make it a whole lot easier to consume. Okay. So this book review is going to be unlike every other one we've ever done before because the three of us are reviewing all three formats of the book because I have the physical copy, which I've had for a long time. Joe has the audio and you have the Kindle. Right.

Yeah. I don't understand this whole hard copy thing. Like, I don't get that. Unless you're planning on clubbing somebody out and about walking down the street, like, why would you carry one of those around? Because I planned on using it as a weapon. Right. Fair enough.

So I did want to hit a couple other highlights before we kind of dive in here. I kind of felt like we needed to describe this book. This is called a DevOps Handbook. But the first part of the book is really kind of a collection of arguments for just DevOps and as a culture and kind of describing what DevOps is. And you got to remember this book came out, I think, 2016. And DevOps is kind of a newer word when you really think about it. It seems like it's been around for a while, but it's not really that old. And so the first part of the book is just kind of the argument for DevOps. And I kind of feel like the world has kind of accepted that DevOps

is good now. So I advocate for skipping that part because it feels like a sales pitch for something that I've already bought. But what's really important about the book what i like about it is that it's like high level guidance for understanding the spirit and the culture of devops and understanding what the people need to do and what you're responsible for in doing devopsy type stuff and so you're not going to find any you know yaml in here there's not going to be anything specific to technologies but it's really just more about the way that you need to align your processes and the And so you're not going to find any, you know, YAML in here. There's not going to be anything specific to technologies, but it's really just more about the way that you need to align your processes

and the way you work in order to get the most benefit. Yep. Hey, and real quick, I do want to call out. So that's a really good description of what this is. I don't know, and we might've said it in the past, but DevOps, it never clicked to me that that was truly development and operations, right? Like that's what the term is coined from is the joining of those two and, and what they mean when they work together. Right. So DevOps is truly your development code plus operationalizing that stuff. So, you know, it's worth calling out because it just,

I don't know, I guess I hear a buzzword and it just, I'm like, okay, whatever. It's a buzzword. Like, I don't, I'm not thinking about it, but it was, it was cool to see that married up like that. Yeah. It's about a recent time. Before we get into some of the meat of the, of the topic, I do have some questions. Like, so one going through from the audio book perspective, like are, are graphs or charts, like, are they mentioned at all? are they mentioned at all or they describe it all or do they just completely skip it and you don't even know that you're missing that i didn't know that they were there but my uh my terrible brain also tends to totally ignore details um even when they matter a lot and so if they did mention the graph i probably just

you know ignored it okay i got you yeah there are some in here so i was just curious to see like how that how that got translated from you know written to verbal i did pick up the uh the kindle version so i've got two formats and when i saw the grass i was like oh that's weird and even i've been re-listening to it too in addition to looking at it and i didn't even on the second listen through i didn't realize that there were any charts gotcha and then uh one last little thing was that we we had this conversation before we started recording and we quickly were like hey stop let's just have this when we when we are recording but uh one of the things about this book that I know I've really enjoyed is that there are case studies throughout the chapters as you're reading through it. So there might be

examples from Amazon or Etsy or Hewlett Packard or all kinds of different companies. Google is in here. The point is there's companies all over the scale you know all over the spectrum some companies never even heard of or i hadn't at least and and i enjoyed those case studies but maybe not everyone did i don't like case studies in general um as part of you know it's that part of my same part of my brain that just like wants to kind of like roll its eyes and plug its ears when it hears about the details of something like that to me the case studies are just kind of examples kind of proving out what they just told us and i kind of feel like i'm taking for granted that like you've done the research and that you know you're talking about what i'm reading in the book and i'm going to kind of take that information and kind of like try to apply it to my own and kind of project my

own experiences on that and see how they uh kind of come out and so the case studies to me just feel redundant um so yeah i just i don't so the case studies to me just feel redundant. So yeah, I just, I don't really like case studies. Man, that's really interesting because I'm completely the opposite. Like anytime I'm seeing something, I prefer seeing the case studies. And I will say, usually I want the case study that is not trying to paint the same picture that whatever the thing is trying to shove at you anyways. But like a third party case study, we're like, hey, yeah, we did this

and it improved by 25%, 30%, whatever, right? Because then it's like, now you have other than just some ideas that people are like, hey, you should do this because it'll make your life better. Instead of that, it's like, yo, if you do this, you'll get 30% of your time back to focus on things that matter, right?

So being able to quantify those things with case studies, to me, is fantastic. So I personally enjoyed the case studies that they did in the book. For me, it wasn't even so much about making it, about quantifying it, although there were some where they did that. It was more about the case studies made it feel more real and tangible. It made it more relatable because you could read something and maybe as you're reading, you're like, oh, I don't know how you'd apply that. And then you read a case and you're like, oh, okay, now, yeah, okay, I get it. Actually, I've done that.

I have done that. Now I can kind of solidify that concept that they're describing. Yeah. I'm definitely a fan of it. But it is funny, like three different perspectives on it, right? Which is what we always have. All three of us have worked on this stuff for a long time. And we always have different takeaways from either the learning medium or what we did and what we got out of something. So I don't know.

Yeah, that's really interesting. Yeah, that's really interesting. Yeah, that's good. Uh, Oh yeah. I wanted to do that. Um, three ways.

Oh, Oh, sorry. I was just gonna say that, you know, I think a big part of the book is really about kind of scaling, but in a kind of a different sense. Um,

like we've talked about scaling, you know, when it comes to like distributed systems, but DevOps to me is really a way about kind of scaling the people and the work that you're doing like scaling features and how people get work done scaling teams people environments all that sort of stuff like that to me is like the true benefit of devops like that's what it means to me well their their capabilities let's say not necessarily the number of people working, right? Because one of the examples they gave was a Google case study that you probably hated.

And it was only like eight people working on the team, but yet those eight people were delivering something like, I forget if it was like Cloud Storage or Cloud SQL, but it was like one of the big GCP products. And it was only eight people that were delivered that were responsible for that that product so it was like more about scaling their capabilities yeah and you can't do that if those eight people are spending half their day like ssh and rdp and copying files and sending links and having meetings because they didn't document why something happened so they've got to kind of spread that tribal knowledge. Like it just doesn't work. Like the people don't scale like that.

And one last thing I want to do to mention is we've talked about the Phoenix project several times. And I think recently we talked about the unicorn project, which was another popular book. These books are both fiction and they're both related to DevOps. They're both super good. The author of those books, they were both kind of written by multiple people, but the kind of the main person recognized for those books is Gene Kim, who's also kind of the main person associated with this book. And so that's something I wanted to call out. But really,

if you look at all the authors on the book several of them have written uh like devops kind of oriented books including the book continuous delivery and i forget the others i should have written it down but uh just really kind of notable books in the industry so it's cool to see kind of like all these kind of big authors and influential people in the kind of devops movement have all kind of come together to like just blast out everything they know on the subject. Cool. And you said, seriously, read The Phoenix Project first. Yeah, it's so good.

I haven't read it. This book, honestly, I already said it, but it was a real struggle for me to get past the first part just because I know, I know, I know. But The Phoenix Project, it's fiction, so everything's grain of salt and things work out a little too conveniently, of course, I know, I know. But the Phoenix Project, it's fiction, you know, so everything's grain and salt. And, like, things work out, you know, a little too conveniently, of course, because it's, like, you know, a short novel. But it's just so good.

And I think it really embodies a lot of the spirit that this book is trying to convey. And it just does it in such a digestible, good way. And it's on Audible. It's a really easy read. I think it's anything, like, all developers are going to be able to relate to a lot of things that happen in the book. A lot of the ups and downs. And then the sequel is really good too the unicorn project it kind of modernizes some things but really i think you get the gist of what this book is all about from those two works of fiction cool so what are we starting to cover this evening here

okay i guess i okay one more thing so this book is uh it kind of defines what they call the three ways um which is kind of a weird to me way of talking about um kind of the spirit of devops and so they've kind of chopped up into like three kind of bigger spirits or principles and then we go through and kind of break those principles down. So the three principles are the principles of flow and then feedback. And there's a principle of continuing and experimentation. Tonight, we're talking about the principles of flow. And we'll talk about what that means. Excellent.

So let's get started with the principles of flow. And, and what they, what they say here first is the deployment pipeline is your foundation, right? And, and they talk about continuous delivery, which I think we've talked about in the past. I was sort of joking before the show that I've forgotten everything that we've ever done on the show, right? Um, I know that we've talked about some of this stuff. I don't remember how in depth on some of it. But the key parts of continuous delivery is it reduces the risk associated with deploying and releasing changes. Right. If you have this thing constantly doing it, then then it's always happening. And it's not something that, you know, everybody says, hey, it's the weekend. It's Friday. Let's all do the build.

Right. Like that was the old way of doing things. And oftentimes I know we've talked about this. That would go into the weekend. Right that's a, that's a common theme throughout the book. So they're, they're wanting you to constantly, and they even make a point of saying, you know, developers integrate their code in to trunk daily. You know, that way,

the idea being that the more often you're committing in a, it makes your merges easier to get in and B, it gets your code out in front of other people sooner so that they can, you know, kind of be testing for you to verify there aren't any problems. And there are tricks of how, you know, that we can get to later on. Like, you know, when your code isn't necessarily ready to be primetime, they have tricks for that too. But it's all about, you know, committing daily. When they say commit daily, they're talking deploying daily too. Everything you commit goes right in.

By committing into the trunk daily, they also make a point of saying that the trunk should always be deployable. Always. By what that means, and I don't know if we get into it directly, this is where some of the diagrams in the book itself will help, is they sort of draw out some nice things where it shows a commit going in, that does a repo, it gets built and deployed out to a development environment somewhere. You don't ever think about it. It happens. If something fails, constant, immediate feedback. Right.

Then there might be some other manual checkpoints that say, OK, once, you know, that's fine. That's that's all being deployed to that point. But at some point you want to go to like a UAT environment where people are going to start testing this stuff, that's going to be a manual button push, right? Because UAT doesn't want to be in the middle of testing out something only to have those changes blown away by something that changed things out from underneath them, right? So that's a manual button push, but the same bits are deployed further up, right? So whatever was built here is going to make its way into UAT. It's just that some of it's going to happen automatically and other things are going to require somebody to say,

okay, yeah, I'm ready for this to go in. So that's where missing the diagrams kind of stinks is they don't necessarily call it out. They just show it in a picture. But with that, that also means that your automated deployments, just like we said, and then that's also going to run these automated tests for you to make sure that the code that you put in there didn't break everything in the entire app. Yeah. So I just want to kind of highlight and kind of reiterate that what we're talking about today is all about the principle of flow. So it's how your workflow and

this, uh, how we workflow works. And this is a prerequisite for those other two principles. So kind of like we talked about a little bit with Docker. Like, Docker's great, but only if you also change how you deploy in order to get that stuff. And you change how your developers work so they can make use of it. So it's one of these things where you kind of need to build up this thing that kind of makes use of other things. And it doesn't really work in isolation. So you can't just set up a CI server and not change anything else about how people work and how people do things and expect to get these benefits. Like you really need to go kind of all in.

If I could make a visual, you can't have the top of the pyramid without the base of it. That's right. Back to this pyramid. Hey, and I do want, I do want to mention here because this is real world stuff here. Like all three of us joined a company at one point that had none of this automation stuff, right?

And builds and deploys and everything were a pain. And something would go wrong at an install somewhere, and it was a pain. And trying to figure out what was going on was also a pain. And so what we're talking about here, all three of us have worked through and done what they're talking about in these practices, building up automated deployment pipelines and all that. And one thing I want to call out, because I know it's not anywhere here in the notes,

is you will get pushback. If you try and do this, right? Like Outlaw, I know you remember, you know, when we talked about, hey, we're going to turn it on so that tests have to pass before things can be merged into the trunk, right? Like Outlaw, I know you remember, you know, when we talked about, hey, we're going to turn it on so that tests have to pass before things can be merged into the trunk, right? And I can tell you right now, 90% of the team was like, no, I don't want to wait on that. And it's because developers and just about anybody in the world, right? They're all resistant to change because you've been doing something away for 10 years. You don't want somebody to come in and move your cheese, right?

So just know that the things that we're talking about here, we've all gone through and implemented and it does make things better. It might seem like it's going to be more painful, but ultimately you have a better product that works better most of the time. And you find things much earlier in the process when you do these things. I should say, you got to build up trust in it. And that's the problem is that initially, there's, you know, people have the trust in the old way that they've been doing things, which even if it was painful, they trust that it would work because they had a process. And when you come along with something shiny new, they don't immediately trust it because it hasn't proven itself.

But once it does, then there's no going back. Right. Yeah, I wanted to say, too, I kind of complained about the first half of the book. But it really makes the strong argument for why this stuff is good and why it's worth pursuing. It was boring to me because I've already kind of bought in on this. So if you're trying to convince people or you're trying to just kind of show them the light, then I think that's, um, the first half of the book is really good for kind of, um, giving you the ammo to kind of have those conversations. Cool. All right. So here's, here's the next section that they went into is these environments on demand. And we talk about this all the time internally with ourselves is you always want to be using environments that are as close to production as possible. And that can't be overstated. Like it's so important. It's why Scaffold was my tip of the week last week. If you're going to be working in a Kubernetes environment, it's really important to be able to try and run all that stuff locally

so that you're mirroring your production environment as close as possible. So that's one. Yeah, they make a point of calling out that like we have this consistent across all developers ever in the world. like there's this, there's this bad pattern or habit where, you know, if we have a working environment, we'll just keep using it forever and ever and ever. Right? You know, like once you have that database server up and running, if you know, assuming you don't have like a repeatable scriptable way to recreate the state, like you'd be like, Oh, no, I've got the data that I need there. I'll just go use that, that one. Yep. And, and even in shared environment, same type thing, right? Like

even if it's not on your local machine, there's some shared, shared database out there. You're like, man, I'm not trying to recreate that thing. So everybody starts sharing development over there and ultimately overriding stored procs from each other. Like it just turns into a mess. So yeah, it's true. It's because the friction to get that thing set up is a problem, which is why this stuff matters. And this is the next point is environments need to be created in an automated fashion. Automated doesn't mean, hey, Mike, go click these buttons

and then Joe come up there and do some configuration behind me. It means have scripts, have configurations that will deploy this thing for you. Their next point is all of these configurations, all the source scripts, all the shells, everything should be stored in source control, just like code, and should require zero intervention from somebody in operations. There should be nobody that has to log into a server to set some sort of flag

to make something happen, right? Like that should all be a part of the automation scripts that are in source control. Yeah. And I think a big change that kind of happened a couple of years ago is services started including like the pipelines and whatnot in the source control for a long time. When I first kind of got started with like Jenkins and stuff like a lifetime ago uh it was like kind of like this whole other system

that you kind of tacked on to your code and the stuff that you did over in jenkins wasn't checked in it was kind of hidden it was separated from the actual code and nowadays uh everyone's kind of leaning towards just putting everything in one spot it doesn't necessarily mean one repository but having everything in kind of one tool so the people who work in QA and have their automated tests are nearby the same as the code and same place that your operations are all defined. So all these things are all kind of next to each other and accessible to everyone across the team. I mean, just even having the like the Jenkins file as something that you can commit into your repository and, and tell Jenkins, that's what it should use. I mean, it's kind of a circular type of reference thing, but I mean, that, that's a huge, you know,

a very awesome change that happened in the last few years. Cause I mean, I remember going back to like in a team city environment where, you know, you had a bunch of logic of what needed to happen, but it wasn't in a, uh, like committable state, you know, because it was all like driven through, you know, UI within team city, you know, and it was like, if you lost your team city server, then guess what? You lost a lot of the logic on how to build your app.

Unless you backed up the server. Right. Yeah. Which, right. Which is not great. So here's one of the case studies. I'm not going to go deep into this thing, but this is one of the reasons why I really enjoy these things. There was this sort of longish type thing where they were talking about this Campbell Pretty story. But basically, they had problems to where they couldn't release stuff quickly, right? They were way behind on everything. They were a bunch of waterfall projects, but more or less what they did is they figured out how to get all of their stuff in source control to allow for automated deployments.

And the time it took to deploy an environment went from eight weeks to one day, right? Because it was a bunch of bespoke knowledge that was, we'll call it tribal knowledge that people had on how to set these systems up and how to tweak these flags and get everything up and running. And that was, you know, anytime somebody needs to spin up another environment, they're like, oh, that's going to take time, which put everybody else behind, right? Because if you're waiting for some team to deploy an environment for you, you can't work on what you need to work on. So eight weeks to one day.

Pretty impressive. Yeah, there's another theme in some portions of the book. I don't remember if it was this specific chapter or what, where they're basically saying you should favor an environment or a situation where like if you have a problem like oh man the server has you know this problem or needs this installed or whatever that instead of like trying to ssh in and fix the problem or whatever your default reaction should be okay just trash it and we'll like rescript it and rebuild it like you you should favor uh recreation rather than repair. Yeah. Like recreate it rather than try to repair it. It's further down. I think Joe put in the notes on that basically immutable environments is what they called them. And, and that makes a lot

of sense. Uh, one other thing that was interesting other than the fact that this thing was eight weeks to a day is they said it also greatly reduced the number of defects in the wild, right? If you have people setting up environments by hand, what are the chances that you forget to flip some sort of flag in some setting on the server, right? Like it could happen. And missing that could cost you hours, if not days of troubleshooting. So this also reduced the number of bugs that made it up. There was another one.

I don't remember if it was this particular company case study, but there was one case study where they were talking about their ops team only got to practice. The dev team got to practice their deployments let's say weekly or whatever pretty frequently.

But the deployments that they were practicing were to developer environments. They really didn't care. They didn't matter. They weren't production. So they were low-risk environments, right? Whereas the ops team, they were only doing their deployments, say, twice a year or four times a year, whatever the schedule was. But the point is it was a much smaller number. And when they got to practice it, it wasn't really practice. It mattered. And it was in a much smaller number. And when they got to practice it,

it wasn't really practice. It mattered. And it was in a production system. And, you know, like everything was on the table. And so by being able to increase this process and bring the ops team into it, then they got to practice their deployments more often, which made for fewer problems

when they would actually go to do the deployments. Yeah, that makes total sense. You get to work those muscles. So here's one of the things that kind of piggybacks on that. One of the problems is a lot of times the first time something's been tested for a production environment is in production. Right. been tested for a production environment is in production, right? And that's one of the big issues is you've developed it, you've deployed it locally, whatever, that's fine. But the first time anybody tests a thing, they're manually pushing it out to production and that's a problem.

And they said also many times test and development environments are not configured at the same time. So if you're not automating that process, then that means that those environments are not going to be the same, right? You might have a different version of the OS. You might have, you know, different, different versions of Java, slightly different versions of Java and solve, whatever, right? Like all of those things become problematic as, as you push forward in, if it's not automated. So we already said developers should be running those things just like a production system as close as possible on their own workstation. And this provides a lot of feedback early and often.

Constant feedback. Constant feedback. This one is funny to me. Joe, I already see you shaking your head on this. I feel attacked. So rather than creating wiki pages on how to set things up, you should be investing your time in the scripts to do it. Now, you were shaking your head.

Yeah, I just love the wiki. I love so many times when someone asks me to do something, if I can share a wiki that shows them how to do it, like it's a big win for me. But I just, I don't like the idea that like if you do something once, like you're the person that has to do it forever. And so I, um,

I don't really know how to check in scripts for a lot of stuff. Like, you know, for example, if you end up backing up a database or upgrading a database or something that involves a couple of steps and you kind of check into a wiki, um, that doesn't necessarily travel very well till if you have to do another type of database

change or you know restoration and backup or whatever it's kind of hard to write like a generic script that will do what you need forever but i also don't like the idea that you know if you did it once you have to be the person that always does that because then after you've worked at a company for seven years uh or more i don't know where seven came from that's weird but if you work somewhere for a while uh then you kind of like end up building up this baggage of stuff that you've done once and now have owned forever after and so as time grows on in the company like all you end up doing is just being the person that people kind of you know asked to do like little one-off tasks because you've done it all before.

And it's just not a good spot to be in. You need to be able to kind of spread that stuff out and get it checked in. So I still like the wiki, but I get it. Outlaw, I saw you slightly nodding your head there. Well, yeah, I mean, the problem. So, okay, one thought that I had as you were saying that, Jay-Z, was, okay, well, you know, don't try to write the perfect script, but just go ahead and put something in. And if somebody's like, yeah, but it doesn't do what I need it to do, then I'm like, yeah, becomes stale and stagnant because you're not being the guy tasked with doing new stuff.

You're the guy that has to maintain old stuff because you have that knowledge. So that's the downside. And you don't want to be that guy, right? Like you want to be able to work on new features or new greenfield projects when they come up. So you need to be able to free up your time so that you can. I mean, I get what he's saying though, because we've joked about it before. If you're the person that writes the ticket, all of a sudden you own the ticket, right? Like, and that's a culture thing. That's something that you've got to sort of train people out of, right. Or, or force people to to to sort of own on their own as you move forward because it can be a problem but yeah i i too am guilty of the wiki page right like here run these five steps and

you're good well why don't i just automate it and save the shell script somewhere the power show yeah checking it's weird though like do you have like a directory of stuff you did once or you know like this is that one thing i worked on for those two days, three years? I think you should. I honestly think that, I mean, I've sometimes created a folder called Playground or Sandbox, you know, where I would put things and name the files in a way that it was complete, right? Like, the problem is that stuff can get stale. Like you said, if you've been in a place for six, seven years, then somebody goes to run that backup script. Okay. That worked on version one,

but it's not going to work on version eight that we're at now. So, you know, I don't know. Maintaining it can be a bit of a problem, but you know, here's where I've struggled with it is that let's say you, um,

because, because let's say you create just like a scripts directory and they're in the root of your repo. Right. And then like any of these type of scripts, you know, backup DB or whatever, um, you know, that you want to do, you could put in there. Right. But now that directory seems to go against everything that we learned from uncle Bob Bob because now that directory feels like a dumping ground for everything, right? Like it's the Win32 of, you know, directory of your system, right? Of your repo. Whereas it feels more appropriate to like put scripts near, you know, like if it's a database, you know, backup database script, like what Joe described, then maybe putting that near the database code makes more sense.

But then it's like for somebody who doesn't know that world, then it's like, oh, well, I got to know all the individual places to go look. And they might think, oh, it doesn't already exist, so let me go recreate it. So I don't know. It's a double-edged sword. Of course, in the scenario that I'm describing, though, I'm thinking more of a mono repo. And so that's why you would have all that like that. If you had smaller repos, then it probably makes more sense to just keep it at the root. Yeah, it's definitely not a cut and dry, simple answer for that one. Today's sponsor is Datadog, a scalable monitoring analytics platform

that unifies metrics, logs, traces, and more. Use Datadog's advanced features to monitor and manage SLO performance in real time. Visualize all of your SLOs in one place, easily search, filter, and sort your SLOs, and share key information with detailed, intuitive dashboards. Plus, Datadog automatically calculates and displays your error budget so you can see your progress at a glance.

You can sign up for a free 14-day trial at datadog.com slash codingblocks. And Datadog will send you a complimentary t-shirt that's super cute. And once again, that was datadog.com slash coding blocks and data dog will send you a complimentary t-shirt that's super cute and once again that was uh datadog.com slash coding blocks all right i gotta say thank you so much for the reviews i really enjoyed doing the song and whatnot and so if you enjoy stuff like that then hey keep it coming because uh it's awesome it's inspiring and my fragile ego needs it i appreciate it how many more more new reviews for another song? Yeah, that's what I was going to ask. Is it a song or does he have to do some sort of stunt?

Like there's got to be something, right? We've already had the song. Now we need the dance. Jay-Z, lord of the dance. Wait, I dance all the time. Y'all don't follow me on TikTok? Oh, man, I deleted TikTok. Yeah, oh yeah i saw everything geez and you should too yeah you absolutely should if you don't man we should find that article and post it on the

thing man it look if you're on tiktok we understand it's fun it get rid of it like just straight up get rid of it if you care at all about privacy security anything at all get it off your devices this is crazy yeah um by the way are we gonna do a book giveaway on this i mean i guess i guess now we have to yeah we should we totally should so i like that idea yeah if uh you know you're listening to this and you would like a chance to get a copy of the DevOps handbook in your choice of Kindle hardback or paperback or whatever these physical copies are or an audio book version. Leave a comment on this particular episode, which will be codingblocks.net slash episode 136. And you will be entered for a chance to win a copy of this. I think that you should have to specify the type of dance you would like to see Jay-Z do in his. Should we get, what was the magic number for reviews? We haven't heard yet.

I think it was 700. Hey, we're already at a thousand. Okay, seven. Oh, man. I suck at this. Math. We got to thousand. Okay, seven. Oh, man. Seven. I suck at this. Math. We got to round down, so seven. No bank head bounce.

That's too easy. We need something with some movement. Well, we could see some, what was the Harlem Shake? You know, maybe some of that. That would be interesting. Oh, yeah. Okay. Okay.

Well, with that, we head into my favorite portion of the show. Survey says. All right. So, back a few episodes ago, we asked, Do you always include new or updated unit tests with your pull requests? Which this fits very well with our topic tonight. Okay. And your choices were, yes, always, of course, I'm not a psychopath. Or, not with every pull request, but more often than not. I mean, what kind of psychopath has the time to do them for every pull request? Or,

no, I rarely include any tests with my pull requests. My friends think I'm crazy. All right. So, uh, we will, we will let Mr. Jay-Z sing us in song, which one, uh, he or serenade us in song with, with his choice choice which one do you think it is okay um not with every pull request but more often not that was by the way that was for dave that was sung to the tune of bohemian rhapsody um so i'm sure you right i didn't have to i don't have to tell you all that of course everyone knows that song. Especially your version of it.

No question. Yeah. I changed the lyrics, but yeah. I mean, it was the same. Yeah, because then – where are you going with the rest of that then? Like a – 51%. I want to hear – That's it.

That's all I got. I don't remember the rest of the words. What was the percent? 51%. Okay. All right. So these are our listeners that I am saying here, not me, right? I'm going to say no.

I rarely include any test with my pull request. My friends think I'm crazy. And I'll go with 35%. Wow, Alan. No faith. Yeah, I mean, not me. All right, so. Well, I'm 100%.

Of course. Of course. Yeah. So, well, I was saying no faith in our listeners from Alan. Right, right. So, Alan's going, you know, he has no faith in our listeners from Alan. Right, right. So Alan's going, you know, he has no faith in our listeners. He's saying that, no, I rarely include any tests with my pull request. You said 35%?

Yeah, 35%. Yeah. 35%. And Joe is going with not with every pull request, but more often than not at 51%, correct? Those people are lying. Just saying. Okay. Well, the answer is not with every pull request, but more often than not, however you overshot the percentage as 38%. But, you know, at least you gave our listeners more credit than I did.

Wait, so I was closer, I guarantee you, on the actual percent and the answer, right? No, I rarely include any test with my pull request was last. Wow. At 24%. Oh, I like it. Very nice. Joe, do you care to take a guess at which one was in second place? Because math is hard. Very nice. Joe, do you care to take a guess at which one was in second place?

Because math is hard. Do I have to sing it? I couldn't resist. All right. So, hey, you know what? Before, I could give you a survey. or I could entertain you with a joke. We need a joke. Which one do you want?

Everybody thinks we're funny, I thought, before we saw that last review. I thought they were talking about how we look or sound, like not what we say. Okay. Maybe I misinterpreted it. Okay, so Zach hit us up with this one from a joke that he found on Twitter. So you're on some website. The computer says, hey, choose a password. And you type in hi-hat. And the computer says password cannot contain symbols.

Which Zach was that, by the way? Ingbritston? Why would you do this to me? You did this to me on purpose, Joe. Showed us the Vim awesomeness. Yes. Yeah, you're on that. But I can't say the name.

Yeah, sorry. Was I close? You said Zach very well. Yeah, you did. Zach was great. I meant on say the name. Was I close? You said Zach very well. Yeah, you did. Zach was great. I'm the last name. Inge Britson? Is that how you'd say that?

All right. I don't know. I'm probably just making it worse. I apologize, Zach. Zach, every time I change value in quotes, I think about you. That's awkward. Okay. i think about you uh that's awkward okay uh well today's today's survey is oh no no oh man an old survey i don't have a survey oh i messed up okay well dang i don't have a survey for you uh i

apologize oh geez um i don't have anything on the fly either. Other than like, do you like Jay-Z singing? Yes or no? I can tell you. What about this one? Making this up on the fly. When you do listen to music, do you do it on YouTube or Spotify or Amazon or Apple? Well, I like it.

What else are we missing? Are you CDs? YouTube, Spotify. Vinyl. Amazon, Apple Music. iHeart Radio. Really, I'm trying to figure out where all I should be publishing this song to. Okay.

So when you listen to music, which service do you use? iHeartRadio? The radio. The radio. The radio. The FM radio. I forgot about the radio, the radio and the FM radio. I forgot about the radio. So Amazon, Amazon music, Apple music, Spotify, YouTube, Pandora,

I heart radio, the music, the radio. What about, isn't there like a Google music? Yeah. Google play or yeah. Google play music. Yeah. Well, Google play music. We'll leave it at those choices and see like oops i did it again oh sorry um hey where's that youtube video hey 700 reviews hey hey see i don't i don't hold our uh my singing hostage based on reviews, unlike some, right? Right.

I'll sing and dance for free. We freely give those away. Okay. Well, before we go on, how about one last joke? Yes. Yes. Okay. Always yes.

So, from our favorite Mike, which sadly isn't me, like Mike RG shares this with us. What do you call a hen who counts her eggs? Don't encounter eggs a mathema chicken oh that's so good i have a new name in all right all right The chicken. Jay-Z, Destroyer Pods, and the Mathemat Chicken. Yeah. That's awesome.

All right. So the next one that they talk about with this whole developers running production type environments, they should be copying virtualized environments, right? Like if you're working in something like AWS, maybe you're making a copy of the AMIs. If you've got VMs that you deploy with VMware or something like that, maybe you should copy those, right? So that's a big one. If you have scripts,

these are the things that you would likely be sticking in source control, by the way. Building environments on bare metal, the scripts to do that. Infrastructure is code, whether you're using Puppet, Chef, Ansible, Salt, they called out a bunch of them, right? Like all of these things should be put in source control. Automated OS configuration, creating environments from virtual images or

containers and creating new environments in public clouds. So, you know, all of these things are things that we probably didn't typically, you know, five plus years ago think about putting in source control. But your code runs in an environment. Your code is a fixed set of things. Why shouldn't your environment be a fixed set of things that is also committed to source control? You know, one thing that stinks about scripts, just want to mention quickly, it's like if you've got people with like multiple different operating systems, for example, like, you know,

people in Windows are doing PowerShell, people on OSX are doing shell scripts or Ubuntu or whatever. Then like if you're just writing a one-off script and you write it in Windows and you check it in, it's kind of gross that you don't support this other stuff

because like these scripts tend to kind of build on each other and kind of get bigger and bigger. And so it kind of also stinks to try and support operating systems that you don't support this other stuff because like these scripts tend to kind of build on each other and kind of get bigger and bigger and so it kind of also stinks to try and support operating systems that you can't really test if you're the one writing the script so i just wanted to kind of throw that out there that is true but i think we've also i don't know if we've done this as a tip of the week or not this actually makes a super strong case to write your scripts in powershell because they have gone cross platform, right? So you can have scripts that will run on Windows, Mac, Linux. I don't know, Outlaw, you've been doing this stuff

probably longer in the Linux-y environments. Is there anything that outside of something like SigWin that is sort of cross plat? I mean, you can run Bash on everything. You have Git Bash environments. You have SIGWIN for years. SIGWIN has been an option. And now with the Windows subsystem for Linux capabilities, it just really depends on what you're trying to do. And where the problem comes in is

the more you need to do anything that's specific to an operating system, that's where you're going to run into problems. But I've had you know, I've been in environments where we had bash scripts that everybody used because we were you know, if you were on Windows, then you just used SigWin and bash was our

common you know, lowest level, you know, yeah, shell uh common denominator across them so cool and maybe that's why i like bash so i've always had like a little affair for bash i don't know maybe yeah um i did want to comment though that that like i wonder okay so this book was written in you know 2016 we said right so it it's four years old, but, you know, so that's not so old. But at the same time, it's like, well, as fast as things change, that's like a lifetime in technology, right? You know, I mean, there have been like 18 releases of React and Angular in that time.

Complete rewrites. So I wonder, though, like how much of a story, like, I know that like, for example, Puppet and Chef are still a thing, but you know, now it feels like, and maybe this is just me in my world that I'm living in, but it feels like Docker and Kubernetes are definitely more of a thing, like when it comes to infrastructure as code. And so I question like, well, how much of a place do Puppet and Chef have and in the current world? And maybe my ignorance to them is like, wait, what problem are they solving that I wouldn't be better off with using a Docker or a Kubernetes? netties, you know? I think I could answer that a little bit. I think the difference mainly boils down to whether or not companies have bought into the cloud versus they're doing things on-prem, right? Like if you're doing things in a data center that you own, then Chef, Ansible,

that kind of stuff makes a whole lot of sense there because you're typically working with virtual live servers, right? And so those, if you're doing that now, those people probably going to the cloud are also sort of replicating that stuff because it's an easier lift and shift up to the cloud. So they're still working with like EC2 instances or something like that, right? Whereas I would think anybody that's getting into the cloud newer, that's not having to support a bunch of legacy stuff. Yeah, absolutely. They should be looking at the Kubernetes way because you can, you can take better advantage of the hardware that you have available to you and all that kind of stuff. But I think it's really that divide of the data center versus people that have bought into, um,

the, the ecosystem that Kubernetes provides, right? Because it's not just stupid simple to get Kubernetes clusters up and running in your own data center, right? You're going to have to install special, whether you're using something like ESXi from VMware or something like that, getting the actual nodes set up to where you can use them for Kubernetes is not just, you know, a push button type thing. So that, that would be my thinking is that's where the divide is. I want to mention too. Yeah. I mean, I don't want to sound, uh, you know, like we're like all in on the Kubernetes Koolool-aid and there's nothing else either because like uh i don't know if you saw this in slack but of course uh guess who shared a link oh if you

guess mike rg uh you get to yell bingo um but uh the k8s.af did you guys see that yeah that link with all the failure stories of kubernetes and it's just like, oh, geez, there are some horrible stories in there. So it's not without its problems is my point. Right. Hey, I wanted to mention too. So Kubernetes, I think of it as being very application-centric. That's a great way to deploy applications. If you're like a large business, though, then you might have things like SharePoint farms or Exchange servers or NAS servers or other things that you need just for kind of daily operations for people in the office to be able to kind of connect and network and share files and whatever. And I think Puppet and Chef and stuff like that still has a really big place there because it allows you to kind of maintain security patches and do upgrades and stuff like

that. So I think that still fits in really, really nicely with those tools. So it's infrastructure as code for your infrastructure servers. Yeah, there you go. Can't win. We like circular references. All right, cool. So those things that we mentioned that are getting thrown into source control, that allows you to spin up these environments and these systems quickly, right? So that's awesome. And this also means that your operations folks aren't having to battle configuration issues, right? If that stuff is committed, then they're just making sure the systems stay alive. And it's a big win for developers, right? If we get those feedback cycles quickly after we

push something out and it breaks in a build system that's trying to do deployments, we get that feedback quick and we can go in and fix those problems. If we didn't get that, like say that in the processes, you know, we talked about weekly builds, even in a nightly build scenario, right? If you deploy something out at eight o'clock in the morning and that thing doesn't get built and deployed until that night, the next day you're looking at something that you've already moved on from, right? So your mental context has switched already. And so that even that 24-hour period could be problematic, right? So if you can get it 30 minutes after you hit commit, that's amazing because you're still sort of in the mindset.

You know, I worked somewhere that had like a long build process. And at the end of it, at night, it would generate like a nightly build that the QA team would come in and test first thing in the morning. What stunk, though, is that it was basically kind of single-threaded. It would kind of do the builds one at a time and so sometimes especially as uh the deadlines came up like you might check something and say three o'clock and you go check where you were in like the build line the build queue and you could be you know like 14th in the list with

long build times it's like oh my build's not going to run through if everything runs well and there's no problems my build isn't even going to start till like 10 o'clock at night and so you might you know go home for the day forget to check on it and then come in the next day and the nightly build broke because your build broke because of someone's change before you or something did something that broke the automation and you didn't know it it didn't get integrated till 10 11 o'clock at night and then the the next day, QA can't do the work because the nightly build was broken. So the only thing you can do is test with the day before's code, which is useless.

Those are big problems. I was in a similar environment, I think we've talked about before, where we had to do the builds at night just because it took so long. And yeah, I mean, that feedback loop was awful. And that project originally event, not originally that project eventually got to the point where like, you remember like JSP Java server pages and like how you, there was depending on your app server,

you could have those pages not compiled until you actually hit them or you can pre compile them all ahead of time. Right. But then that meant you know like having the capability to like go through and compile all this so it could be problematic and it eventually evolved to the point to where uh every developer was given a server to sit at their desktop like Like you had your normal, you know, developer machine or laptop, but then you were also given a server to sit at your desk that you could run all your stuff on so that you could just have

your portion of the app on it so that you could compile just your portion independently to like get some feedback on your part. Because similar to what Joe was describing, this nightly process was just so problematic and it takes so long. It was just like by the time you would know that there was a problem, you've already moved on to something else and you've forgotten. Now you have to context switch back to like, wait, what was I doing there? Why was I doing it? Why was I doing it?

What was this even for? So yeah, those worlds, I wouldn't want to go back to that. That was awful. That's brutal. Well, you know what's funny? With our build process, what we started doing is saying, okay, well, the build is too slow. It's getting behind. Let's go ahead and batch up and say, we'll grab five uh change lists combine them and then do a build

and then guess what the build breaks the next day you come in and now they're like okay well one of you five people broke the build and like of course all five people are like oh that's definitely not my stuff so now you have five teams chasing their tails instead of one yeah that's that's rough so there there was a quote that I love that I pulled out of this thing. It says, when developers put all their application source files and configurations and version control,

it becomes the single repository of truth that contains the precise intended state of the system. That's beautiful. The intended state of the system, right? That's really what you're shooting for. And that's why we love Docker, right? You know what the state of that particular insides of that thing are. So all you got to do is deploy your app to it. Love that. ahead and skip ahead to that part but um sure i guess i kind of will so because they talk about like committing in your repo not just your source in any like scripts or whatever that you need to but they talk they take it to another step where like hey also commit in the same repo all

documentation for it which i'm like okay if we're talking about like maybe release notes change logs or readme file that i can get on board with. I've never seen like other documentation like, hey, here's a requirements document that I've never seen committed in. So I was kind of curious about that one. But where it really goes, you know, exponential is where they talk about committing the, like all of your third party dependencies, even things like the compiler would be committed in. And I went, whoa, that is... I can see as a configuration maybe, like, hey, here's a script, like a Docker file, this is what we we use to do the build and it relies on a specific version of this particular package and whatever.

But the actual compiler? Yeah, they definitely go pretty far with this because they even say check in the actual tools, right? And by tools, they meant things like all your AMI images or your VMware images or whatever. So yeah, I mean, committing binaries for that kind of stuff feels wrong, but I get why they're saying it, right? Like if you check in the compiler that you're going to use to actually compile the code,

chances are it's going to work exactly the same every time. You're not worried about whether or not he had compiler 1.1 and the latest one that somebody was using was 1.3, right? So I get it, but it does feel heavy handed. They went all in. Yeah. And, and here was a really cool one too, that like I hadn't, um, I've never seen done. I thought it was awesome when, when I read it though, but they were talking about like, even like networking and firewall rules, for example,

those should be committed in the project. And I'm like, yeah, that sounds cool. it also sounds like it's super specific to you know what type of development you're doing maybe you know because not every project is going to necessarily care but you know about that kind of thing maybe but that's a good point but uh i know i spent a lot of time chasing down environmental problems so anything i can do to kind of iron those, that would be great. Yeah. Yeah.

Well, I mean, think about like how many times have you been in a project where your network administrator was like had anything to do with your code repository, right? I mean, it's zero for me, right? But that's what we're talking about that would be committed here is like, you know, that person would be committing like, hey hey here's the firewall rules that need to go along with this particular version to support this particular version and not only is he committing it but he has something that's going to actually use that to apply it into you know once it gets into a production environment yep all right jay-z you want to pick up here, sure. So one thing they kind of hit on several times, and they kind of sum up here in one section, is making infrastructure easier to rebuild than to repair.

So if you ever, you know, back in the old days, kind of contrast it as like you would have this kind of one magical server and things would go wrong, and you would SSH there, install the certificates or fix whatever, and then you would kind of keep maintaining this one server, and it was like, you know, super important. so disaster recovery and all this stuff was really important too and backups and all sorts of stuff and the world's kind of gone on a little bit differently on that especially with like devops type stuff and so now we've got these notions of kind of treating

servers like cattle instead of pets and also the notion of kind of fix forward. So something bad breaks, then rather than rolling back, the idea is to kind of try to make a fix that will fix it so that you can just release the next thing and take care of that. Hey, hold on. On the cattle versus pets, so this one was actually a little bit disturbing. Yeah, I don't like that. But you put it in here, so we have to call it out because people are going to be like, what are you talking about?

So they said with pets, if your pet gets sick, you nurture it back to health, right? Cattle get sick, they shoot it in the head, they move on to the next one, right? So it's super insensitive, but it's basically the way that it works. And that's kind of what they're saying. Hey, if you have something that's a problem, kill it and re redeploy it. Right. So, um,

you know, for the context, there's, it's, so it's in the notes. Yeah. You know, I think my name in Slack right now is a Jay-Z destroyer of pods.

Cause man, I, I, I, I, I kill some pods, the leader of pods. Okay.

Well, I delete them too. Yeah. Well, and that's the point though too about this whole section this is this is the section i was referring to earlier was that you know if you can just execute a single command to delete the pod and then another command to recreate it then you don't care right like it's easy you'll do it you'll do it every time it's ever needed like you won't even think to like oh

let me go ssh into it but if you had to go through the trouble of like, hey, let me take an estimate as to like how much of a server I need to buy. Now let me go through the process of getting a quote for that. Okay, I got the server. Now let me schedule some time with an admin to go and configure it.

Hey, here's every package and every version of every package I need installed. Please don't just run updates without letting me know. Right. Like you're going to be really hesitant to touch that thing ever. And that's why once that thing got built and it had proven itself to work, why as developers, we were like,

Oh, Hey, no, we already got a database server. I'll just, I'll just use that one. Right. And you know,

just, just to mention, so he said he deletes pods and destroys pods. just, I'll just use that one. Right. And you know, just, just to mention, so he said he deletes pods and destroys pods. Again, that's back into the Kubernetes Docker world. Right. But those things are also available in other ways. Right. So if you're talking about, I don't know, Azure, you could use arm templates to basically tell how to deploy an entire section of VMs or whatever else. Right. Same thing in AWS, you have cloud formation templates and that kind of stuff.

So this isn't something that, yeah, Terraform, this isn't something, what I wanted to call out, this isn't specific to containers, right? Like all of this stuff, you can set up ways to basically be like, kill that, start fresh, run it, right? So it just so happens that Kubernetes makes it really easy for you to do, especially in the development world. Go ahead. And should we pour one out for Swarm? Because we haven't even mentioned Docker Swarm yet. Yeah, sorry, Docker. Man, it's so

sad. How do you go up against the 800-pound gorilla? That's what Google was. And from what I understand, Swarm was better in many respects than what Kubernetes was, but Kubernetes already had such the head start that, I mean, what are you going to do? Yeah, it's crazy.

I wanted to mention, too, that treating infrastructure and servers like this also helps you keep things fluid. An example I like here is sometimes someone will know, sometimes someone will say, oh, I think we need to upgrade to some service, some database service to a newer version to make use of some feature. Then for me, working locally, it's like, all right, let me just try it. So I'll upgrade the, you know, I'll update the version number in the Docker file

and just try it. And it's cheap to do that. It's easy to make changes. It's easy to test because of the automation. So it it's really nice and it kind of helps you keep things like loosely coupled and just make changes quickly um like that's not something like five years ago i would have been able to say yeah let's just try a version of seven and see how it works like no it's you know we've got these unicorn servers it's really crazy we can't really run this stuff easily locally if i do this it's going to trash my local environment and I'm not going to be able to work on bugs tomorrow or some problem that

might come up. So my own development environment was a magical unicorn. But now since I'm running everything some other way, I trash my local environment multiple times a day. It's awesome. Yeah. I mean, I've joked that if I try to use Docker in all kinds of weird ways to keep from installing things, is if you go that route and you force it to where operations people and nobody can even log into those systems, then it creates an environment where people are sort of forced to. The culture is you have to go in and fix it and commit that code to get it fixed in the environment instead of, oh, hey, I went and SSH'd into it and changed some things and now it's working again. Because the problem is you doing that that one time means that nobody else knew about it. If it wasn't committed into the source control, then when you run into that problem again, then somebody has to go figure it out again, right? So it forces a culture to where you commit your changes always,

right? And that's super helpful. You know, on that note, we're going to... Go ahead, on that note. Oh, sorry. On that note, later in the... We'll touch on this later, but this kind of makes cowboy coding harder. And at first, this can be really frustrating because if you're used to SSHing into the server and rebooting some service or something or changing something on the fly, it seems faster and it feels easier to just know what you need to do and just be able to log in and do it. But we're making the tradeoff. We're saying we care less about individual productivity, whether you can make this change in 30 seconds or 30 minutes. We care less about that. We care more about the team productivity because it kind of stinks if you can make this change really fast.

But then tomorrow when you're out, nobody knows what the heck you did or whatever. And so we're saying that it's worse and that we're willing to make that sacrifice and willing to wait a few more minutes in order to be able to make it so anyone can do this and anyone can see what happened there's a there's another thing too that isn't quite um that i question if like if it's just a happy uh accident happy side effect of doing some of this though but like once you get into immutable infrastructure then maybe maybe like maybe this is one of those things where causality not you know it doesn't equal correlation you know type of thing but uh it seems like you definitely are setting yourself up to be in a world where like from a scalability point of view you can horizontally scale easier now i I realized like,

you know, there's probably an asterisk there, like, well, that's not always gonna be the case, but you know, if your, if your app server is immutable and you have it as infrastructure as code, so maybe you could have like, you know, one of those, or maybe you can have 18,000 of them. Like you don't, you don't have to never care at that point. I just wonder, I think there's a relationship there but it's not exactly a hard, concrete relationship. You know what I mean? Does that make sense? Yeah, I agree. It's totally true. If your app isn't built to work in a way that scaling allows, then yeah, it won't happen. But the fact that you're treating

it as something that can just be terminated and restarted means that your ability to do that easier at scale is much better and to Joe's point it sounds like I was going to say because it sounds like

at that point you've like had to solve things like hey where's state going to be where it's going to be kept? Like if this thing's immutable, like I must have state somewhere else because at any point in time I might just blow this server away, right? Right. So you've already had to solve some of these other problems. And so, I mean, I realize that's not the point of the book, but there are some like happy accidents that I think, you know, even if they don't like come for free, you get really close to getting them. Definitely. You know?

And, and what Joe was saying, right? Like what we're saying there is you are trading short term work, like things that you need to do now for longterm payoffs. Right. And I think that's even going back to the test driven development things, right? Like there,

there's so many people that are against it because they're like, well, it's going to take more time to develop, but that's being short because they're like, well, it's going to take more time to develop. But that's being short-sighted because by not taking the approach of including the test in your code, then that means it's harder to change in the future, right? But you can't see that right now. And that's the exact same thing you're saying. Sure, I could log into the server and make that change, but that's a short-term benefit that is ultimately going to cause me pain not too terribly far in the future yeah totally uh you know this is one of my favorite sections too um this is something i've

been trying to take to heart uh and this section is called uh or relates to the definition of done and what they say is that done means running in a production-like environment and actually go on to expand a little bit more and talk about source control and what all that means. But to me, what it really means is like, if you've got a ticket or something, it's not really done until it's running. And how many times have you been working on some sort of large project or something

that's spanning multiple weeks or months or years? And this piece is done, this piece is done, this piece is done, but you can't connect them because some other dependency or something else needs to be finished before you actually get it and then everyone says done but nothing works right things are crashing because of little slight differences and there's just like this huge and really hairy kind of integration period which is just rough and so i i like the idea of not being able to really cross those things out until they are truly published.

I like that. Yeah. Yeah. Said another way, like how many times have you ever marked a ticket in your ticketing system of choice as done without it being deployed before it was, you know, before it was deployed, right? Every day. Yeah. And sometimes it's really tough. But if you find yourself not being able to see stuff running and being published and you're still marking it as done because you don't really have a choice, I think you need to kind of take a look at your environment and your people and your processes and try to figure out why it is that you're checking in stuff that can't be published. Like what is wrong about the situation?

What needs to change in order to make it so that you can see this? And, uh, that's a really tough question. And it usually has a lot of dependencies and changes that you need from other people in order to kind of make that work. But I think that's a good thought exercise. Yep.

Yeah. I mean, you had a question there about like experiences where done quote done code has bit you but i think that's like every production bug ever that's true that's true right because somebody thought it was done only to find out like uh i guess it's not yeah if you're explaining if you find yourself explaining to your manager why uh you're working on something that you don't have a ticket for it's probably because you're working on something that you don't have a ticket for. It's probably because you're working on something that's technically done. I mean,

I, I know, uh, Alan, a, a, an example that would, uh,

hurt us, you know, near and dear to our heart that would hurt us that involved PayPal that we thought countless times was done only to find out that it wasn't. Oh, that was a brutal couple months. Yeah. was times was done only to find out that it wasn't. That was a brutal couple months. Yeah, it was something like we were trying to, it had to do with like being able to do like holds. Like if you paid by PayPal and you wanted,

you didn't want to charge the account, you just wanted to put a hold on it. But then all the complications that would be related to depending on like when the order shipped, want to put a hold on it, but then all the complications that would be related to, um, depending on like when the order shipped, if the funds were still available, which they might not be if it was a PayPal charge. And then also how to deal with, uh, refunds. If I recall, it was like all kinds of complications that like any other credit card system, for example, they were super easy to deal with. But in PayPal world at the time, hopefully that API has gotten better.

Yeah, it was problematic. But now we get to talk about some fun stuff like enabling fast and reliable automated testing. Now, I know we haven't ever talked about testing before. So let's let's take a now we've talked about it so much. But, I mean, okay, so no surprise. I mean, we've said this before, but like automated tests allow you to move faster. They give you more confidence as you're writing your code or making your changes. Uncle Bob had a whole section that we covered related to that. And, you know, it shortens the feedback cycle because, you know, to even not only to just catch problems,

but to see like, hey, is what I've written, does it work yet? And, you know, that's not even trying to take into consideration if you go into like a test-driven development type of phase. But they go on, this is another one of those areas where it goes on steroids, like the testing process, which I love, but I have not had the fortune of working in an environment where this was done, where they talk about'm, I don't remember if it was this specific section or, um, you know, one of the other chapters later on, but they talk about the, we all, we've all talked about like, uh, unit tests and integration tests, right? But their idea, like they define continuous integration and continuous delivery. They specifically define continuous integration as how do they word it here? It's, it's the continuous integration mandates running on

production like environments and passing acceptance and integration tests. So they're talking about like automating everything. So, you know, the building of some in production, like environment, running all your tests on that and, you know, um, and then, and then, you know, providing feedback. I am curious because when I was reading this and I couldn't remember, this is also where I was joking about, I can't remember anything we've ever talked about on the podcast. Right. So unit tests, I know, right? Unit tests, you're testing class methods, whatever. Integration tests, you're testing external services and stuff. These user acceptance tests, the closest I could come to in my mind of going back to this were testing APIs. I mean, do you guys know exactly what they were talking about, the user acceptance test?

It reminds me of that user group meeting we went to where they talked about the business being able to define a set of tests or whatever and run it like that. I don't know. I just couldn't come up with anything off the top of my head. So, uh, coming from like a not formal definition and also a, uh, like an old, uh, you know, kind of experience with it, like where I've always thought of is like user acceptance testing was like, you know, you could be given directions on like how you want to, to, to look or function, but that doesn't necessarily mean that like, um, maybe your hover overstates were, were like what the user like actually wanted to see, you know, like the, the business owner, you know, what they want to see. So, so like that was

historically like what user acceptance testing meant for me was like, somebody would actually like poke around and click on it, but how you would automate that. Yeah. Like how you would automate that. Like, I don't know. It was more, but it was more like visually like, Hey, is this, does this look the way I want it to look? Does it flow the way I want it to flow? Does it interact the way I want it to interact with, with a user? Yeah, that was kind of my thing.

I was having the same type thing. And Joe, you're about to say something as well. What were your thoughts on it? Pretty much what Al said is, you know, I think about user acceptance has been that, you know, essentially a human kind of looks at it and says like, yeah, this looks good. And that automating that is tough. But, you know, I kind of think of it a little bit like integration tests. Like the user should be able to still log into the site.

They should be able to check out. They should be able to log out. So you can kind of define these things. And if you can kind of codify that and make it automatable, then awesome. But yeah, it's weird to me to think about automating user acceptance tests. I still think about it as being like a human saying, yeah, this is good. Okay. Yeah. Because that's sort of where I was going is this feels like it falls way more closely into the integration test. Meaning that like if it is a user can log into the system, if you're talking about a website and you're probably gonna run something like

Selenium on top of it, which is very close to the integration test type side, right? Very brittle, very hard to, to automate and keep running type tests. I don't know. Arlene is probably hating our casual definitions here. Like, like an example might be, you know, Arlene is probably hating our casual definitions here. Sorry, Arlene. Like an example might be, you know, that I'm kind of thinking of like the type of thing I would have experienced is like, let's say that you had like a running balance that was shown like if you had like a ledger, right? And you might want to say like, hey, if your balance is negative, that it would have a certain treatment. So maybe it's in a red color font, right?

And, you know, from a unit test point of view, like you could easily test like, hey, is it less than zero? Okay, cool. And then from a integration point of view, like, hey, were you able to get to the ledger? Were you able to see it? Yes, you were. Okay, cool. But, you know, there's some of those visual aspects that it's like, you know, at least,

you know, years ago they were more difficult. Now we have like in react, for example, in angular, like there's whole testing frameworks in the UI, there's Cypress, for example, like you could, I could see where you could have a little bit more control now, or I should say a lot more control, over being able to test specific UI elements. Like, hey, when the balance is less than zero, is this class applied, yes or no? By the way, I love every meme I've ever seen where it's like 100% unit test, no integration test. Like people can't open the doors. Right. Yeah, that's awesome.

I love those. So if you have those, I would love to see them because every one of them is amazing. All right. We've got some fun stats here. I feel like Outlaw is super good at stuff like this. Oh, really? All right. So this was coming from the case

study of Google back in 2016. And they had some, some fun commit, um, stats of like how they were using their, their, uh, like how they evolved. Um, cause they basically talked about if, if this was the same case study that I remember, they talk about how, um, back in like the early two thousands, it was like really problematic for Google to do, uh, deploys. They have like monolithic applications and they talked about like the evolution of how that monolithic monolithic application, uh, was created over time. And, and it was really problematic for the Google web server team to, to do deployments. And it was kind of like, Google web server team to do deployments, and it was kind of like brutal and you didn't want to be that involved with it if you could help it.

But they worked on the system over time so that they could practice more of a DevOps approach and break things apart so that they could. So they got to the point where they were doing 40,000 commits per day. And that mattered because, like I said earlier, the authors make a point of the more often that you're committing, then that means that you're merging in more often. So you're avoiding any kind of merge problems. So 40,000 commits per day, which translates into 50,000 builds per day, which is awesome. And these numbers were from 2016, by the way, because that's when the book was written. So this is four years old. Who knows how many they're doing today? Well, I'm sure if you work at Google, you know, but yeah, so 50,000 builds per day back in 2016. And if it was a weekday, then it was more like

90,000 builds per day that are done. And then they go on to say that the automated test suites, they had 120,000 automated test suites, 75 million test cases. And it really goes to the point that there's another theme in the book here where they really describe that you need to parallelize your tests. And when you hear numbers like 75 million test cases and 90,000 builds per day, you can easily envision why that matters. Because sure, you could like, oh, well, if you're going to like do them serially, you know, just infinitely scale out, you know, because, you know, we'll just do it in another pod or whatever. But it's not infinite. There is a finite number. So, you know, the more you can do these in parallel, then you get faster feedback loops and you're able to hit these big numbers. They also said that half a percent of the R and D team is dedicated to release engineering.

That's amazing. So yeah, it was, it was, uh, just crazy numbers in general though. It's like, um, it, and it really made me think that like, cause know I'm guilty, having worked on several build server environments, the ability to parallelize those tests is... I mean, I've set up parallelized builds so that if you are in a mono repo, that maybe multiple portions of the repo could be built at one time. And that build might include running tests along with it. But in any one of those builds, parallelizing the tests within it, I'm guilty of not having done. Hey, so check this out. You mentioned just sort of in passing that a lot of the problems that they had in the early 2000s. So one of the ones that they called out was their Google web server product, which was basically what was being deployed to the website, right?

They were one of the least productive groups in the company. Like they were having major problems because there were a bunch of teams that wrote their own C++ code. And then when it would come together, it would break and they just, they were having massive problems with it. So they basically started this automation trip, right? Like trying to get these things into a DevOps type pipeline.

And they went from being the least productive group in the company to one of the most productive groups in the company because now when stuff got merged in it automatically either passed or failed or whatever and went through the pipeline right and one of the interesting things that they called out here and we've mentioned in the past is it was because of the confidence so what they said was people that were new to these projects because there was this high chance that they were going to break something, they weren't confident enough to even write the changes that they needed to make to improve the product. And then people that actually knew about the product knew that there were so many problems that they also wouldn't write it. So you had people that didn't have the knowledge that were scared to write it. You had the people that did have the knowledge that wouldn't write it because they knew better.

And so this whole automation thing that they put in place made it to where they were extremely successful because now you had the confidence, right? Putting in these automated tests, putting in the deployments and all that. So really cool story to point out. And then it also really like I don't know if you guys had this, um, takeaway from it when you read it though, but like, I've always held Google in like the highest regard, right? Like, I mean, of all the companies out there, like they're, they're up there, you know, at the top, right. Uh, you know, and, and you think about it, like you, you kind of put them on this pedestal of like,

you know, they're, they come out with these search algorithms, they come out with, um, you know, the, their machine learning algorithms, like they do all these amazing things. And so you can't help, but just like keep them in this highest regard over and over and over. And then you hear a story like this, where almost sounds like how bad things were internally that it kind of humanizes it more and makes it feel like, oh, man. Even Google, a company like a Google suffered from some of these same problems. So there is hope, right? right like it just it just really uh you know it felt it made it feel like it was um you know more relatable you know that they struggle with the same types of problems

you know everything isn't like because you want to just think that like everything's rosy because you only hear about the good things right and it definitely isn't right we know in our day-to-day work that you just keep running into things. You're just like, how is programming this hard, you know, freaking 50 years later? Like, how? And it's just getting harder, right? Because there's more stuff to do. There were a couple more fun facts that I wanted to point out that weren't in like the bulleted section.

So here's one. Google has over 4,000 small teams that work together. So without this automation, impossible. Like straight up, I can tell you with 20 people working together, it's difficult. Not 4,000 small teams, right? So that's one. All of their code, at least at the time of this writing, was in a single shared repository with billions of files. So, you know, we've talked about briefly in the past about the mono repos and whatnot.

They are right. Billions of files. And then the other thing that is really interesting, and I don't even know how this is possible, is they say that 50% of their code changes every month. That is insane. That is a crazy statistic right there. So, yeah, man. That's all really cool stuff. And by the way, Joe, that's why I like the white paper or the side stories that come in with that. Yeah. The case studies. Yeah, the case studies. And so remember all those big numbers we mentioned,

and 0.5% of the team is dedicated to release engineering. And so I think that to me is what kind of highlights. It's like doing DevOps work is really about freeing yourself up to do other work. So it's kind of like a paradox here that we're saying, like if you put the time into automating this stuff and do that work up front, then you're free to make changes. You don't have to have people that are afraid of doing stuff. You can have, you know, scalable workers. You can have people's scalable features.

You're able to do more because more of your time is freed up and you're not RDPing, copying files around manually and making mistakes. Yeah, and hopefully if you've never even been introduced to this, this will open up your mind, right? Like if you've always been somebody that ran the build and then copied and zipped files and moved them over and did all that kind of stuff, right? Like think about how much better things could be if you didn't ever touch any of that, right? You committed your code and it created these artifacts that automatically got pushed out. I mean, it's a better world for developers because then you're not having to worry about that kind of garbage, right? So. All right. So with that, we hope you've enjoyed this dive into the DevOps handbook. We'll be continuing to cover this one for a little bit. It's an interesting book. So

like we said earlier, be sure to comment on this episode for your chance to win a copy of it. But also while you're there commenting on the show notes, you can also check out the rest of the show notes. We'll have links to things that we'd like in the resources we'd like section. Specifically, we'll have a link to this book, the Phoenix Project, the Unicorn Project. I mentioned k8s.af, so that will be a fun, entertaining read for you if you're in a Kubernetes world. And with that, we head into Alan's favorite portion of the show. It's the tip of the week. And I have realized that i've given this tip before

so i'm going to give a different tip um i was going to mention the unicorn project what we talked about earlier and yeah i've done before so um i just found out a new intellij shortcut that is fun uh shift shift if you hit shift twice i thought maybe it was both shifts but that's not the case if you literally hit shift shift then uh it maybe it was both shifts, but that's not the case. If you literally hit shift, shift, then it brings up a little console where you can start typing. It's kind of like doing control T and visual studio code or everywhere else.

You can do stuff like that. But if you know the name of a file and you don't want to browse for it, you can shift, shift and just start typing it. Oh, that's awesome. So navigate to file. Yep. And it's just,

it's easy to remember. So shift, shift. What was that in? Uh, Intelli awesome. So navigate to file. Yep. And it's just, it's easy to remember. So shift, shift. What was that in? IntelliJ. Ah, I like it. Yeah. Well, in keeping with the JetBrains themes here for tips, I have a data grip tip for you,

which you know how we've all been in, uh, we've talked about this before where like you're in SQL server management studio, for example, and, uh, you're like, Hey, what columns were in this table? And you already have a select statement. So, you know, select start from my table and you're like, Hey, what columns are available in my table? And, and you could, you know, select my table and then alt F one, and it would run a query that would show you in the results pane all of the columns that were available to it. And I think we've also mentioned in a DB2 kind of world where you could do describe select star from my table and it would do, hey, show you all the things that are available from whatever your statement is. Well, in data grip, if you, in that select star

from my table example, you select my table and you do alt F1 instead, it brings up a menu. And in that menu, it has, uh, uh, you know, like, you know, find in file option as well as, um, let me pull up the specific example. Cause I don't remember all of them, but one of them was like navigate to database. If I remember right, I'll look up the exact wording. I should have been prepared. But yeah, you can easily navigate in the left-hand explorer view of your data grip environment to see what the

structure of that table is. So if you want to see what the structure of that table is. So if you want to see what the foreign keys are, what the columns are, etc. All right, rock on. I don't use data grip enough. It's really good, but I don't use it enough either. I use the built-in IntelliJ, the pro version but uh it's just not the same there's something nice about just having it all in one spot everything like tuned for the database yep totally agree man i'm not gonna be able to like find it i had a screenshot of it but

apparently like the google hangouts can be i shared it in google hangouts but google hangouts can be frustrating because uh like if the conversation gets too old, it just doesn't even show you the thing. And you're like, well, wait, where did it go? I want to see that. And it's like, nah, we'll show you. Here's the five things that were said yesterday. And then here's things from six months ago. And you're like, wait, what about the conversation in between? And it's like, ah, you probably don't care about that. Yeah. It's irritating. All right. So mine is going to be sort of in the vein of the show here, which is, you know, this automation of things. And if you've ever worked with databases, you know that they can be a bit of a pain to initialize and move through versions.

Right. There are tools out there for it. And so here's a few of them. I've only really looked at a couple of them in depth at any point in time, but it's worth knowing that they exist. So one's called Flyway. We'll have a link to that. There's another one called Roundhouse, which I believe Outlaw, you have some experience with. And then there's another one called liquid base. So if you're looking for something to be able to automate the creation and deployment of databases, which will basically run through your source control versions and get it all the way up to current standards, these are three tools that will get the job done for you. So know that they're available and they are extremely useful.

Yeah, very awesome. For database migrations. So know that they're available and they are extremely useful. Yeah. Very awesome. Migrations. Yeah. Flyway. Know that Flyway and roundhouse are pretty similar in functionality. Roundhouse was purchased. I'm not roundhouse.

Flyway was purchased by Redgate. Yep. Which happened in the past year or two. Yeah, it wasn't that long ago. Yeah, it was last year, I believe. Yeah. So they obviously saw some value. Another thing that's pretty interesting too, and I didn't mention it in here, but if you use Entity Framework, they also have database migrations that are built into the code, right? So if you write your code, then entity framework will actually migrate your database

for you, which it's funny. It's that whole thing that I talked about earlier with database, with, with developers having no trust in changing systems. Like I know so many people, including myself back in the day, it was like, there's no way I would trust my code to, to migrate a database. And now I'd be like, I wouldn't do it any other way. Right? Like I don't want somebody in there running scripts every time a database is updated. So yeah. Yeah. So with that, we hope you enjoyed this,

our beginning journey into the DevOps handbook as we've made our way through some of the first way and how important it is to enable fast and safe changes and how that can make everyone happy from the developers to the management and then profit. I think that's how it works. Everybody gets happy, then profit. I think that's how it works. Everybody gets happy, then profit. With that, if you aren't already subscribed to us, maybe you got a link from a friend or, you know, listening on their device or something like that. You can find us on iTunes, Spotify,

Stitcher, more using your favorite podcast app. Please subscribe to us there. We'd appreciate it. And as Joe mentioned earlier, if you would like to see him dance, head to www.codingblocks.net slash review. And you can find some helpful links there where you can leave a review and increase the chances of us seeing him dance. Totally. And while you're up there, definitely check out our show notes, all our examples, discussions, and more. And send your feedback, questions, and rants uh to slack and by the way i just noticed we've actually uh

passed uh our number of twitter uh followers i'm saying this in the weirdest way we have more subs on youtube now than we have twitter followers really just noticed that yep really yep um just uh just eclipsed uh recently so uh make sure to follow us on Twitter and, uh, catch, catch us up or head it over to YouTube and, uh, sub there, or you can go to the top of the website and see all the social links at the top of the page. That's pretty cool. That grew pretty fast. Yeah. Yeah. I didn't even think about it the other day, but yeah. Go follow us on YouTube. Yeah. It's, it's my chair review. Hey, people have been asking for a butt pad review, so that's coming. Yes. Awesome. Yes. So, before we sign off, though, I do have one thing to say, if you'll allow me for a moment here. Hey, do you like jokes? You're asking us? Yeah, yeah, yeah. Sure, sure. Do you like jokes yeah jokes so what about what about if like when a joke is played on you do you like do you like it then uh yeah yeah

i don't like it as much i prefer it on other people oh well i don't know if you're gonna like this joke uh so maybe do you remember a couple several episodes back where like you were like oh we could just put in uh you know whatever reviews we wanted to and you wouldn't know and you would read it we might not have gotten the 17. Oh, no. Saboteur. Why would you want to hear the sink? Well, okay. Okay.

So here's the thing. Oh, my God. For the past couple weeks, I have felt my God, for the past couple of weeks, I have felt so like, just every time I think about it, my heart would sink and I'm like, I would feel bad and it would only get worse and worse over time. I didn't realize you were going to put the effort into it that you did. It was supposed to just like a little thing, like how I did the, the, you know, never going to give you up or the, the, you know, oops, I made it, I did it again thing. Like it was supposed to be just something, you know, never going to give you up or the, you know, oops, I made it. I did it again thing. Like it was supposed to be just something, you know, off the cuff and, you know, fast,

not a big deal, not a big production that you made it into. You literally made it into. So then I made a music video for you. I know. I know. So immediately when last episode, when you were like, okay, well, I'm not going to do it live, I'm like, what? Like, I had all these jokes that I was prepared to go back to, and then you didn't do it. And I'm like, I don't even know what to do.

I'm so torn. torn and and like at one point i remember like while we were putting the notes together alan was like hey i don't see these names in here he was going to delete him i'm like no no it's there it's there and then like and and then i'm like on the side like alan so i thought alan was in on the joke and then like when you started like oh hey look i, look, I'm going to get to – here's the solo for the song. And I'm like – the guitar solo. And I'm like, what? And so I ping Alan and I'm like, oh, man, have I taken the joke too far? And Alan is like, what joke?

And I'm like, oh, God. Oh, no, no. I wasn't. I knew what you were talking about. But I was like, I mean, dude, he's making a music video. How are you going to stop that? I put in one hour per review. Oh, man.

Hey, by the way, you need to take this bit that you just did right here and put it at the front of the show. No, I purposely left this. This is purposely at the end for a reason yeah no this is on purpose yo i paid 20 bucks to get that guy to sing on it wait i thought you sang wait what we got to cut this out so so needless to say like uh i i, I felt like I was a super jerk. Would you like to hear the fake names? Was it outlaw one and outlaw two?

I was kind of suspicious of those. Cause the most hilarious thing too is like at the start of this episode, when Alan, Alan was like adding additional, uh, uh, reviews, because I'm sure Alan, when you looked at the last episode you just looked at like the last couple and you were like well okay i don't see this one and let me go and add it that that's why i was like no no i got this and i deleted them because because the real end of the list was in the middle so so you're ready you You ready for this? Here are the fake names that I made up. Thank you very much.

I want to say from a creative point of view, I thought I did pretty good. Asaurus Rex. Yeah. Brains Wart. Prophet with a zero and a three. Joe's got talent. One of my personal favorites here runs with scissors. And I wanted some of them to sound like they were,

they were, uh, real. So I also did like, hey, what are some other names? And what are some hard to pronounce names? Because I was like, okay, I would trip up on some of these names. So you'll hear this come now. Only Raul,

Agent Shrapnel, Yael, Developer, with threes and zeros because I wanted to make it kind of geeky sounding. And my other second favorite, Eats Glue.

So wait, so that's like all the reviews. So really what he tells, no one wanted to hear me sing. It was... As it should be. It was about half of the reviews, yeah. So what we got here, one, two, three, five, and then five more.

So, yeah. There were ten fakes. Wow. That's awesome. I'm sorry. I'm sorry. That's awesome. I was going to say, I always wanted to make a music video.

So, you know, bucket list. Okay. Can we still be friends? No, I'll never trust you again. Well, I was putting the show notes together last episode. I'm like,

ah, dang it. We didn't get them. And you guys weren't, you guys weren't online. So you guys didn't see. And I'm like, okay,

I got plenty of time here. Let me see. I can up make up some new ones and i was like i really struggled with it like i i spent way more time trying to come up with these fake names than i would care to admit to like i might have as much time into creating these fake names as you have in making the song it's awesome that's amazing wow all right so i'm sorry i did a bad thing well i don't feel so bad for using driver for the vocals now awesome all right if we got to record some uh some some ads. Yeah, just a single ad. I haven't honestly felt bad for the past couple weeks. I started working on the song as soon as we said it.

I was going to do it no matter what. Dude, I knew when you weren't ready to sing last time, I was going to be like, oh, no. I totally was. What did you say, though, Joe? You were what? Oh, I started recording the song. So I worked on the song immediately after we recorded the episode.

So I was going to release it no matter what. Oh. Okay. Well, that makes me feel a little bit better. My stomach has been turning for weeks now. Like, oh, God. I'm an a**hole. That's awesome.

I'm going to lose a friend. There's somebody... Did somebody go back and do the... That's wrong. Hold on. Oh, I added in a couple of people at the beginning because I didn't see it in the previous episode. They were there. Gavin152 of Mustard Maker Deluxe. Yep. Oh, I didn't see it in the previous episode. They were there. Gavin 152 and mustard maker deluxe.

Yep. Oh, I didn't see him. Yeah, no worries.