Coding Blocks - The DevOps Handbook – Architecting for Low-Risk Releases

Episode Date: July 20, 2020

Our journey into the world of DevOps continues with The DevOps Handbook as Michael doesn't take enough tangents, Joe regrets automating the build, err, wait never regrets (sorry), and ducks really like Allen.

You're listening to Coding Blocks, episode 137. Subscribe to us and leave us a review on iTunes, Spotify, Stitcher, you know, wherever you like to go for your podcast. And visit us at codingblocks.net where you can find show notes, examples, discussion, and a whole lot more. Hey, you're not done. Oh, and feedback questions and rants to outlaw at codingblocks.net. You can follow us on the Twitters at codingblocks or head to www.codingblocks.net and find all our social links there at the top of the page. With that, I am Alan Underwood. I am Joe Zach.

And I am Mathema chicken. See, I tried to hone in my inner Joe Zach there and just mess with the intro just to see how it would go. Embrace the chaos, man. Doesn't it feel good to just mess with stuff? I do like chaos. This episode is sponsored by Datadog, the cloud scale monitoring and analytics platform for end-to-end visibility into modern applications. And our newest sponsor, SecureCodeWarrior. Build your security posture and defend your organization from cybersecurity threats by empowering your developers with the skills and expertise they need to write secure code from the start. So this episode, we're going to continue our dive

into the DevOps handbook. And as we make our way, continue making our way through the discussion of the first way. But before we get into that, let's hear some podcast news. So real quick, we're on what, page two of this chapter? Totally kidding. In fairness, we completely covered one of the chapters of the first way last episode and got into a second chapter. That might be a record for us. That is a record. Definitely. All right.

So I've got iTunes this time. And the first one up we got here is just some dude writing a review. StuPub, Andrew.Diamond, who we know and love, and SipioMarcellus. So thank you all for leaving the reviews on iTunes. Except if we can interject like, interject real quick, Andrew Diamond, Michael doesn't need to sing. Doesn't OJZ. Like, Michael already sang. Have you not listened

to the last several episodes? I think I already sang. Oops. He did. It was Andrew's bunny. It was the image for last episode too. That was Peanut. Oh, yeah. Oh, is that his, really? Yeah. Oh, that's awesome.

Yeah. So not only do I not – I used your money. I did. Yeah. So that's fine. All right. So from Stitcher, we have Greg Bowler. Definitely not Outlaw.

I feel like Jay-Z should have read this one. And Threads on Parade. Wow Watch. That's what I was thinking. Yes. I'm glad I'm not the only one that was thinking Rage Against the Machine when I read that name. That's such a great name, too. Threads on Parade.

That one sounds really good. I loved that one. Some people are so creative. So yes, thank you all who took the time to write those reviews. We read them all and enjoyed them all very much. And so as he tends to do when we were talking about some things, either Google related or even build stuff now, DevOps, I'm going to try and say his name properly some things, either Google-related or even build stuff now, DevOps.

I'm going to try and say his name properly, and I expect him to let me know in Slack. Well, if you want a more deniability, you'll just let me do it. But no, you go right ahead, sir. So it's Merle Suriar. He left us information back on the Google stuff, like with the code reviews. Man, he dropped some great stuff in the episode discussion channel in our Slack community the other day, right after this episode, the previous episode released. And man, it was just golden. So there is a build server out there called Basil.

And Outlaw and Joe. I finally sent you the links tonight. I had totally forgotten to do it. So apologies there. I was going to call it out, but yeah, no, that's fine.

Yeah. Yeah. I told you two weeks ago, Alan gets all these links and I'm like, Oh, Hey man, can you do me a favor? Cause he'd already emailed it to himself so that he could go back and read.

I'm like, Hey man, can you do me a favor? Like I'm not going to have a time right now. Would you mind forwarding a copy of that? So the problem is it was three separate emails and I had could go back and read it. I'm like, hey, man, can you do me a favor? Like, I'm not going to have time right now. Would you mind forwarding a copy of that? So the problem is it was three separate emails, and I had to go back and scrub all the links back together. Because basically what happened is he kept just dropping amazing links

and knowledge in this particular chat. So the first thing that I want to call out is this Bazel build server thing. And more or less, there are two things about it that are just amazing. And their tagline kind of says it. It says build and test software of any size quickly and reliably. So the killer parts about this

that are worth thinking about is if you work in any particular area, like if you're in.NET, you probably use MSBuild or you use ASP.NET Core to do your builds, right? If you are in Java, you might be using Maven or Gradle or something, right? But it's a very specific thing for that. So what Bazel allows you to do is build for any language, right? You can use whatever build tools you need for any language that you're supporting. So if you have a hodgepodge of things between Kotlin, Java, C sharp, um, go whatever, right? Like this thing will work for that environment. The other thing that is really nice about it is it will actually

see if it needs to rebuild something. So like I know Jay-Z, you and I have worked in some Java builds and we're using Maven. And a lot of times it just goes through and rebuilds the entire thing every single time, right? And I've heard that Gradle can be better about that, but Gradle is like a whole new language on its own, right? Like you've really got to learn the ins and outs of it. So at any rate,

this thing will intelligently figure out what has been built, what needs to be built. And I, there was some sort of quote in their documentation that said it is not uncommon at, um, and I don't remember, but it might've been Google, but I can't, I can't remember exactly, but they could have more than a hundred thousand source files and it'd take less than 200 milliseconds to build because it can intelligently determine what it needs to build and what it doesn't. Now, one of the things that he called out to me on this, and I have not deep dive researched this thing yet, but he said, this is essentially the open source version of what

Google does for their internal builds, Right. So super cool stuff. Definitely worth checking out. So the website is basil dot build. It's actually HTTPS colon slash slash B-A-Z-E-L dot build. So a couple of thoughts on that. Like, you know, it makes sense now. Like what was it that we talked about in the last episode? It was like 150,000 builds a day or something like that.

Do you remember the number? Oh yeah. I'm trying to go back and find all those stats yeah i mean it makes sense now that like you consider the mono repo approach that google takes uh from based on you know things that we've learned over the years you know from from their posts and whatnot that as as large as that repo has got to be now you see like, Oh, how, this is how they're building. Now it makes sense. Like how they can get in a mono repo. Like, you know, like we wouldn't even be able to clone that fast, let alone do that many builds, you know, here it was, I found it. It was, it was 120,000 automated test suites, but it was 50,000, uh, builds per day on average, but more, but weekdays were more like 90,000 builds a day.

And that was from a couple of years ago, right? Yeah, I think this book was in 2016. So, yeah, that's dated information. Right. Now, the other thing, though, too, is that you were talking about the languages that it supports. Sadly, our go-to favorite is not one of the native out-of-the-box it looks like because there's native rules that ship with it. And sadly, C Sharp isn't in that list or any of the.NET languages really.

Fork incoming or PR incoming. Well, yeah. I mean, there's apparently ways that you could create your own rules, but the native ones doesn't include it. Real quick, one thing worth calling out, too, is if you are halfway interested in this, if you click on their Getting Started link right there at the top, they actually have this running in a Docker container. So if you are like me, an outlaw, and we don't like to install anything for any reason whatsoever, you can try this thing out just by running a Docker container and seeing how it works out for you. And you know how I love a good joke, and I can't believe you let this one slide because their tagline there at the top of the page is fast, correct? Choose two.

That's awesome. So here's the other thing. And he explicitly was like, look, man, if you share that on the show, please also make sure you share this because he feels like it doesn't get enough love online and for good reason. So there is an I think it was I don't even know if it's a white paper, what you want to call it, but basically there was a guy who wrote something that, uh, here, let me find, this is from, this is from the ACM. So this was a, this was a paper, you know, or a, an article written for an ACM magazine for For the Turing Award that was won back in, I think, 1984, 1984 by Ken Thompson. All right. So now that I've given credit where credit is due, the gist of this thing is you cannot trust software that you didn't completely write yourself.

And so without going too deep into the article, more or less what it is, can you trust your compiler? Like what we talked about last episode was checking your compiler, right? And then we started talking about the nuances of maybe you shouldn't be checking it in source control. Maybe it's just something that goes into your artifact repository, whatever. But instead of, you know, pulling your compiler from some remote source or, you know, basically living off the developers compilers on their machines or whatever, put the compiler that you plan on using in some sort of repo that you can then use to do your builds. And this takes it a step further instead of, you know, hey, it's not that everybody's builds differently, anything can be compromised. So more or less what he's saying is,

if you didn't write it, you basically can't trust it. Now, that's a really hard pill to swallow in today's world, right? With the fact that we're all coding, we're using compilers that we didn't write. We're using compilers. Forget about the compilers. What about the 18,000 NPM packages that you have to pull down? And that's kind of what this is getting to. So it's sort of crazy. This article, they go through this thing where you can inject something into some C code, right?

But then he talks about, okay, well, then I can put out a backdoor that would go through the C code and then rewrite your compiler and put some Trojan horses into your compiler, right? So the thing is, this whole chain of things that can happen here are because you're basically relying on untrustable sources. And are you really, can you guarantee that a code scanning tool is going to find every problem? Some inconspicuous bug that was purposely put there or whatever, right? Like it's a, it's a, it's sort of, it's a really hard thing to do. Right. And, and I don't know that we talked about it last episode, or maybe we did because it was in the same context,

is you probably shouldn't be pulling from public NPM packages if you're building and pushing to your production server. There should probably be some sort of intermediate thing that's either scanning those or caching the ones that you actually want to use that are stored in some local repo that you would then push to production. It's not a small thing. I was going to bring up something you like,

you use some of the J frog products to, to do that for you, right? Like you, you bring in an artifactory to be that, that, uh, cash or proxy that like could proxy out your, your, uh, request to like NPM or Nuget or whatever. And then, you know, through that service, you could have in like a tied into x-ray, for example, to, to do that kind of scanning.

So there, you know, cause J frog has like a whole set of, of tools that where they can do those types of scannings. But to your point, it's scanning. So you have to trust that the scan is correct. And it assumes that they find like a problem that has already been identified so that they have a signature to look for.

But if it's something that hasn't been already discovered that's still sitting out there, then whatever. But that's assuming now that you're going to create some kind of private repository within your organization, and that's where you're going to put your GCC compiler to then pull it from there as a dependency in your build chain, which is cool. I mean, yeah, it's neat. Who scans the scanners? Right. Right. I mean, seriously, when you think about it, it really just gives you a headache. Right.

Because it's this whole notion of trust nobody. Well, you as a developer don't have infinite amounts of time and resources. Right. Like you can't write your own logger. That doesn't make sense. You can't you can't write your own authentication thing, right? Like you're going to go set up your own OAuth server. Like there's so many things that we now interact with that the reality is you got to trust something.

Until your organization keeps growing though, right? Like starting out Amazon, yeah, you know, starting out Amazon. Yeah. You can't, you can't do all that, but you know, now that size of Amazon, yeah, you can start writing some of those things yourself. Right. If you want, if you want to totally take this trust, no one approach and same with like a Google, right? Like at some point, you know, depending on your size, you have to like draw your line in the sand like okay yeah i gotta trust somebody because you know it's just me but okay now there's two of us now there's a hundred of now there's three thousand there's two hundred thousand like at some point the scale tips to where you can start to do that

but it also depends on like you know within your company culture like how near and dear is that right so like to a google or an amazon they're heavy tech companies you could see where they might be doing something like that on their own but to like you know i'm just trying to think like a random company like uh what's that marsk or you know the shipping company right do you know yeah yeah uh i mean you see them all the time i don't know are they a big tech company would they i don't know maybe they, but I kind of suspect that they wouldn't. I kind of suspect that they would be like a company that would just take it for granted. I think I might write more suspect code accidentally than hackers are slipping into my stuff anyway.

There is that too. At any rate, I've seen just code so oh yeah um that you know real quick uh you want to be a project i heard about uh have you heard about linux from scratch no no it's a a free book you can i think you can buy on amazon too and it's a website too it's it that walks through setting up your own Linux distro from scratch and I mean it's from scratch I've looked at it a few times it's very low level so you're going to be

working on your own drivers and all sorts of stuff and it's a whole book that will guide you through this very very low level it just helps you really understand what's going on and you learn a lot about the process or so I've heard so I've been meaning to do it for years but I haven't quite done it yet but I just like the idea of like really putting together your own kind of basic,

uh, Linux is true, man. That reminds me of a rant that I have. If you allow me, we will allow for this. Yes. Why is it that laptop manufacturers feel the need to include a keypad,

a numpad on the keyboard. Why? Like, I feel like Apple is one of the few that just got it right. That has the keyboard, right? That's, that's part of what made their,

their Mac so popular was everybody aligned to the keyboard back in the day. But if you notice, like it's just got, you know, there's no numpad on it and it's perfectly centered and guess what's also centered along with that is their gigantic no trackpad now don't even talk to me about that i'm not saying i'm not saying i'm a fan of the gigantic trackpad but it is kind of humorous but but but the point is it's all centered right like

you know and and so many of them aren't. They like include these numpads on it, which now the whole keyboard has to be left justified to the orientation of the laptop. And you're like, outlaw, what does this have to do about Linux? I'll tell you why it has to do something with Linux because I've been considering like, oh, you know, maybe I'll get a new laptop.

And I've been looking at the System76 laptops. And specifically, there's's the I don't know how this is supposed to be pronounced but I'm going to try like I'm going to call it the Oryx Pro series which you know looks pretty nice but it's got the the numpad on it so everything including the trackpad are left justified which means your left hand has like not a lot of space on, on it. But the right hand has like all kinds of freedom and room to move about. It's ridiculous.

Oh, it bugs me so much. So I do want to point something out. You've made this argument in the past too, right? About the centered track pad. And I was like, is it?

Because I remember you were looking at the msi snelf g65 right gs65 at one point you were like dude the trackpad is not centered and that drives you crazy it is not centered on the max either so if you look at that it is not completely centered either what are you talking about? It might be like millimeters off, but that is – But it's centered, but not to the keyboard, not to the space bar. No, no, no. Okay, okay. Whoa, whoa, whoa, whoa, whoa.

When I was asking about the center, I'm not talking about like space allotment for the keyboard. I mean for the – I'm sorry, for the space bar. I'm talking about centered to the whole device. Oh, I want it centered to the space bar. No, because – okay. So the – okay. But the deal there though that I'm willing to forgive is that like other companies will put various keys there to the right of the space bar. So they might take away some of the size of the space bar in order to accommodate those keys.

And so I'm like, yeah, I get it. Cause I'm probably going to like, you know, it's fine. Like that part I'm willing to forgive. Yeah. So it's funny. I was actually on the opposite end of the spectrum. I wanted to number pad online.

And so when I got my gigabyte, I explicitly wanted that and I found that I'd never use it. So it might as well not be there. It's just a waste, man. And that's, that's my point. That's why I don't like there. It's just a waste, man. And that's my point. That's why I don't like it.

It's like I'm not – for heavy, number-crunching things, I'm not going to use that on a laptop anyways. So I'd probably connect to a full-size keyboard anyways. So why put it there? You're just wasting space. I would rather have everything ergonomically fit, like that MSI that you talked about, like you couldn't type on it without touching the trackpad and moving the mouse and like randomly selecting other things. Now, like after five minutes of typing on this thing, I was like, oh, man, that's sad because I so badly wanted to buy that. It was a nice machine.

But the Dell, that's one of the reasons why I like the Dell laptops is those precisions because they don't include the numpad. Alright, sorry. You started with Linux and that's what blew me off. Linux went to trackpads. Well, Windows 70, System 76 is what immediately, because

I've been strongly considering a System 76 but I'm like, they got the wrong keyboard. I can't do it. Yup. So, sorry. Let's talk about DevOps. Let's do it.

Let's do it. All right. So pick it up where we left off. Uh, now we're talking about continuously building, testing and integrating, uh, our code environments.

And I wanted to point out here that two of the coauthors of the book, like, uh, people will frequently mention Gene Kim because of his involvement with environments and i wanted to point out here that two of the co-authors of the book like uh people frequently mention gene kim because of his involvement with phoenix project and the unicorn project but that two of the co-authors actually literally wrote the book on continuous delivery it's literally called that so it's pretty awesome to see them uh being involved in these chapters and i mean the whole book they're co-authors did you say their names uh no jez hubble hubble why did you do this to me oh that's why you didn't say the name jez hubble and dav david farley farley all right i apologize I'm so bad with last names. So how?

So we're basically talking about continuously building, testing, and integrating. And I think even just a couple of years ago when the book came out, these were still kind of more novel ideas. I think most people now kind of are familiar with it. But basically we're talking about building and testing processes that are running constantly on some other system, like a build server or some sort of online, you know, cloud service somewhere. And,

um, by doing this, it makes sure that we understand and have codified all over dependency. So they're, they're written down, they're checked in the build server, no side of put together. And it's not somebody's special laptop or the computer over there near the

water cooler that has to do the builds i think remember those days oh yes it went that long like you like that was your intro to devops as you you technically had a build server but it was a build server and it was a big deal that you built that build server and when there was a problem with that build server you felt it uh aka team city hey did you install the dependency you needed what version of visual studio have you got what version of ms build is installed like yeah oh i wasn't even thinking about team city but yeah the you know that that's an example sure but but that i think, is way down the road, though, of what I was describing.

Oh, you were talking back in the day. Yeah, like literally there was the one box. You know, so Jenkins, and before I used Hudson briefly, that was the first build server I'd ever heard of.

Was there a build server before Hudson? I have no idea. Yeah, I don't know. I think Hudson kind of came around not too long after Java. I think it actually was a Sun project. But that was the first one I heard of, and before that was basically, I mean, you weren't working with Java. You were doing make.

But as soon as Java came out, man, if you ever compiled Java by hand, it's super, super annoying how you have to do it. You have to specify all the arguments to get all the stuff in the class path. You have to get all your libs. You end up with these insane, crazy, long commands that you have to run. And if you ever took Java in college in the 1990s, then you know what I'm talking about because it was all like notepad and JVAC. Yeah. I mean, that stuff still exists. It's just that it's being automated for

you there. Like if you ever look at some of the output from say an MS build, it's no better than what you're describing for Java. It's just, it's done for you. Oh, check this out. So I looked up what was the original, and I haven't found that. But what I did find is Jenkins was originally Hudson. Yeah. Yeah, I think it forked because of some sort of licensing thing. They had to change the name. I want to say that for a while there, you'd still see references to Hudson, like code and library and stuff. Oh, you still do?

Oh, really? Nice. I've still seen some references in Jenkins in documentation for Jenkins where it's like, okay, in Hudson it was called this or whatever. Wow, that's cool. One last thing here is...

Go ahead. I was saying that's recently yeah that's that's crazy uh nothing ever goes away no uh the final point they had here is that ensures repeatable deployments and configuration management i wanted to point out one other thing that they didn't specifically mention but i think that the whole kind of build server revolution made it easier for us to do more complicated things like you know javascript builds is one example but also just on like local computers like by codifying that stuff and having to like really explicitly list out how to build yourself build your stuff and like repeatable you know algorithmic instructions also benefited local development because it means that you also

had those same instructions for doing local builds on your computer and it wasn't like up to the readme or the wiki or the person in the cubicle next to you to teach you how to build. Okay, well, two things. One, is it really pronounced codify and not codify? Okay. Two,

everything I'm like, why are we talking about fish? But, but, but, uh, oh man, now I forgot the other point. Um, where you were saying, oh, the, the, I've had this thought where like, it's all, we're always building, like not my thought, but we're always building on the shoulders of giants, right? So every little improvement we make as a society, as a group of people, allows us to automate something. And now that is no longer a problem that we worry about, right? So going back to what you were saying with in the 90s, running Javaeties, uh, running Java C and like having to manually cobble together these command lines, right. That's no longer a thing now. Cause now we have,

uh, a variety of different, uh, you know, build tools specific for Java. So like, you know, Maven, for example, or Gradle or whatever. And, and now, you know, add onto that build servers that, you know, so you don't worry about those command line things because that's been automated for you. And now we can put build servers together to automate builds. So that's one less thing that we worry about. And now in this crazy world, we can be like, oh, hey, here's a Docker file that represents the Jenkins, you know, environment that you would need. And here's the Jenkins file to run in that Docker environment to build it and run all the tests and everything. It's like we just keep building and like, you know, it's like imagine if you were to start building an anthill, right?

And you just keep piling on a little bit more dirt and a little bit more dirt. And pretty soon we're going to reach the moon, right? Yeah, we're not far off there. And I did look up the word. Did you too, Joe? Yeah. I couldn't read the pronunciation. Am I wrong? I'm like, whoa. Oh, come on.

It could be both. It could be either codify or codify. It depends on if you're talking about fish there, right? Or Cape Cod, maybe. Yes. So, he was right both ways. Always.

Alright. So the idea is that once your changes actually make it into source control, then the packages and binaries are created He was right both ways. Always. All right. So the idea is that once your changes actually make it into source control, then the packages and binaries are created only once, and those same packages are reused at the end of the pipeline. That's if you got your pipeline done right, which, you know, it's hard. Yeah, it's hard, but it's what you want. We touched on this in the previous episode where the whole point of that is if you build those binaries the one time, those same binaries are used in development, QA, prod, everywhere, right? And I know Outlaw is a huge fan of this. He's probably said it a million times since we started this podcast is eat your own dog food, right?

You want to be testing the same thing everywhere, right? If you can have your development environment as close to production as possible, that's ideal because then you don't have surprises when that thing makes it out into the wild. The reality is, though, it's also a measure of, like, how mature is your development environment, right? Oh, totally. Because, you know, we, we say like, Hey, you only want to build these packages and binaries once, and then, you know, you're going to use them throughout. Right. I mean, when you kind of hear that, you kind of think of like, okay, that means that you're going to like build a package. You're going to push it up to like

an internal NPM be that like, maybe we mentioned artifactory earlier. Maybe that's what you're using, or maybe you're using Azure DevOps as your, your package feed, you know, whatever. There's like, you know, my get my new get or something like that. But, but sometimes, sometimes, you know, managing all those dependencies is a total pain. And so you might just be like, you know what, we'll just rebuild every time, right? We'll rebuild all of those dependencies every time. And that's just a sign of how mature is it, right? So, yeah, I mean, it's a painful reality, but

it goes back to what we were talking about earlier about sometimes it's like, okay, how much can I bite off right now? And you can't beat yourself up about it if it's not this perfect world. So check this out. One thing I want to point out is we've talked about this stuff. And I think in a lot of our conversation, we've been talking about sort of teams, right? Like there's teams of people that are helping manage pipelines and all that. This isn't something that you have to have 20, 30, 40 people or a thousand people in order to make this useful. And I bring this up. So I was listening to an episode of the MS Dev show. There was a guy who basically wrote some 20, 30, 40 people or a thousand people in order to make this useful. And I bring this up. So I was listening to an episode of the MS Dev show.

There was a guy who basically wrote some sort of stock, not a screener application, but he wrote something right. And he did it in his spare time. Well, in the conversation that he had with Carl and Jason, he was talking about the fact that every time a commits put in, it automatically builds, tests, deploys, right? And check this out. Because he's doing this on the side, because this is something that he's not got all day to spend on, setting up that automation for him enables him to be way more aggressive and agile as he moves on, because instead of spending his time trying to build something and test it and make sure it's working, that stuff's happening for him. And so he can focus on the features that he's trying to do. Right. So that's I want to say that when I

listen to that episode, it was just a two man show. But but having these bits in place and learning how to make these things happen can really free you up to work your magic on the features and not focus on the non-functional things that consume a ton of time. by embracing DevOps and automating some of these things and putting these practices into place that, yeah, it's really – you should want to do it because it's freeing up your time to be more productive to do other things. I never said, oh, man, such a waste. I wish I never automated that build. Oh, man, I miss writing my Java C command lines. Yeah, exactly.

Exactly. So what are the results of this? What does it mean for our culture? Because remember, the whole book is basically like a high-level kind of philosophy of what can happen and what should happen and why you should start moving in this direction and what it means for competing and the changes that you have to make as an organization. And in addition to setting up this build server, there's a couple other things you need to do as a human in order to make this thing work and be valuable. And we've got three here. So you need to maintain reliable automated tests

that truly validate deployability. Because it's pretty scary to slop some bits up there, and you don't know if they work, you don't know if they're right. Nobody knows. Nobody knows when stuff broke. It's also a big problem if you don't have good tests. Something's been up there. Has the thing been broken for six months or six minutes?

It dramatically affects the scope of things you have to look at to figure out what went wrong. You need to stop the production pipeline when validation fails. Have you heard about the Andon core and the new me plans and all that stuff? If you haven't, it's just the idea that you stop the whole assembly line as soon as something goes wrong because it's faster to fix it at the time than it is to try and retrofix it after the fact. I think they mentioned, though, in the book that that was a concept that specifically started at Toyota. Yep, yep.

Yeah, the Numi plant was the one. It was like a business journal in 1980 that blew up, that everyone refers to and on cords, Numi plants and Toyota. Because at the time, American companies were basically sending things down the line, and the philosophy was never stop the assembly line. And then at the end, they would have a big QA process that would look at all the cars and then send these over here to have this fixed or that fixed or look at this or look at that. Or there's a note on this one to check out the steering wheel or whatever.

And then people would have to go to those car stores and half reassemble them to go fix the steering wheel that was crooked or to the left too much or whatever and uh like that kind of the revolution was like toyota came in did a couple plants in america and um they got there was kind of like a weird i don't know the full story but there was almost like a weird like kind of like exchange student thing going on with like the american car companies and toyota at the time and so there was like some american people and some uh japanese toyota people working in the same plants the time. And so there was like some American people and some, uh, Japanese Toyota people working in the same plans. And they were just, uh,

there was initially some confusion. It sounded like over like how things were done and how they were different. And then people started seeing the benefits of the way that Toyota was doing things. And, uh, yeah, it was just basically by stopping the assembly line,

which was like horrible, you know, like the last thing you would ever do at the time. And most of the factories to stop and fix one thing on the car and move this thing to the left six inches or whatever fix the problem as soon as it happened had dramatic effects because it had so much less fixing to do at the end yeah and it in uh in the book they also caught out like you're you're making the entire team aware like when you stop that production line, everybody can jump in and, and,

you know, Oh, I see what the problem is or, Oh, I'm aware or, you know, I can help or whatever. Like you're,

you're able to get more hands and, you know, to help out. And so like, if you were able to bring this into your, your, your software, uh,

production, you know, or pipeline in the same kind of ability, you know, like, Hey, your commit has broken, broken, you know, pipeline in the same kind of ability you know like hey your commit has broken broken you know the trunk now now we can't build anything like you know now everybody's aware right as soon as everybody pulls in the latest code they're like oh now i can't build and so

now you get more eyes that are trying to help you you know figure out what's the new what's the problem that got introduced yeah and you can imagine why companies hated this initially. Imagine if the first thing that happened on the car ends up stopping the plant. And so the first car of the first day is stopped at step one of 193, and you've got 95% of the plant that's just sitting there waiting. And so that's a horrifying thought that you've got all these people that aren't working for who knows how long until they fix whatever that first issue was. And that's such a crazy thought.

But the overall effect is like, yeah, sure, maybe that first couple of cars or first couple of things is going to be slower. But people are going to notice when the assembly line stops and what causes it. And they're going to fix those problems at the root. And they're going to stop making those mistakes that stop those lines because it's still terrible to stop the line. It's just less terrible than letting it go through yeah and the end product was a much better product right because it was basically being qa'd throughout the entire thing right that was the whole point of it and it was the actual physical cable this and on cord that anybody could go pull to stop the entire thing and it's easy, too, to think about it from the respects of, like,

you know, continuing along with the car production. Like, you could see where, like, the complexity of what you're producing can also, like, significantly matter, right? Like, you could think about this from the point of view of, like, picture Ford in the early days of the Ford Motor Company, right? And, like, creating Model Ts, right? Like, those cars, by comparison to today's car, point of view of like picture ford in the early days of the ford motor company right and and like creating model t's right like those cars by comparison to today's car you you might be easy you could easily think to yourself like oh well i mean if there's a problem in you know that there's

so few moving parts anyway it's like if there's one little problem it's so easy to get to everything's like wide open there's so much space to to in and whatever. Nowadays on a modern car, you want to like get under the hood of the car. There's literally no room. You got to take the whole front end apart. So, you know, the point is, is like, you know, because there's so much more complexity to those cars now, and, and it is so much more difficult to get it, to get to some of those pieces. I mean, just to change a light bulb on some car or a headlight on some cars you have to take a tire off just because the way the lights wrap around so the point is like you could see where we didn't used to think about that sort of thing in the days

of ford right and it like to joe's point about like how critical that would have been to stop the whole production line because there's an issue with the light bulb, but in, you know, a more modern car, it's far easier to fix that problem before you finished assembling it. And before that tire is on to just be, Oh, you forgot to put the light bulb in or you put the wrong one in. Right.

It's all, it's a lot easier to fix that at that time. And it enabled us to make different kinds of cars, things that would have been unthinkable. Yeah, totally. And it's the same thing with our code, right?

Our code has gotten drastically more complex than it was 20, 30 years ago, right? Like it's, it's not even, you can't even compare the two. I mean, to that end though, really,

we've kind of talked about this, but maybe not to this exact point, but, um, you know, We've kind of talked about this, but maybe not to this exact point. But we can only imagine what it was like for software developers 40, 50 years ago, right? Where it was like, oh, you wanted a double-link list? Guess what, buddy? There's no library. You got to go write it.

You had to know these patterns. You had to know these data structures. Now there's so many things that even we take for granted and we've been doing this for a while now. Imagine like the kids who are coming out of school today, like how they can easily, or maybe they don't even have to go to school for it because there's so many online resources. Imagine the things that they can take for granted because now it's like, oh, well, there's so many different libraries. You know, if you, if you wanted a doubly linked list for your JavaScript, you know, they're they're you know just pull down doubly dot uh you know from npm and you know or what i'm sure there's a doubly but you know what i'm saying

like there's there's there's like so many other utilities and libraries out there that you don't have to it's easy for you to not focus on data structures or some of these algorithms or even forget them if you have uh you know no no no pointing fingers at us if you have ever studied these things because you know there are there's so many utilities and libraries already out there that it's it's far easier for you to use and you don't have to worry about doing that and and so you know now we're just talking about it from like a devops perspective but yeah it's's all the same kind of concepts. I want to throw two things out there real quick. One is Pat Flynn from Smart Passive Income

did an episode a while back about small batches. And he talked about wedding, how he did, have you heard this, Alan? I have, yeah. Yeah, wedding invites with his wife. And so the idea was like, okay, we'll fold all the cards and the next person will put all the cards in and the next person will do all this. so they kind of like batched it out so

they were able to just like do all the folding at one time and then do all the next step and then all the signing and then all the stamping and you get to the end you're like oh crap i was supposed to put uh this sticker in each one of them and so it's like you could end up these situations where you've got a big mess up that's really hard to fix and you don't know it until the very end of the process. Whereas if you had just done things quote unquote slower and done one thing at a time then you would have discovered that organizational

problem with the first invite and you would have fixed it and you would have been good and you would have saved so much time. And actually one last example I wanted to give is back when I went to school back in the Stone Ages Ages, when you get in trouble, sometimes they have you write sentences when you're young and learning to write. So it would be like you'd have to write I will not play Game Boy in class like 100 times or whatever.

Have you ever done the thing where you're like – Did your teacher watch too much The Simpsons that they had to give you the Bart Simpson punishment? Yeah. He's doing it the wrong way. So part of it is like they want to punish you, but other part is they want you to you know develop your writing skills so of course what i would do being a young programmer is like eyes all the way down 50 times and then w's all the way down right and you think you're so smart and then when you look at it when you're like even halfway done and there's like this gigantic wave going on. Like there's no doubt in your mind what the person did.

There's no hiding it. And you're just like, well, I guess I didn't learn that lesson. I do recall those. I'm just trying to imagine like what could Jay-Z have possibly done that he would have to write something on the board. Bringing Nintendo Power to school or, you know, something. Nintendo Power. Yeah, I'm sneaking the turtles into my desk. Did you see, speaking of Nintendo, if we go short, we don't go on enough tangents.

Have you ever noticed that? We never have. Did you guys see that Lego is introducing a Nintendo NES set? No. Oh, God. Really? Okay okay let me see if i can find this uh nintendo that's in the show notes folks yeah oh whoa it's way cooler than i thought yeah it is it is really cool here here's a link to it i'll put this in i'll hear a bunch of dings um uh yeah just continue while you're all right so it's a it's a lego nintendo set oh playing super mario brothers yes yeah yeah and not just the nes isn't that awesome yeah it's even the old school tv with the click the tv that's amazing oh man the tvs have dial for like contrast or whatever yeah oh yeah red green tint it's got an antenna click click

yeah so i'll include a link to this in the uh the show notes but yeah this is awesome like if you look at the back of the tv there's like the connections wow this is amazing you know what you know what this reminds me of, man, we're going to have to go, go back in the past year. I said something to my kids one day about tapes, you know,

I used to listen to tapes or I make a mix tape, right. For, for the girl that I was, you know, currently a swooning over whatever. For your mom. Like,

yep. For your mom. They're like, yeah, exactly. No, this was before their time. Um, but, but they're like yeah exactly no this was before their time um but but they're like tapes i was like oh man my kids not only do they have the glory of

netflix and no commercials right they did never have to fast forward to find the end of a song man like that actually bothered me. I was sort of angry about it for a minute. Like, I feel like I lost part of my youth. Fast forward and rewinding and, oh, is it on side A or side B or whatever? Like, man. Well, not that I ever had to do this, but I've heard that, you know, there was a time where people would, like, listen to the radio

waiting for their favorite song so that they they were at the ready to record it on tape you know just waiting for and they would never get the beginning just right it was always like you know a second or two off yeah totally no i don't know about it per from personal experience but from what i've heard that was just awful yeah because by the time you were listening to the radio, Spotify was already a thing, right? It's so sad with the tapes too because you had to really appreciate it because every time you listened to it, it got worse. Same with Vital.

All right. Yeah, we have completely come off the rails. All right. That was amazing. Guys, stop going off on tangents. What was that? I said stop going off on tangents. What was that? I said stop going off on tangents. All right. Yeah.

All right. So integrate performance testing into the test suite. So the idea is that you should likely build the performance testing environment at the beginning of the project so it can be used throughout. So this is something that is still kind of, I don't know, controversial. But it seems to me like a lot of people want to put off performance testing to the end. And I think you can make a strong argument for performance testing at the end of a cycle rather than the beginning. Because what are you testing in the beginning? But they're making the argument here that it's harder to get that stuff up and running at the beginning.

And a lot of times, if you don't set up your performance testing, get it incorporated into your CI CD pipeline, then you just never do it. I'm so guilty about this one. Yeah, me too. Me too. Well, I've made that argument.

That's why I know it. Well, it's the next part that actually kind of makes you think that you really should do it at the beginning. And this is the whole notion that the only way for you to find out if you're making measurable improvements is if you're actually measuring it. So if you never had any performance tests in the first place that you were logging those results somewhere, how do you know if you're doing anything that will scale or will improve as you do? Or how do you know when you

introduce some code that made it run two times slower, right? Like it's that kind of thing. I think depending on like what you're using as a test suite there though, like even in TeamCity, right? Like you could track, you could take, you could keep statistics on builds and tests and things like that, right?

Well, this is a build performance. Yeah, but these are performance tests, though, right? And so I'm saying you could keep track of tests, stats of tests over time. I think that's slow. Yeah. So, I mean, depending on...

My point is that depending on your environment, you might have the ability to do some of that logging easier than you might think, right? It might already be available to you and you don't realize that it's there. I guess it would depend on whether or not you were testing something that was actually a performance thing, right? Like if we're talking about unit tests, unit tests shouldn't be doing anything that's necessarily a performance type benchmark, right? Like how many threads are running or whatever. So yes, totally.

You could be testing the output of some method or whatever you're trying to test with your unit test. Well, what I mean though by that is like you could envision a world where like you have a TeamCity or a Jenkins or whatever set up and there's a pipeline specifically set up to to run a set of tests that are all about like hey let's spawn 5 000 threads and beat this web api and see what happens right like let's build the code oh the code built did it test it test okay deploy it out to this performance environment and now let's bang

away at it and see what happens yeah true i mean you imagine amazon or google not performance testing as part of their pipeline for mission critical areas like i'm sure whatever like the true core of the search engine for google gets updated they got to run a battery of performance tests out to see if it gets infinitely slower except you know that could bring the whole thing down and have cascading problems. So yeah, 120,000 of them. Yeah. Yeah. So it's really important. Yeah.

That's the, I'm really guilty of sucking on for sure. And I always wish I had it and I never want to do it. But also it's not really fun. So, well, this is another one of those things that it's like, once you take the time to automate it that first time,

like you take that hit the first time. So true, not fun, but if you can then automate it, it, it really comes, it becomes a point though, of like, so going to your point about it being hard to do upfront is it's kind of like, well, you have to know, well, what are you going to, what are you going to test? Like, like I mentioned some web APIs that you're going to like beat up, right? Okay. Do those already exist? No. Well, then how can you do it up front? Right.

You know? So, I mean, this is, and this is also just generally like where test driven development can be hard for so many people, myself included, because it's like, I don't know what the thing is yet. How do I know what tests are right? You know? Yep. Yeah. It's hard. the thing is yet. How do I know what tests are right? You know? Yep. Yeah.

It's hard. Coding is hard. Everything about it is hard. I quit. You really did. I quit. All right. Well, why don't we just move on then since Joe quit and we'll talk about enabling, enable and practice continuous integration.

Yeah. And so this is just kind of bringing it back to the people again. Like, what can you do in order to truly get the most bang out of this? And basically, it's the same thing we said. If you can keep things in small batches, small commits, do Andon Cord-style development where you find problems early and you stop problems early and you stop things from piling on, then you're optimizing for team productivity over individual productivity. So yeah, it stinks to be a developer and you just need to think something

and you know, what's going to work and you don't want to have to write performance tests. You don't want to have to write unit tests and you don't want to have to deal with all this other stuff. The thing is that you might be right. And maybe that is easier for this one little thing. But when you kind of look back on the year or two years or 10 years or however long that code's going to live and you look at all these little things that were just easy and probably fine, then you've got a real problem and you've got this beast that you can't reckon with. Yeah, that's a good point that you kind of like quickly gloss over there. It's kind of like a central theme in some of this, uh, discussion in the book was like favoring team productivity over individual productivity.

And there, there are things that you would think like, Oh, as an individual, like, I don't want to take my time to do that. Like, I'm going to lose my ability to be productive. But it's like, yeah, okay. Maybe for like, you know, a day, a week, whatever. But in the long run, that small amount of effort that you're going to make is going to reap huge benefits for you and the remaining, the rest of the team that are going to more than make up for it. Yeah. And anecdotally, I mean, again, I mentioned it last time we've seen anytime you're trying to introduce stuff like this, you get a lot of pushback. Like it's natural for that to happen.

You'll get pushback for that reason too, right? Like, Hey, we need of pushback. Like it's natural for that to happen. You'll get pushback for that reason too, right? Like, Hey, we need this feature out. You're slowing things down. But if you truly fight for it, you will improve it like massively. You can't even measure it. It's to the point to where once somebody has it, they can't live without it anymore, right? Like, Oh, well, these builds are happening. What do we got to do to get a build out now? Oh, well, somebody's going to have to go do this, this, and this. Okay, can we get the build server back up? Like that's, it will actually change the way people view this stuff if you can get that into an environment.

Yeah, if you're in a small shop that, and you don't have like any kind of automation in there, and you automate the build for your boss, I promise you, he will be forever in your debt. He'll never want to go back or she will never want to go back to a world where, uh, you know, they had to manually type something in or manually, you know, go off and download the code or whatever in order to build it. Yep. Even if it was something as simple as like running a script, right? Like if you can make it a repeatable process that where it's like already done and ready for them, like at their, they're like, Oh, Hey, I want to get the latest. Oh, it's already done. I just download it.

Oh, great. Yep. So the next one up we have here. So we said the small batch optimizes for the team productivity feature branches optimized for individual productivity. Feature branches optimize for individual productivity. Yeah, and so the argument here is basically that feature branches are DevOps

smell, if you will. So the idea that feature branches, if you're not familiar, it's basically the idea that like, hey, we're rolling out customer service 2.0 so we're going to make a branch customer service CS 2.0 and we're all going to work in here for three months. And then when it gets closer to being ready, we'll merge it in, and then we'll deploy it.

And so we'll kind of keep this stuff on its own little island until we're ready to move it in. And the argument that you're really making there that's kind of central to that is that I don't want to mess with anyone else's changes. I don't want anyone else bricking me. I don't want to deal with any merge conflicts right now because I just need to get this stuff done. I'm going to go in a hidey hole. I'm going to focus on my stuff and then I'll come up when I'm ready and then we'll get emerged in. But that's very much a large batch argument. And it's optimizing for the people working on that product and not the organization as a whole. And so that's one of the things where if you're doing that then you're optimizing for individual productivity at the behest of team productivity and uh for an organization that's no bueno

yeah and uh yeah i didn't have to say that we have notes right here so it requires painful integration periods uh which is often invisible work that's really tough to estimate like how long does it take to get things truly working together i don't know or it could just be like how how long do you spend just trying to merge yeah oh yeah that too yeah absolutely yeah how many times do we think something was done and let's just realize that nothing worked end to end you know a lot of times people will joke and like kind of blame me unit tests and the you know like we talked about those memes unit tests for expression you know? A lot of times people will joke and, like, kind of blame the unit tests and the, you know, like we talked about those memes, unit tests versus production tests,

but, like, a lot of it, it's not so much the problem with the tests as it was that people were going off and working in silos and then expecting that things would just work together. Also, this is harder for the pipelines. You know, if someone says, like, hey, we're doing CS 2.0, we need a new environment to spin up that's just going to

have our stuff, and there's going to be, you know, someone need a new environment to spin up that is just going to have our stuff. And there's going to be someone in the background kind of setting up the environment and kind of keeping all this stuff around. But it's ultimately throwaway work because that stuff's going to disappear in a few months when it gets merged in. And same with that environment. It's just all this stuff that you have to set up just for once. But a lot of times the developers don't have to feel that pain because there's some sort of operational team that's handling that. Yeah, like I know Outlaw got hit with these pains in the past. Like you have a feature branch. Oh, well we now need a website. So this thing can be deployed to this website on the server. And we need, we need another database over here. So like it's, it's, it's not just, Hey,

build this branch. It's now, Hey, you have to build this and know of all these other deployment features out here to make this thing run. right? So it's a whole lot of repetitive work just to get one thing set up for this temporary branch. And I think it's fair to point out, though, but when we talk about these feature branches, we're talking about long-lived branches. Not a week. Yeah, yeah, yeah. Not a week. Because you might be thinking, but hey, wait a minute. Yeah, we, yeah. Not a week. Because you might be thinking like, but hey, wait a minute. Yeah, we're talking about branches that hang around for a while, like weeks.

Months. Months. Right. Kind of timeframe before you get merged in. That's why it becomes so painful because there's so much accumulation that's happened in various places. And now you're trying to merge that in. You ever been somewhere with a feature branch and you're working on it for six months or whatever, you know, the newest version of the system and like you're not pulling master because you know,

or a debt, you know, whatever you call it, you're not, you're pulling your main line because you don't want to break anything. You want a stable environment. Oh, but you need to pull in someone's changes to test with something.

Oh, but, oh, but that brought in other stuff that you weren't ready to deal with. And so now, oh, now you gotta, right?

Nope, nope, nope. All that stuff is harder to deal with because you're bringing it all in at once and you've got your mind on this one little thing that you need and now you've got to deal with all this other stuff so rather than bringing it in piecemeal and dealing with it piece by piece you're you know you're paying the compounded interest on it it is fair to point out though because even in like you know that uh clarification on the feature branch, though, and we said, hey, Alan, you specifically caught out, hey, it could be a week.

According to the book, though, you're merging back into the main trunk every day. Daily. Finish what you're working on. And if you're not finished, then you have it protected in such a way that it won't matter. Yeah, I have a hard time with that too. Yeah, I agree with you. That's a tough pill for me as well. I'll tell you though, everything that we've just said in the last hour here are all things that I've argued against at one point in my career so maybe they're right about this too i don't know meh yeah i've argued i remember

arguing against git because i are my argument against switching to git was that git solved a bunch of problems i didn't have i wanted a centralized server i don't care about branches why would like we shouldn't be branching anyway. It's painful. And it's just going to complicate things. Yeah. We had a, I had a friend who, uh, who was the early advocate for get,

but we were a material shop and I'm like, yeah, I don't, I don't get it. Like, who cares, man? Like this is good enough,

right? Whatever. I don't care. Yeah. Yeah. Yeah. Now look, I'm wrong about. Yep. Now look at it. I'm wrong about everything before I'm right about it.

Wow. Sometimes I flip-flop. I go back. Who knows? You're crazy like that. That's my little opinions. We just can't. We just don't know.

I don't measure my coffee. That's right. We live in chaos and enjoy it. Go ahead. I mean, why you got to throw me under the bus about that? It's just so much fun. Why was that necessary? Oh, man.

It's just adding a little. I'm bringing the chaos to your life so you can enjoy it too. Oh, okay. Well, I want to talk about integration complexity instead. Yeah, so we talked about feature branches. We didn't really say that. If you've got one feature branch for customer service 2.0, you probably have other feature branches going on from other teams at the same time.

And so we look at one slice of it. But you think from kind of the mainline perspective, you've got 6, 8, 10, 14, who knows how many feature branches that are out there doing this stuff at one time. That's a lot of stuff that people have to keep in their heads and have to know about. And what that means is stuff is getting dropped. Balls are getting dropped. People are forgetting to tell people about relevant changes, and stuff is just getting lost and messed up all over the place. And the net result

is that the more feature branches that you have out there, the more you have those long-lived branches, the more you got to deal with these pants. And that's why they argue for checking in daily. Which I'm thinking about. When we talk about these feature branches, you might think like, oh, maybe this isn't me. But if you go back to episode 90,

when we talked about various, uh, get workflows, uh, one of the one, one of the strategies that we had talked about was like, imagine if you tried to keep, um, how did it work? Now you, you wanted to have all your, your different versions in parallel, and you wanted to be able to merge in from one to the next. So you make your change in your trunk, and you want to merge it into the various releases in order to correct the problem for every future release. Do you remember that conversation? That was the Git flow, wasn't it? Yeah. Yeah. Yeah. And, and like the hassles that would come with that,

like it would get exponentially harder. Like the first, the first merge was painful. Now you've gotten past that and you want to like basically imagine if you have like versions ABC, right. And, and let's say that a is the oldest branch and C is the newest branch. So you found a bug that goes all the way back to version A. So you make your bug fix there. Okay, that's not so bad. Now you want to merge the changes from A to bring it forward into B. Okay. Well, there's some conflicts there. Some things changed. Oh, this page doesn't work like that anymore, whatever.

So you now come past that one. Okay, fine. That one wasn't so bad. Now you get to version C, right? Now you're carrying all the baggage of A and B as you bring it forward, right? And it just kept getting more and complex as you have more versions that you're trying to maintain simultaneously, right? That's an example of this feature branch kind of problem. You know, those are, in that case, those are long-lived branches where you're trying to

maintain multiple versions concurrently and it can get painful. And they say that the complexity scales exponentially, right? So it's not just three times harder because you have three branches. It's probably freaking 10 times harder. And that's true. And there's likely, you know, your changes, the only thing that was brought in, you know, that might've been your bug that you fixed might not be the only one that was made in either A or B that's then being brought forward into C. And then what happens is like, there's changes you aren't even the author of that you have to figure out how to merge in by the time

you get to C and that's where it ends up becoming complex. And that's why I like with these long lived feature branches, you know, uh, that, that they talk about in the book, you know, Joe Allen and I each have a, you know, code that we've been working on for, say, 90 days. And Alan merges his in first. Then Joe goes through the pain of trying to get his in. And he's dealing with conflicts with his code and Alan's code. And he's making decisions about Alan's code that he doesn't know anything about. And then I come along and I have the same problem.

It gets really complex fast with long-lived branches. And maybe that should really be the terminology for it rather than feature branch because feature branch makes it sound like it's like a brand new thing that you're trying to add and that's why i'm trying to call out that it's not it could be the release the versions that you're trying to maintain you know so maybe it should just be called, like, long-lived branches. Agreed. I like that. Yeah, but there's some long-lived branches that I think are fine. Like multiple concurrent releases, you know, we've talked about that before.

Like if you've got a Windows 95 and a Windows 98 and a Windows 2000, like those are all branches that should live for a long time. Well, I mean, okay. You don't plan on merging back in, so that's why it's called a feature branch because you're going to merge it back into the trunk at some point. Fine. Okay. Because I was going to say, like, it really depends on,, you know, in the case of your Windows 95 and Windows XP long-lived branches, you know, you're not necessarily trying to merge those changes in. You're just cherry-picking from one to the next. Yep.

Well, you know, as much as I love that guidance, I haven't cherry-picked in like six months just because of what I've been working on. I haven't missed it at all. Not one iota. But if you had to go back to merging code, you would appreciate it again. Yes. You absolutely would. So it sounds like somebody only works on brand new greenfield stuff. So maybe I should be jealous.

Yeah. Yeah. Yeah. Making new ones. What we said here, too, is when you have these feature branches, these long-lived feature branches, it makes adding new features, teams, and individuals really difficult. I mean, just imagine coming

into that mess. Yeah, it makes leaving tough too. It's not when you lose somebody and it's like, okay, we're merging this stuff in. Who wrote this? Oh, they left six weeks ago. Okay. I was going to say, if you're the person leaving, no, that's easy. Bye, Felicia. That's true. I just marked to say, if you're the person leaving, now that's easy. Yeah, that's true. Bye, Felicia.

That's true. I just marked all my tickets as done and I just quit. That's even colder. I wasn't even thinking about that. These suckers aren't going to know for three more months. Oh, that's hilarious. She's going to mark all the tickets done in your ticketing system. It's so wrong, Joe.

Why would you do that? Why would you suggest that and even throw that out there on the Internet? Now somebody's going to do it. Send your complaints to Joe Mathemachicken on Slack. That's so wonderful. So next up we have a trunk-based development and basically merging more often means that you're finding these problems sooner in the process and it moves us closer to the single piece flow one big assembly line yeah when you think about these uh you know the opposite the feature branches then

it's a it's a complicated graph of what's moving and going on but when you think about everything going into the trunk or main line as often often as possible, it makes it much more like a true assembly line, where things are just constantly moving down one pipeline. Not to mention you have that many more people like accidentally testing your code for you. That's a great point. That's a very great point. So all the time. Yeah. Constantly building, constantly testing, you know, and using. Today's episode of Coding Blocks is sponsored by Datadog, the unified monitoring platform for full visibility into all of your serverless functions. Troubleshoot performance issues faster by seamlessly navigating between logs, lambda metrics, and distributed request traces all within one unified platform. And Datadog provides real-time screen boards and service mapping

so you can get complete observability into your serverless environment. Start monitoring today with a free 14-day trial and receive a free Datadog t-shirt after creating one dashboard. Visit datadoghq.com slash codingblocks to learn more about how Datadog can help you optimize your serverless environment. Again, that's datadoghq.com slash codingblocks. All right. So please leave us a review. I really appreciate it. I forget if I promised something last time. So, you know, if I did, just leave me a comment in the comment section. And you might also win a book. So, that would be pretty cool. But we've been getting so many great reviews.

So, I feel kind of like a jerk asking for it. But it's just so good. And, like, it's so helpful. I can't not. So, if you would consider it, if you go to cooking box on that slash review, we try to make it, uh, we try to make it easy for you. So we've got links there.

We'll tell you all the places that we'd love to hear from you. And, uh, that would be fantastic. So again, it's cooking blocks.net slash review. Were you supposed to give us like a target number for a dance? I think you were talking about like that,

like Harlem shake or something you know i think my wife said that i wasn't allowed to dance i don't know why i guess i guess i'm too good at it i think i don't know that's not the way i remember it you helen i don't know yeah yeah it's capoeira, okay. That's the only... Okay. We'll take what we can get. I played Eddie in Tekken 2, so I got this. Here's a deep cut for you.

I better get ready, though, because if you do pull off a dance, then next thing I'm going to be hearing Andrew Diamond's going to be like, hey, outlaw, where's your dance? I'm going to be like, oh, God. Right? That's right. Okay. Uh, all right.

So, uh, that will be a little video, uh, fun for you. Um, so with that, it's time for my favorite portion of the show.

Survey says, all right. So a few episodes back, we asked, hey, how likely are you to advocate for working from home in the future? Oh, speaking of, this was a suggestion from Andrew Diamond.

Andrew Diamond? Whoa. Yeah. MVP. Yeah. All right. So, your choices were after this pandemic, every opportunity I get. Or after this pandemic, never. Or work from home. Is that even an option? All right. So let's start with Alan. What answer in percent do you think? And this one's tough and I'll tell you why real quick. I think it's going to be after this pandemic, every opportunity I get, I hope that's what it is. But it's all going to boil down to how people actually did this. So there's the kid factor that I can't even account for. But then there's the other factor, which is if everybody got on meetings or whatever and they didn't turn on cameras and they didn't become personal during that time, then it's going to be a different experience.

But I'm going to go with the first one being optimistic and I'm going to say 38%. 38%. Okay. Mathema chicken. We have a permanent nickname for you now. It's such a good one too. Especially we've seen your math skills so it fits so well right right geez

it's rid of only real numbers sir only real numbers so I'm thinking like I've got some you know co-workers that basically as long as I've known them they've been remote but they're just now kind of going back to offices and stuff. And I'm trying to think whether they like it or not.

There's a lot of people in big cities that just don't have great setups. They're kind of living in smaller places. So finding that room is tough. So, geez, we don't have like a middle of the road, do we? Jeez. Whatever Alan said. Minus one? No, plus one.

I want to win. Wait, wait, wait. But what's your percentage? Whatever Alan said. So you're telling me that you're going to go the exact same answer the exact same percent. Yep.

That kind of takes away the spirit of the game. Chaos. Okay. I'm going to go with Allen and plus one. Then I'll go 39% and you can stick with the original Allen. I like that. Yeah. I got original Allen.

Okay. All right. So Alan goes with, after this pandemic, every opportunity I get at 38 and then 39%. And Jay-Z goes with, after this pandemic, never sticking with 38%.

And the winner is... It's Alan. Sweet! Wow. Except you underestimate how much people enjoy working from home. By a massive amount, it sounds like. 71%. Oh, that's amazing.

To every opportunity I get? Every opportunity I get, they would advocate for working from home. And, and it was interesting too, because like the, uh, uh, not,

is that even an option? Like work from home? Is that even an option? Was, uh, way distant last place at 3%. So at 3% you said, yeah.

Oh, wow. Okay. So the, the people that said never, it's probably the people like Jay-Z said that have cramped spaces, or they've got kids and dogs and everything else, and it's just impossible to control the environment, which I get. Well, kids that are either maybe too old that they can't put in a daycare, but too young that they can't really fend for themselves. Right. So yeah, that could be a problem or,

or like roommates, you know, maybe you live in a small, you know, New York city apartment and you have to share it with roommates too. Like that would be a problem too. Just from like a thing, you know,

could be noise space actions. Yeah. Internet speed connectivity. It could be a problem. Yeah. A lot, a lot of factors Yeah, internet speed, connectivity. It could be a problem, yeah. A lot of factors that some people might not enjoy working from home. There was – what's – Joe's going to know who I'm talking about. Gary something – I can't remember his name.

We've met him at – Gary K. Ray? The Orlando Code Camp. He was one of the speakers. And don't ever accuse him of LARPing. Oh, guy, uh,

guy Roy Roy. Sorry. Yes. Yes. He, he was, I think it was him that tweeted recently that, um,

see, I mess up even on the easy names. Uh, guy. Yeah. He, he tweeted recently where something like, you know,

everybody talks about working from home. It's like, Oh, you know, everybody talks about working from home and it's like, oh, you know, you don't have to wear pants. You can just wear pajamas, right? You don't have to get dressed up. And he was like, when's the last time you wore shoes? I was thinking about that. One of my coworkers is at the office. I was just thinking,

he's been coating in shoes all day. Like, that sucks. I had to put on socks. Can you imagine how hot that would put on socks. That sucks. Can you imagine how hot that would be on your feet? Yeah. I haven't worn shoes in months. Yeah.

Other than Florida, that's, you know. It was such a great point, though. When I read it, I'm like, oh, yeah, man. Like, you're totally right. You're on to something. All right. You're onto something. All right. So for this episode survey, we ask, what is your meeting limit? So your choices are one is too many. There's a reason I work with computers or less than five hours a week, it's okay. I still have plenty of me time.

Or, all right, I can handle 10 hours a week, but ain't nobody happy about it. Or, lastly, bring it on. The trick is to mute when you go to the bathroom. That assumes work from home, by the way. Does it? Nah. Nah.

It definitely assumes you're taking that call from a, one assumes that the meeting is a call and that you're taking that call from a cell phone or some kind of mobile phone, you know, some kind of portable phone. Yeah. I really can't argue against phone. Yeah. Yeah. I really can't argue against that. Yeah. Yeah.

This episode is sponsored by secure code warrior, secure code warriors. Gamification lets you learn how to write secure code from the start and identify bad code already present. So check this out. You want, okay, here, here it is. When I, when I say that, you know, gamify in and learn how to write secure code, what I mean is like secure code warriors, a platform that you can bring in your organization, right? And, uh, you can have your team learn how to write a secure code, but they do it in a way that allows you to like gamify the whole system,

right. And get points for it and see what's going on. And it doesn't matter like what your, uh, you know, area is. I'm sure like whatever language or DevOps tools you're using, they've got truckloads of stuff. So you want to do cloud formation? Great. They got that. You want to do Docker? Cool. Go. They've got that. Like, what do you want to focus on? Java. Okay. What about Java? Do you want to like specifically look at maybe the Spring API, maybe the Enterprise Edition, or, you know, JavaScript? You want to focus on React? They've got the language. They've got the DevOps tools that you're using. You can gamify and build learn good security practices right from the start and that's what i mean and the cool part about this is when we say gamify it's truly making it fun for developers right so it doesn't it's not like a chore like these cbt trainings that

everybody's familiar with right like oh here you got to go take your 30 minute training on how to make this code secure. And you probably were half listening to during this entire time. Right. No, this is competing against your friends to see who can get there. But it's not the mindless, you know, boring type thing that you're doing. You're actually looking at code problems and trying to figure out what it is. And as you get answers right, your score goes up on a gamer board that everybody can see. And as a dev, I love Secure Code Warrior because I'm actually learning things and doing

things that make me a better and safer developer. So for my experience, I went in and I picked four or five languages and frameworks that I enjoyed. So I picked like.NET Core. I picked, I think, Node. Or maybe I did Angular. I did Docker and Kubernetes. I forget what the other one was. But I started doing the Kubernetes one, and I was working with the scenarios. And the deal is you have these challenges where you have to go through projects and find where the vulnerabilities are. And they tell you what the kind of vulnerability is. And I got stuck on one of the first challenges, to tell you the truth, because I was just missing something.

There was something that I just didn't know. I didn't even know that I didn't know it. And so I ended up having to use a few hints for it, which did drop my score, which, you know, hurt a little bit. But, you know, ultimately I learned something out of it. And that's not something I'm going to be forgetting anytime soon. So I love that I'm getting something out of this training. And because of the gamification, it's actually fun. It kind of feeds into that kind of collector feedback cycle that I like. And so I'm looking forward to trying the Docker one next,

and then I'm going to kind of dive into the languages there. And I don't even really know what I'm going to be looking at there when I get to Docker or Angular, for example. Those are areas where I haven't really thought too much about security, and so I'm really interested to see what they've got there. And that's just a small piece of what Secure Code Warrior has to offer. Yeah, so head to discover.securecodewarrior.com slash codingbox to start your next game.

Score 5,000 points and you get a cool t-shirt. That's discover.securecodewarrior.com slash codingblocks. Alright, so speaking about the things that you can do in order to bring about the DevOps revolution, we're going to be talking about automating and enabling low-risk releases.

So like we mentioned, small batch changes are inherently less risky because less has changed since the last time you did a release, which was presumably okay. And like we mentioned earlier, I think a lot of this is kind of recap, but the time to fix the time in order to find the resolution or remediate a problem is strongly correlated with the time to remediate. And I'll give you two three-letter acronyms that are really interesting to look up if you're not familiar with them, MTF and MTR. Sometimes you'll see other acronyms, but it's basically the same thing.

It's mean time to find and mean time to remediate. And there's actually a bunch of really good white papers out there that look at these two numbers and kind of look at them in a few different ways. But for the most part, like, this is one of the few things that like all the studies agree on is the shorter the MTF, the shorter the MTR. And it just kind of makes sense if you think about it. And like we mentioned before, like if you know, you find something six years later, the person is much more likely to have moved on or something else. Or there's other things that are dependent on that thing being broken that's hard to fix.

Whereas if you find something five minutes after you wrote it, it's much easier to just undo before you even commit it, you know, for example. So it's extreme examples there. But over and over again, like MTF and MTR strongly correlated. You see that all over the place. Let's see what else. Oh, this is cool. So automation needs to include operational changes. So, you know, we talk about deploys a lot of times as if it's just like dumping a bunch of bits somewhere.

But a lot of times there's things like database changes or maybe a service needs to be restarted when configurations change or maybe something needs to be i don't know backed up or paused before deployment happens and then resumed or maybe a maintenance message needs to go up there's all sorts of things that are related to uh to deployments that needs to happen and all that stuff needs to be codified too thank you uh remember though the last episode like the examples that was given in the book related was like firewall rules as an example of like that operational change that you should commit, you know, you should codify and commit the,

you know, automate that, that deployment. And you can imagine like, you know, somewhere like a Google or something that's had a long time to kind of mature and has a lot of people working all the cool stuff they must have in their deployment pipeline. So like a release gets ready, they probably dynamically spin up network rules to allow that release to even happen. They probably generate keys or, you keys or whatever secrets just for that one release, and they probably tear that stuff down afterwards.

There's all sorts of cool stuff you could do as a developer. It just takes time to do it. And so, of course, that's extreme cases, but I'm sure we've all been familiar with things that like, oh, we did the release, but we forgot to restart, and that's why you're not seeing the change or something else happened. We forgot to update the field in the database that turned the feature on or something like that oh yeah especially like in a feature uh if you're using feature flags uh for that you know which this book is uh you know advocates for feature flags as a way to um release code without enabling it.

Yep. Right. Which going back to our supporting the trunk based development, then, you know, that could be one of your ways, maybe that could work for you. Yep. And the final point here is enabling self-service deployments. And this is great for teams or individuals if you work on a significantly complex project or uh you know application

sometimes you need to have a whole deployment that would happen just for a developer because you can't run all that stuff on a single machine or it's like if it's significantly cloudy or involves cache registers or you know like physical hardware in the real world like it's it's difficult to be able to do that. And so you need to be able to enable these self-service deployments. So not only is your build pipeline able to deploy out to production, but it also needs to be able to deploy on demand to a new environment. So someone can demo a new feature, can turn on flag and check something out.

Or like say you've got a bug, you need to be able to spin up that environment in an alternate environment and kind of mess with it in order to try and recreate the conditions that make that possible. And these things are all much easier to do once you have a good pipeline. And that pipeline enables that. But you've got to be careful not to create these magical special unicorns like kind of like we talked about with like the laptop in the corner that can only do the builds. Same thing with an environment that's been manually modified. So that's the only place you can deploy because it's the only thing that's set up for it i'm surprised you didn't mention uh qa when we were talking oh yeah like um you know a long time ago you know i worked in an environment where we had our team city

environment set up to where um qa owned you owned, you know, part of the process. So if they wanted the ability to, uh, excuse me, to test, to test, uh, you know, the latest code or a specific code, you know, they, they had access to go into, in the ability to go into team city. And, you know, we were using that for the for deployments for uh qa environments they could go in and click a button like literally press a button boom and they're they're you know the deployment will go to spin off their environment that was you know before octopus deploys and things like that but yep you know i worked somewhere that had um feature branches and also

labs what i mean by they called them. So it basically meant like a couple different environments that stuff could be deployed to. But the deployments weren't automatable. You couldn't arbitrarily deploy whatever you want to wherever you want. There were like two or three labs set up. And this lab had the SharePoint farm. And this lab had Exchange and Active Directory.

And this lab had the Linux bosses. And so any given day, the QA would say, okay, we're deploying this feature branch to this environment and we're testing a SharePoint with it. And you can imagine that was all sorts of problems and things get backed up and the build would be broken that day. And so you'd have to bump it to the next day and get the whole schedule off. And so there's just all these headaches that are around that stuff. And if you kind of contrast that with how things are done today within the modern tooling that enables those changes it's crazy to think that we dealt with that but you

know that was a long time ago now your whole uh lab could be just a kubernetes configuration yeah yeah yeah nowadays like if it doesn't fit in kubernetes like i don't want to do it i used to remember thinking like like if your project doesn't have a Docker container, I don't want to mess with it. And now I'm like, if you don't have a Kubernetes operator, custom resource definitions, I don't want to mess with it. I've heard that Docker is the new Git, but I don't know. I know, man. Maybe now that author would say that Kubernetes is the new Docker, is the new Git. There's a lot of pushback on Kubernetes.

I still see it on Twitter all the time, but I don't know. I still think that – I think Kubernetes is here for a long time. I think it's a safe, super safe bet at this point. And there's some people I feel like are listening now that are like, no doy. And there's some other people that are like, that's ridiculous. It's so far from my needs that I just think you're crazy. And then there's some people that are like, oh, I've heard a lot about it, but it's a huge undertaking and I haven't gotten into it yet. Oh, yeah.

And it's super hard and complicated and it's eating my soul. Yeah. I'm in that camp. Okay. So decouple your deployments from the releases this was an interesting concept here right where they were saying like your releases are market driven uh you know that that's a market driven capability or i wanted to call it a feature but uh you know the concept and market driven concept was called that.

And it refers to when features are made available to the customer. But going back to that whole feature flag capability, you could deploy code without it being part of a release, right? And I think it was maybe this chapter or maybe I'm thinking of something else that I read where they were talking about Facebook in particular as an example where with the Facebook chat. Did we talk about that in the last episode? Is that what I'm thinking of? Oh, yeah, yeah. How they rolled it out? Yeah, yeah.

Okay. So it was the last episode that I remember where they slowly – they had deployed the code and only some people could get access to it. So by the time they were ready to like turn that feature flag on for the remainder of the world, it was like, oh, wow, how did you do that? Like, how did that go over so smoothly? It's because, well, technically the code had already been deployed multiple times and had already gotten a lot of tests. They just hadn't like released it for everyone. Yeah, remember recently Facebook announced they had released a new reaction, and some people had and some people didn't.

People were getting all up in arms, and Facebook was like, how did you do that reaction on my post? I don't have that option. It was awesome. I was entertaining like two days. They do call out here that it is important to be very careful about

when you mention a release versus a deployment because they mean two different things, right? And it's about building the culture around this. Yeah, if you aren't careful with your words, then you're not careful with your principles. It's like one kind of follows the other. If you don't make that distinction, then there won't be a distinction. And if you don't encourage people to say deployments and releases when they mean two different things, then the concept itself just gets muddied.

Yeah, there was another thing, too, though. We haven't even talked about this, but remember LaunchDarkly? Yep. we haven't even talked about this, but remember launch darkly, right? Their whole, their whole business model was all about like feature flags and being able to like, you know, slow roll users into your, your new feature so that you could see like what the,

you know, how well it worked and like, and be able to target like, Hey, maybe I want people from, you know, specific regions or demographics or whatever, like in order to, to test your feature, right?

So that, I don't know, it just came to mind. Yeah, that's another one of those where I was like, first heard about it, I'm like, terrible idea. I mean, I should have started that company. That's so dumb. I can't believe people are here for that. And like two days later, I'm like, man, that's so great, so fantastic. I didn't even realize how important this was.

I'm an idiot. Oh, no. Here's so great. So fantastic. I didn't even realize how important this was. I'm an idiot. Oh, no. Here's how it always is for me. When I hear of any new thing like a launch darkly or anything that turns out to be a thing, I'm like, you can't make money from that. That's such a dumb idea. And then they do. And I'm like, oh, yeah, well, that makes sense why we need it. Selling books on the internet, it'll never work.

So if you ever have like a million dollar idea and you ask my opinion and I'm like, that makes sense why we need it. Selling books on the internet, it'll never work. So if you ever have like a million dollar idea and you ask my opinion and I'm like, that's a dumb idea, do it. Do it now. Yeah. Yeah. The next thing we got here is they say feature flags, they actually, you can toggle your functionality independent of the deployment, right? So maybe it's configuration or something out there, whatever, right? But the key is you don't have to deploy to get that feature turned on or off yeah feature flags release functionality that's already been deployed and they allow you to enable rollback and this graceful degradation

right like if something goes wrong turn it off right it's it's an easy flip of the switch instead of having to oh man we need to queue another build We got to wait an hour before we have this thing out there. We released this feature to 10% of our customers and 10% of our customers can't check out. It's a whole lot better than 100%. For sure. So how did we get there? So we talked about a lot of ideas. We talked about culture change. How do you bring this about especially in a large organization or an organization that's resistant to change

and we've got a couple points of guidance here and this is a whole big section that's really good but the kind of the main points here is uh they encourage you not to start over and we've talked about how starting over is you know commonly referred to as like the worst thing you can ever do and a lot of that is because a lot of, if you aren't changing your processes and you're not changing the way you work and not changing the people, then you just end up building the same thing. You make the same mistakes and you just end up at the same exact place with

more broken stuff that you weren't aware of. That takes more time to kind of settle. So they recommend fixing forward. Oh, really like that one. Then there was this strangler pattern. Had you heard of this before this? I think you've told me about it.

Really? A couple on a past episode. Yeah. Martin Fowler. So there is a vine, I think in Australia that grows up on the trees and then are maybe goes down the trees.

I don't know. Either way, it ends up kind of taking over the tree and it uses the tree to get up to the sunshine and down to the ground. And over time, it just keeps growing and kind of strangling the life out of the tree. So, eventually, by the time it's kind of done, the tree is basically dead and gone. But the vine is strong enough to just kind of stand on its own so it actually like eventually just eats the tree it consumes it and it doesn't need anymore so that's uh the recommendation here is basically the name is awful strangler it sounds like someone's gonna you know grab you in a dark hallway but uh that is just basically push the good stuff in, push the bad stuff out, and eventually you're left with all good.

Oh, yeah, it was a fig tree. I don't think that it was – I'd like to be able to take credit for that. I don't think I ever said anything about the Strangler pattern. Yeah. Oh, well. I'm pretty sure. They should call it the Kudzu pattern. Yeah.

So we talked about this in 2014. What? We did? Really? No. You're making garbage up. He's getting you back for the singing. Google auto-corrected me to stranger. Apparently we talked about the stranger pattern in 2014.

So I don't know. I guess maybe I made it up. That would make more sense um yeah so they talk about like decoupling your code and your architecture yeah and uh that's something i kind of struggle with too like it still kind of feels wrong to me to have like deployment bits mixed in with your source code and like i guess i should say like i don't disagree with it it just still feels off to me i've seen some things like the get ops patterns that kind of end up having some things in separate repositories which i think is

kind of nice because the deal here is like sometimes you want to be able to deploy your code to many different places so it's kind of weird to have all these different kind of support structures for deploying to Azure or Amazon or local dev or whatever. And just all in one repository and like what separate folders or override files or whatever. But it seems like though, because like if you're talking about your architecture, like separating your decoupling your code from your architecture, but then we're going to talk about like, oh, it's so awesome when we have a Docker file in our repository that sits alongside of our code and a Jenkins file right there. Those are architectural bits. And then if you have any Kubernetes YAML that's also included, that's so much architecture that you might put in with your code that, wait, what does it mean to say that we're going to decouple that? Yeah, it's just kind of like you think about, let's say, MySQL, like an open source project used in a variety of different ways.

Some people bundle up with WordPress. Some people deploy it via Kubernetes. Some people do it with Docker and local development. There's so many different ways you can use it. Should all of those have the various bits about where they end up in the MySQL main repository? Should those be brought

out into separate repositories? This whole notion of having your DevOps stuff checked in to the same repository with code too kind of encourages, I think, votes for having a monorepo too. Because repository with code to kind of encourages i think votes for having a mono repo too because it's hard to kind of have a really modular deployment strategy well i mean even if you take the point of like um uh oh shoot i forgot it but but you know you you look at like it doesn't have to you talk at, like, it doesn't have to, you talk about the monorepo.

It doesn't have to be a big repository. It could be, like, a small, like, a QIT project. And you're like, oh, hey, here's all the Docker files that you need. And now, like, that's, oh, that's architecture. But now I remember the other point I was going to make is, like, last episode we were talking about committing things like the compiler. Like, that's an architectural bit, right? This is the version of GCC that we're going to use, for example, right? Or here's the version of.NET or Java, right? I mean, it's maybe not the type of architecture. They might

mean more like server layout or whatever. And maybe that's what they mean by the decouple architecture. They're saying like, you know, you're describing the servers and the firewall rules. And it's not like a physical asset only, right? Yeah. Like you don't want to see like Azure logic and Azure stuff inside your actual code. Like those things are separate and should be separate. But it doesn't really have to do anything with the repository, so I kind of went on a weird tangent there. Mysql, by the way, does have a separate project for their Docker files.

Huh. Oh, yeah, it's just kind of hard. Even with QIT, we talked about whether to do QIT as a monorepo or a bunch of separate ones. And so if you go the separate route, okay, great. Now we want to Dockerize it. Okay, great. Docker file per project, that works out great.

Now Kubernetes and some of these services are dependent on each other. So we need one big project that knows about all the other projects. That's where in open source projects that are like that, that can get complicated but you know maybe within your own company if you have like uh you know a nougat feed or an artifact or something like that where you could push like

common um you know compiled bits which even in an open source project if you were like you know npm things or like push up to nGet or whatever, then you can get away with it, right? Oh my gosh. So here's an analogy for you. I just decided on this and it's maybe not good, but you know, we talked about SQL optimizers a couple of episodes back and how if you're doing, working with a NoSQL database, you really have to think about how your data is going to be used together because it's up to you to come up with a good schema that's able to fetch everything that you need. Otherwise you're going to be doing a lot of stupid work and just a lot of waste. So splitting up your repositories is kind of the same way.

Like you really have to know how to split those up so that you're not, you know, doing things in a terrible way and that things that are related to each other and that you're changing together aren't totally separated. So it's up to you to really have a good understanding of that. Whereas if you have a modern repo, it's like, screw it. Let's just throw it all in one place and let the DevOps person figure it out. I mean, it solves a lot of problems when you have one repo. It solves a ton of problems, but it also does create some other ones. It creates a ton too. Yeah.

I mean, it really depends on what your threshold for various types of pain are. And so it might solve all the problems that you're not willing to, to live with. And the ones that it introduces, you're like, eh, I can, I can deal with that. Right. Um, but also too, I mean, like I kind of view repositories and code is like a living, breathing thing and it's going to like breathe in, it's I mean, like, I kind of view repositories of code as, like, a living, breathing thing. And it's going to, like, breathe in. It's going to breathe out. It's going to expand. It's going to contract.

And, like, we've seen repos where we've had, like, you know, we shrink them down and so they contract. And we have individual repos for every little thing. And then, you know, it starts to expand back out and it becomes a monorepo, you know, as it breathes in. And then it, you know, mono repo, you know, as it breathes in and then it, you know, comes back, you know, so like they, they organically come in, you know, uh, grow and shrink, you know, as, as, um, things change within the team and the organization and the needs. Right. Yeah.

So, well, we should do a survey on that sometime that's going to uh wrap up our review of the first way the technical practices of flow yeah for sure you guys have uh any like what was your biggest takeaway what was the thing that you got the most out of or anything you liked or didn't like the first way i mean for me it was some of the case studies that I really liked. We talked about the hand-on court. I really liked that one. I liked hearing about the Facebook chat because it was also something like this Facebook chat one especially was something that you kind of relate to. You can think back to over a decade ago when they introduced it.

I liked some of those case studies. I know you hated them, but. So funny. That's actually my biggest takeaway is like reading this as I enjoyed the case studies because it brought some things to life that it's hard for people to see, right? You can tell somebody, Hey, if we do this, then we'll get this return. And, and unless people can see it, touch it, feel it, it just doesn't, it doesn't materialize for them. And these case studies bring some of that stuff to life, right? Like the fact that people could, could deploy an environment and it saved eight weeks, right?

That kind of stuff was just awesome to hear. I mean, I've already bought into the whole DevOps story. So a lot of the stuff that we talked about here was, if it was a brand new idea to me, then maybe it would have been more impactful. But that's why the case study spoke to me more than necessarily the content. I agree with that too because like yeah having already bought into it and and liking it the idea of it and agreeing with it then it's like okay i mean you know at least some of the parts so far it's like okay now like we've talked in the past about like the you know is it a a role versus a culture so i think that was another thing that you know we're we're getting

into like that kind of idea too is like you know and we said like we would probably wait till we got you know later on to to have that discussion but yeah or have it again say one other thing to piggy back on here and this is probably my last thought on this this section here is the reason i have bought into it is because i've seen it pay the dividends that this entire section was talking about, right? It's not mythical. It's not some made-up thing. If you get these things in place, it simplifies your life as a developer, as a manager, as a QA group, as a team. It just makes everything so much better in a day to day. And it's, it's not only measurable, but it also has

a big morale impact because when you're not dealing with tedious stuff, like why did this not work today? Whatever. Don't get me wrong. Some builds have problems, right? Like things are going to happen, versions change, whatever. But when you're not thinking about this 90% of the time, it frees you up to actually concentrate on the interesting business problems. We'll say that there are some concepts that we've talked about, though, that are like taking it to more of an extreme than I've been in environments where it's more extreme than the environments I've been in to date. That I'm kind of excited to see like, oh, I can't wait to get to that type of environment, right? We're like using the same binaries throughout the entire pipeline, that kind of thing.

Well, no, not like that. I mean, even the three of us, we've been in environments where we've done that, right? Where with like, you know, Nougat specifically, you know, or, you know, even artifactory cases for, you know, um, Maven, for example, being pulling from an artifactory, but, um, you know, I was thinking more like along the lines of like automating, you know, user acceptance testing. That was something that we talked about last time. Like, you know, how do you do that? What does that look like? But, um, you know, automating that automating more performance testing, you know, I'd like to see more things like that happening. So what about you, Joe, any takeaways? Yeah, I think a high level, just talking about how important it is to get the kind of the rest of your team on board. Um, how, how much of DevOps

is really about getting, uh, the people thinking, you know, all in the same way in the line, which is really tough, but also just the definition of done. I really liked that part and I'm still chewing on how to really get it because it's, it's so easy. Just mark something is done when you check it in and you can even link to

the pull request, but it's really not done until it's deployed to a production like environment with appropriate tests in place that verify its correctness and performance and just everything that, that really, when your boss or whoever asks you when something's done, if you can truly say yes. It's so hard to capture that, and I really want that. I like that. All right, well, very cool.

We will have plenty of links in the resources we Like section, including links to the DevOps Handbook, the Phoenix Project, and the Unicorn Project. And with that, we head into Alan's favorite portion of the show. It's the tip of the week. All right. It looks like I'm up first. So, yeah, I'm not so great at the tips but one tip i do have uh is to play lots of video games to help you stay sane i particularly enjoy um card games and i guess like stupidly violent games sorry when you say card games are you

thinking like a hearthstone yes okay slay. Slay the Spire, Monster Train. Gotcha. Yeah. Spinoff Takes a Sultan. Grifflands, no. No. If I'm going to be playing cards, they should have devils or orcs or something on them. I don't want to see spades and clubs.

Get out of here. I mean, I want the same games. I just want cooler pictures. With taunts. Exactly. And so, yeah. So so play lots of video games. And if you do, Coding Blocks has a gaming channel that has a bunch of awesome people in it that have turned me on to all those games I just listed. And also there's a spreadsheet at the top of Google Drive.

You can click the pin notes and see people's names that you can play with on Steam or other places. What else is there? That's it. I didn't know this. I was going to say the same thing. I don't look up enough. Oh my gosh, yeah. I never noticed there was a spreadsheet. Yeah. That's in our Slack

community. I feel like that should have been your tip of the week is look up in the Slack channel. Read the description because there might be some cool stuff up there. Yeah, and there's a Discord too. There's occasional activity in the Discord, so if you're looking for someone to play Deep Rock Galactic or

something in, check the pinned posts. Love it. I don't like how you didn't agree with my Overwatch. But, you know, whatever. We'll let it slide. I used to play Overwatch. I remember that game. That's good.

We should do that again. All right. Well, with that, so my tip of the week is we've talked. Alan, I think, specifically gave a tip that was similar to this, but it was at the operating system level. But it turns out that JetBrains has baked in a similar feature at the IDE level. So within like a DataGrip or an IntelliJ, those are the ones I've tried, but I'm assuming it's any of the idea-based IDEs. ease, you can have a clipboard of recent, uh, how to say this, a clipboard of your clipboard. So whatever you've copied, you know, like as you've copied multiple things into it, right. And you've, you've built up this queue in JetBrains, you can press control shift V

and it'll bring up a window that'll show you that queue of everything that you've recently copied into the clipboard. And then you can pick just the one specific thing that you want to paste in. Now, I will say that unlike Alan's tip, when he mentioned it at the operating system level, which I believe was a Windows 10 specific feature, this is not cross IDEs. So if you have a couple things pasted in DataGrip and then you go over to IntelliJ, you're not going to see that same queue.

But maybe that works in your favor, right? Because then, you know, as you copied something into the clipboard from one application, and then you go back to another, you know, you're like, oh man, what was it? You know, at least it's still there, right? That's still pretty good. Yeah, the one that I shared was Windows V. And I remember too too, when you mentioned that one, though, there was like a security gotcha with that because if you put a password

in there, you would literally be able to see it, right? Like it showed up in the notifications bar, if I remember right? I don't know about that, but you do have to turn the feature on. So if you were to hit Windows V in Windows right now, it'll basically say, hey, if you want to see it all,

you have to turn on clipboard history. I don't know if it'll actually show you the passwords. There was a security gotcha there with it. I recall you mentioned it at the time. I just don't recall specifically what it is. But at least with this one, though, you can have that same similar type of benefit without that, without worrying about that security benefit. And I guess like I haven't tried this on a Mac, but I would assume that it would be like a command shift V on a Mac for the same feature.

Yeah, I'm not. I haven't tried it. Yeah. Very nice. All right. So I have I have two and these both actually stem from the same day so jozak said you know you need to play some video games to stay sane i think that we all need to do things to stay sane right now right like we've now been locked up for four months

you know for those of us that are sort of adhering to whatever the guidelines are for, you know, staying away from people and isolating or whatever. And man, there's just some days, I don't know about you guys, where I'm just like, I need to get out, right? Like, I love working from home. Like, I would have been in that percentage. I love the fact that if I want to go run an errand, I can. I just I really to a park down the road from me to where, you know, there's geese, there's ducks, there's deer, there's, you know, it was just nice.

And so my tip is if you got a cell phone and you have a data plan, it's usually like an extra 10 bucks a month to add tethering to it. That's cheaper than renting office space. And it's probably cheaper than the two meals that you bought in any given day. You know, add that and take your laptop out and, you know, go work for a few hours outside or somewhere else, somewhere new. Right. And you can still do it in a way that will keep you safe from the pandemic type stuff that's going on. But it is nice to get that mental break. Yeah. And as Alan said, like, for example, if you live in Georgia, you only get two weeks out of the year that you can do that.

In Florida, I think it's like a day. Oh, my God. It got hot this past week. I was out there until after lunch. And by the time I got back into my vehicle, I had sweat running down every line of my body. It was ridiculous. And I'll have a link to the Twitter thing where I actually have a picture of where my office was that day. And it was awesome.

And I need to dig up another. Yeah, my office, my secondary office. I even have another tweet that I put out there where this duck just came over next to me and kind of looked up at me like, yo, I'm going to do some yoga here for a minute. Cool with that? And then right after he was done, he just laid down right next to me and took a nap. And I was like, yeah, I didn't get that at home. He was probably expecting you to feed him. He was, but these ducks know me now and they know that they can just come over to me and I'm not some little kid that's going to chase them around screaming at them. Right. So, so they kind of come to me as the comfort guy at the park right now. So, um, and so here's the next thing,

right? So I've mentioned my absolute hatred for the Mac touch bar. And when I say absolute, like it's pure hatred. There's not much that I dislike more than this on a laptop. Outlaw didn't like the number pad on a keyboard. I can forgive that all day long. That touch pad offends me. Wait, touch pad or touch bar? The touch bar. The touch pad also offends me because it's gargantuanously big but the touch bar makes me angry but you don't have the false touches on the touch pad that you have like on the msi right it depends on the windows drivers right like if you have a good touch pad in windows true but the mac the macbook pros touch pad is by far the gold standard okay right i'll give you that so it's just the touch bar is by far the gold standard. Okay. Right? I'll give you that.

So it's just the touch bar is the nonsense. I want to focus in on the touch bar. The touch bar is the nonsense. So here was the thing that made me mad, right? So I work in a browser most of the time. Now, here's the deal, right? So where we all work now, we are forced to use G Suite, which is the Google set of tools. Guess what?

They all run in a browser. So you're always in a browser all day long with everything. And man, I don't know about you guys, but when I go to hit the escape button with my pinky finger, never had to think about my ring finger. And if it was touching anything else, because it didn't matter,

right? It wasn't putting enough pressure on anything to where it was ever a concern. Well, that's the real thing. Even if it was touching it, it wasn't putting pressure on it. So it wasn't putting enough pressure on anything to where it was ever a concern well that's the real thing even if it was touching it it wasn't putting pressure on it so it wasn't a problem it didn't matter and that's why mechanical keyboards are so amazing have you ever if you ever got like mad i'm gonna have to go into it i'm getting hot right now thinking about this like i'm almost sweating right now. So have you ever done something on a computer and it keeps doing something that it shouldn't do and you can just feel bile rising

in you? Like you want to break a keyboard in half, you want to throw the monitor out the window, and you just want to leave, right? I was out there programming at my secondary peaceful office and my blood pressure started rising because i hit the escape button to do something i was on a glyphic diagram which by the way things are already infuriating all right so so let's go ahead and paint the picture you're in a browser making a visio drawing which should have been done in vis. I can see this going sideways so many different ways. Dude, to unselect something, the only way most of the time is to hit the escape button.

You can even try and click on other things and it still won't work, but you got to hit the escape button. Did it take you back out to the wiki page? Dude, here's the deal. In Chrome, the freaking back button was the virtual key that was right next to the escape button. I'm in the middle of drawing a diagram that I've spent 30 minutes on, and I keep hitting this escape thing, and then my ring finger brushes that back button. Man, I almost tossed my MacBook Pro into the lake. I swear to you, I was so furious.

Okay, here's where I thought you were going with this, though, because I've had similar complaints specific to Gliffy. This isn't a thing with the MacBook Pro Touch Bar. Because for those not familiar with Gliffy, like I think you described, it's like a, imagine if you were to draw like Visio diagrams or any kind of like, any kind of drawing where you want to like connect, you know, create these boxes, draw lines. And then as you move the boxes around, the lines can redraw themselves. You know, it's that type of a tool. I'm, if I recall, cause I've read into this, you can be in that drawing and if you press escape, it'll take you back out to – like if you were in Confluence, for example, and you do a –

You hit the backspace button. Yes. The backspace button. Oh, is it the backspace? I thought it was the escape. That's what happened to you. Because I've had that happen. You thought you were probably editing text, and you weren't in the text box, and you hit hit backspace and it took you back to the previous

page i couldn't remember the exact scenario but i have had a scenario where like where that happened to me and i was like you i was ready to just destroy the world right like there weren't enough games to keep me satisfied i don't care like how many slay the spire jay-Z wants to play. I was not happy. But to Gliffy's credit, though, I could go back and re-edit that drawing and it had saved where I was and it just picked it back up. I didn't lose anything. Usually, it was pretty good about saving enough drafts.

I did lose some changes, but the problem is that happened to me like five times. Oh, yeah, because that back button is so easy to gra like five times. Oh yeah. Because that touch, that back button is like so easy to graze as you're going, reaching for that escape key. Okay. So now that my temperature, my internal temperature is now above 102 or somewhere. Um, now I will give you as a MacBook pro user that has that lame touch bar, there is a solution, at least in Chrome. You can go to the view menu and go to customize touch bar and you can delete that back button out of there. Now, here's what stinks about it.

They don't have a way to just put like a finger size space right there because that's really what you need, right? You just need one finger. Now, if you're somebody that lays your entire hand down and you hit the escape button, just delete all the buttons on the entire touch bar, right? But for me, it was just that one little graze of the fingertip for the finger. So what I did is I deleted those buttons and then they have a little spacer that you can drag up there. So I drug about 30 of those things up into the touch bar. And so now Chrome no longer makes me want to burn down the

world. So that's a huge tip. Just know it's application dependent whether or not they've implemented these touch bar controls. But if there's an app to where you find yourself hitting the escape button and something else accidentally happening, go up into that view menu and there's probably a customized touch bar setting to where you can customize the buttons that show up for that application. And here Apple thought they were doing the right thing just by adding the physical escape key back. Right? It is better. It is better. It is better. Just bring the rest of the physical keys back and we'll be okay.

Just get rid of the touch bar, man. The only time I ever use it is to do the volume or the brightness. I was perfectly fine when those were physical function key toggles that you did on the thing. I'll tell you the one that bothers me, though, and I don't remember which specific Mac MacBook Pro or just Mac and Macs in general laptops specifically to the laptops where they introduced this. But in one of the models, they used to have the the power button used to be like its own physical little round button that was off in like the top corner and then they moved it into the keyboard and it was like right next to uh like the delete key or something like that and and like there's so many times where like if i'm on that model mac or or one of those model macs and i'll be typing along and I'm like, Oh, let me delete. Boom. And I just hit the, turned it off.

I'm like, dang it. Didn't mean to turn it off. Yeah. Hey, for the record, I will say so as much vitriol and hatred as I have for that touch bar, like truly,

truly hate that thing. The, this model of the MacBook pro, which I think is a 2019 or 2020, but it's a 16 inch. It's the 16 inch. It's really a good machine because they put the headphone jack back in it. They added the hard escape button. The only thing, the only two things that truly stink about it are the touch bar and the track

pad. That's the size of the entire base of it. But outside of that, 50% of the base for sure. It's massive. At least it is good. So, so I'll give it that.

I think we've kind of described this before, but like how in the world is it that the company who focused so hard on keyboards specific to like iOS, for example, and that was like a big part of why their platform took off. How is it that they have gone the exact opposite direction and messed it all up and lost so much, uh, you know, so many people from it. And as a, as a side effect of that, like reputation and everything that goes along with that, it's like, okay,

I mean, I commend you for trying some things, but at some point, you got to stop being stubborn. Recognize, hey, this was a failure and bring it back. And they did that with the Escape Key.

Do the rest of them. They did it with the headphone jack too, which by the way, they killed off on the previous one. And I would have never bought a MacBook pro without the headphone jack in it. Like that's just doesn't make sense. Yeah. So,

so outside of those two things, it is a great little machine. I'll give it that. But, um, yeah, man, God,

just make it a touchscreen. Get rid of the freaking touch bar. Make a touchscreen and be on with it, right? Double it with iOS and let's call it a day. Like, I don't know. And if you work for System76, please take away the numpad. Just center the rest of the keyboard. I want that 17.

It looks gorgeous except for that stupid keyboard. All right. Yeah. With that off my chest. Yeah, so as I said, we hope you've enjoyed this talk about the first way and the principles of flow in the DevOps handbook. And with that,

in case if you happen to hear this show because a friend sent you a certain link or let you borrow a device or whatever, you can subscribe to us yourself on iTunes, Spotify, Stitcher, you know, wherever you like to find your podcast apps, we're cool, you know? And if there's a place where you want to find a podcast that you can't find us, that you'd like to get your podcast from, let us know and we will correct that.

We got a lot of feedback like at Orlando Code Camp about SoundCloud. And guess what? We fixed that. And as Jay-Z said, we do appreciate those reviews. So if you'd like to find some helpful links, you can head to www.codingblocks.net slash review, and you can find some helpful links. Well done. And while you're up there, definitely check out our show notes. I mean,

if, if there's anything that you heard during the show that you think that you want to remember, just go up there. It's probably in the show notes. So check those out. And we also have other articles, videos.

We have music videos. So, you know, definitely go check out cutting blocks.net. It's ridiculous. Yeah. Oh, ridiculous. Yeah. Oh, geez. I realized what I should have named

the record company afterwards in the music video, but you have to wait for the next one for that. Okay. Send your feedback, questions, and rants to the Slack, and make sure to follow us on Twitter at codingblocks,

or head over to codingblocks.net and links to the social on twitter at coding blocks or head over to coding box net net and links that's uh social on top page i don't know how it's in there ants rats and slingslots i don't know how it's codified it's all good