Coding Blocks - The DevOps Handbook – The Value of A/B Testing

Episode Date: September 28, 2020

We wrap up the second way from The DevOps Handbook, while Joe has a mystery episode, Michael doesn't like ketchup, and Allen has a Costco problem.

You're listening to Coding Blocks, episode 142. Subscribe to us and leave us a review on iTunes, Spotify, Stitcher, and more using your favorite podcast app. Please go to the website where you can find show notes, examples, discussion, and more at codingblocks.net. And feedback, questions, and answers can go to email address comments at codingblocks.net. Email. You can follow us on Twitter at codingblocks or head to www.codingblocks.net and find all our social links there at the top of the page.

And with that, I am Alan Underwood. I'm Joe Zak. And I'm Michael Outlaw. This episode is sponsored by Datadog, the cloud-scale monitoring analytics platform for end-to-end visibility into modern applications. All right. Hey, today we are finishing up the second way in the DevOps handbook. We're talking about A-B testing. So let's get to some news.

Are you sure we're going to finish it up? Multi-arm octopus testing. Okay. All right. So yes, uh, in our news, as we like to do, we like to thank those that have taken the time to go and leave us some

words on, you know, the various different places where you can do reviews. So I have iTunes and that is jinx protocol. So thank you there. All right. Now I'm going to read off some of these. Tell me if you can guess which one of these is Mr.

Smarty pants. Okay. You ready? Or Mrs. Smarty pants. Okay. You ready? Or Mrs. Smarty pants. How about that? I'll keep it easy for you, Mike and King J author.

Now, one of those is not like the other. That's amazing. That was good. And, we don't have any other news to uh to talk about because uh we have just been looking at nvidia stuff all week so oh my gosh you had to start with that right alan have you got your 3080 on order no but i want one i was gonna get an xbox i was like okay i'm an xbox and switch kind of person now but now i don't

know dude i so i i think i've got pre-orders or i will be doing pre-orders for the xbox x the playstation 5 and and i don't know what else will be coming this way but i need all the things so that i can look at them i'll never actually get to play them but i want to look at them i say like are you going to order a new n I want to look at them. I say like, are you going to order a new Nintendo while you're at it? Because you know, why not? Yeah.

I mean, although I did get Tony Hawk for two plus one, because Joe mentioned it, man, I did play that a little bit over the weekend. Awesome. Right. That was fun.

That brought back some memories. Yeah. So, all right. Um, I guess we'll go ahead and jump into this, which will be the last portion of the second way. This is like 2.5 or 2.6.

Like where are we in the second way? We're a few in. Well, for those who that are like, uh, you know, who have the book, it might be realized,

wait a minute. That's not technically the end of the second way. It's going to be the end of our second way. How's that? Because like the real end of the second way is more akin to like the episodes that we've already had that followed the, like the Google engineering practices and, uh,

you know, poor requests and code reviews and things like that. Like that, that's what the, the real end of the second way is. That's in here. That's it. Oh,

you did put that in there. Yeah. Yeah. I hadn't finished up the show notes before we started recording tonight. So yeah, I mean, I don't want to like take it away cause we're definitely gonna be talking about that.

Yeah. Yeah. We are wrapping up the second way. Yeah. Wait, buckle on. It's going to be a long one. Hey.

And, and also it's probably worth noting, um, the, what are the ways in DevOps handbook it's the first way was feedback yes is this a test

technical practices of flow the technical practices of flow the technical practices of feedback and the technical practices of continual learning and of feedback and the technical practices of continual learning and experimentation right so we are wrapping up the feedback okay so first one is really uh continuous integration uh and then the way we're talking about now basically deals with the kind of feedback and the third one just gets weird i I like it. Spoiler.

Yeah, spoilers. So look for the upcoming weirdness. It's kind of weird. I mean, it's not weird. It's, I don't know. I didn't love the third way. Third way is not my favorite. So speaking of which, because we always forget this until the very end. So only the people that hang out with us for the next three, four or five hours ever get there.

If you want a chance to win your own copy of the DevOps handbook, be it Kindle, hardback, or if you prefer audio book, leave a comment on this show, codingblocks.net slash episode 142. And you'll have a chance to enter and win that. All right. So that out of the way, let's actually dive in here. So the first section that we're talking about tonight is integrate hypothesis driven development and A-B testing. All right. And so the quote we got here is the most inefficient way to test a business model or product is to build the complete product to see whether the predicted demand actually exists that sounds like a lot of stuff we've

talked with like uh building mvp and uh if you ever heard of the notion of trying to sell something before you actually build it that ties in nicely it's pretty much like the old way of of doing things right like you would actually build a thing it's kind of like the the field of dreams except it's shit like yeah like like that that that movie really had it wrong i guess is what we're learning build it and they will come yeah build it and they will come that was the whole thing about the field of dreams right but uh really what we're learning is that's not the way to do it maybe build a portion of it Build like home plate and then see if everybody comes. That's kind of what they get into when they're talking about this. They say you need to constantly be asking, should we build it and why should we

build it? That doesn't sort of go hand in hand with designing it all up front and building it for six months. It's more about, hey, is this what we should be doing? And one of the things that they got into is Intuit, the tax software company, they really support employees doing this whole high velocity experimentation. The case studies again, I Joe, I don't understand how you don't like the case studies.

Because the cool thing about Intuit, especially that they were talking about, you can slap me later if you already were going to plan on talking about this later. But the cool thing about Intuit was that they would actually do their experiments during their peak season, right? Like they wouldn't wait and like, oh, let's see if this works. Like during peak tax season, it's like, hey, let's try all this. And the authors like make a point of like, by doing your experimentations during the peak season, that's when you can really learn like if something works or if your customers like it, rather than if you did it during an off-peak season, you might find like, okay, yeah, this totally works. But maybe the portion of your customer population that you

tested it on is a small fraction that doesn't really represent your real business during your peak season. Hey, and the important part to point out here is why that's such a big deal is in all three of us have worked at big companies. A lot of times where you get into peak season, like UPS, for instance, right? Their peak season is around Christmas. That whole month of December, they lock everything down. They're like, yo, you're not changing anything because we need to make sure that we can deliver these packages and we don't have any hiccups right and so it's very counter to what they said they did with intuit because their whole thing was you know this is the best time to find out what our customers are actually going to like or dislike about what we do right like and so we can get those incredibly fast feedback loops

and the important part here is that ties back into DevOps is the only reasons that it's possible is because they, they have enabled themselves to release quickly and, and continue to release as fast as possible, right? Like their, their deployment pipeline is solid and that's why they can do this. But going back to your question about like should uh constantly ask like should we build it though right like um by doing some of that experimentation during that that peak season though you can also get that kind of answer like hey is this even worth doing right right yeah it's ridiculous i'm sorry why would you do that why i haven't now to be fair i have not read that case study because i think I already know everything.

It's kind of like math algebra. It's like you read the proof. They expect you to do problems. It's just an insult. I already read the proof. I got it. Let's move on. So you don't need to do math problems.

So says Matt, I'm a chicken. So case studies are wasted on Joe. Like, hey, you told me this is how it works. I just believe you. That's why I'm so good at math. Alan and I loved the case studies in this book, and this was just one more of them that was awesome. Yeah, I mean, one of the things that they also said about Intuit, and I don't typically put a lot of the case studies in the notes because I think that's the goodness that you'll get when you read the book that I mean we only hit on parts of them but one of the

cool parts they said is by doing this they were able to out experiment their competition basically meaning while their competition's locked down during peak season they're like hey let's put this change in and see if people are digging this, right? Like if this is working for them, if they do, then they can iterate on that particular feature. Like whichever one of the 10 they tried, the two of them that click, they could be like, let's focus on those, right? And that's amazing. Yeah. And this only works because they have fast feedback and the ability to quickly adapt. So if it's not working, they can stop quickly. And if it is working, they can invest further in it right now rather than waiting a whole other year to realize the benefits.

I feel like there's a related spoiler to the Phoenix project that I guess we're not allowed to talk about in this episode because it would be a spoiler. No, we've got to do a spoiler cast. A spoiler cast, okay. Or we ruin a book for everybody. Yeah, maybe we could do a Zoom. We should do something. We should stream it or something

and invite people and just get it all out there. Did you end up listening to me, Alan? I did. I've listened to the entire Phoenix project. I really enjoyed it. Awesome. What about the Unicorn? I have not done the Unicorn project yet. Yeah. I can only listen to so much fake,

sort of real tech audio books at once. So, you know, I got to take a break and then I'll refresh and come back to it. Okay. Which one? Okay. So between DevOps and Phoenix, which one you like him best? I think DevOps. Okay.

That's not to take anything away from the Phoenix project. No, I get it. It's a tough call. I think the reason why I like DevOps is because I, just what you said a second ago, I really like the real-world use case. What are they? Case studies. Case studies, yes.

The case studies. Because I really like hearing that, hey, Yahoo did this, they were able to increase their their deploys by a thousand times. Right. Or or Intuit did this. And so they were able to beat out their competition. Like it's one thing to read a novel where somebody could take liberties with how they want to do something. And it's another when you hear, you know, this company actually did it and it went against what everybody out there thought was the right thing to do. And guess what? They're kicking everybody's tail now because they saw that opportunity.

And that, to me, I think that's why the DevOps Handbook sort of wins out. Okay. All right. So then let's get into the integrating A-B testing, which is pretty much what this section is really about. So this is also known as split testing, or at least that's what they refer to it. I don't know that I've ever heard it as referred to as split testing. Have you guys?

I have. Really? Yeah. I thought A-B testing and UIs, I thought it kind of got out of favor because of split testing where people kind of have different ways of doing things. It isn't necessarily like A-B kind of implies almost like 50-50 and only testing one thing at a time. And split testing gets a little bit weirder. Yeah, because it's technically like you could just shave off like, hey, I want 10% of my traffic to go to this other thing. Let's see how they respond to it.

Do they even use the new widget? Or if they do, does it work? And that's one thing that we get into here in a second that is almost appalling. So for anybody that's not aware of what A, B, or split testing is, it's pretty simple, right? Like if we're talking about a web page and I think even there was some company that was doing this for Mac users versus Windows users at some point. I can't remember exactly what it was. I want to say it was like a travel site and the Mac users, they would show more expensive things because and then the Windows users would get a lower end type package that

would be presented. And the thinking was people who buy Macs are willing to pay a premium for that thing. And so that's how they did it. So at any rate, A-B testing is really taking a portion of whoever's coming to your software. And some of them are going to see one version of something. And it may not be a whole page. It might just be a feature on the page. And then the others will see the older regular version. Right.

So it's a way of testing these features out. And these aren't these aren't ads. Don't confuse these with like ads that might be specific to one one user versus the next. But these are like features of the of the site. Uh, for example, if we're talking about webpages, uh, you know, features of the site that might, you might only show like 10% of your users. Um, you know, that version, an example might be, let's say that you changed the navigation and you wanted to see, Hey, are people able to find my products better now with my new navigation, right? You might show your new navigation to 25% of the population because you don't want a hundred percent of your population being frustrated if it

doesn't work. Right. And then, and then you trace, you know, what was the average order value or whatever after that. Right. So, so you start putting in metrics to where you can see that stuff. You've likely seen this and maybe not even realize, especially if you shop Amazon, because I I've seen it all the time with Amazon where like the layout of like the page, the product details page will change. And then if I open up a second window to, you know,

comparison shop, you know, uh, you know, or not, not comparison shop against a different company, but just like look at a different product, both on Amazon,

I might see different product detail representations. Have you guys ever had that happen? Yeah. Yep. It's annoying, too. It kind of is. Some of you get different prices and stuff. It's weird.

That's the part that's annoying. So if you want to see this happen on Amazon and you really want to go a little bit crazy, if you see a product that you're sort of interested in, sometimes you'll see it pop up and say, hey, there's a 5% or 10% coupon if you order it right now. Open it up in a totally different browser and you may not get that coupon offer. Like that kind of stuff drives me crazy because I know that that coupon's there. Why can't i find it again you know it's you know whatever yeah real quick uh on while we're still talking about testing uh i guess we're gonna be testing about talking about testing for a while but i looked up um

what i was thinking about for um split testing uh ab testing and uh what i was thinking of which is multivariate testing and um kind of what i was getting at is um when you talk about traditional ab testing you're generally only testing one thing so you might change the background color or the link color or drop shadows or move a picture by 20 but the idea is to test something very small and the op or the uh alternative here is multivariate testing where you test multiple variables and the downside is how do I know which variable worked? But the upside is you get to test combinations of things. If you think about doing

one thing at a time, it's like, well, we made the heading bigger and it worked. Now we made the picture better and it worked. Now we put them together and it doesn't work. It makes it more complicated to do the math and split things up in smart ways, but ultimately you get kind of split things up in, in smart ways. But,

uh, ultimately you get to test more things and more things, a combination faster. And so that, that's why I'm kind of, I think of a, a B testing is kind of becoming less popular, but that's only in the very specific,

like kind of website, e-commerce oriented, uh, marketing way of thinking. Yeah. So this now is where, is where we get into the part that is a little bit depressing. So there was a study from Microsoft where they found that only about a third of features improved the key metric they were actually trying to move. So if you think about that,

if you made three major changes, only one of those three would actually do what you wanted it to do. And that's why going back to the Intuit thing, why that's so important is the fact that they can quickly experiment and iterate on that. They can quickly throw away those two thirds of things that they know aren't going to work, right? Like, hey, we did it. Get it out of the code base right because they probably didn't spend a ton of time building this massive feature that it's going to get launched and then people are gonna be like yeah i don't really care right it's a whole lot easier to throw away that cruft when it's a very small chunk of stuff that you put out there instead of some massive feature set well this is why like features like a um like okay so we've been we've been talking

about recently telemetry and metrics and everything and if we go back to okay yeah i can wait a moment um i think that was awkward uh if we go back to uh i think we mentioned in the past i had mentioned like uh the ibm core metrics right when you can tie some of these changes in with metrics where you can actually see like dollar impacts, right? Like, you know, it's easy for all of us to be able to equate to, you know, hey, did this actually make me more money versus not? Because other things can kind of be a little bit more subjective. Like, you know, did they stay on my page longer?

Like, okay, maybe. I don't know. I mean, it depends if that's what you want. Um, but you know, it's really easy when it's just money. And so like, uh, like the IBM core metrics, you could actually see and put like attributes on specific buttons. Like, Hey, did they click this? Did it actually convert into a cell? And how many more sales did I get? Because that thing was there or even present on the page, right? Totally. And the important takeaway with this entire thing, even what he just said is if you make a change, you add a feature, you do something different and you're not measuring the

impact of it, you have no idea whether or not you actually increased the value of what you're delivering, right? And there's a danger to that other than just maybe you didn't make any more money. You spent a lot of time building this thing and you didn't make more money. The worst thing is as you add more things to your code base, you're making this thing less maintainable over time, right? It's actually harder to introduce new changes because it's going to get more complex. So you might be increasing the complexity at the same time,

also driving down the value of each person that's hitting your site or your application or whatever, right? It doesn't have to be a website, but yeah, metrics are so key into actually understanding if what you're doing is being effective. Yeah, it's tough though.

Like there's a lot of like single, like if you're focusing on one single metric for something, it can be a real big pain. How many times have you gone to a website and you had some stupid thing pop up to ask you to join the mailing list? Or take a survey as soon as you go to the website and you're like, you haven't gotten anywhere yet and you've got to do this thing. And it's so annoying. But if someone looked at the survey results for that, they're like, oh my gosh, we popped up this big annoying thing and our survey responses are up 700 but uh revenues down 10 because some you know some number of people left and same with the microsoft thing like you could

say okay well um let's say we added a feature that was designed to drive engagement call it export reports and so now people are going to our website less because they're getting emailed reports and so they don't have to go to the website as often but the idea behind that or why that could still be good is because maybe those subscribers or some other way of making money from these people has actually improved and that can sometimes take a lot longer to measure so sometimes the metrics can be tricky but i still think just like i said you have to know what you're going after so if you're adding like an export feature or automatic or scheduled something that prevents people from going back to your website, then it has to provide value in some other way or else what are you doing? Right.

Now imagine this though. Like start – because we're basically getting to the point where we're combining a bunch of things that we've been talking about over the past few episodes, right? So this is all fine and dandy for a webpage, right? Like it's easy to wrap your head around that because it's like every time a new browser hits that website, like you're delivering a new copy of the thing. And so each one you could tailor. But hey, what if the thing that you want to AB test is an application in the iOS app store, right? The Apple app store, like Apple wasn't going to let you have two versions out there. And now that's where you would feature flag things to say like, okay, for this feature, you know, maybe I I ping back to a server and 10% of my user base can see this new feature or everybody else is disabled.

They don't even see the button, right? So you kind of see putting it all together, the value that it could provide. But again, you know, you know, that's, that's combining our conversation about feature flags, combining the conversation about, uh, metrics and telemetry, as well as now we're talking about AB testing. Yeah. And go ahead.

Uh, you can go ahead. I'm on. It's kind of a diverge. Uh, we'll go ahead and diverge. Okay. Well,

I was going to say, uh, the third way that I mentioned being kind of weird, it's also the same kind of idea here where it's like if you get your stuff automated and you can actually measure what you're doing then you're you open yourself up to a lot of different kinds of experiments and different things that you can do to try and stay ahead so it's kind of like that uh that old notion of like uh kind of crawl walk run where you've got the basics down now you can go exploring

although some would say you should run walk crawl right basically get it out there as fast as possible and and then iterate on it a little bit slower right let's let's polish the thing and then and then slow it down when you've got it where it needs to be then stop messing with that thing and then do the run, walk, crawl. Run, walk, crawl. We got another feature. Say it three times fast. I dare you. You run to build the features until you sell your company

and then you walk to the bank and cash a check and then you're out of there while the company slowly crawls to their death. I was going to say you crawl out of the bar after you celebrate. There you go. Where are your priorities, sir? So you kind of creeped into the next section, though, there, Outlaw, with the feature flag. So the next part that they wanted to talk about was you need to integrate your A-B testing into your releases.

And that basically means effectively putting feature flags into your product, right? And then by having a release pipeline that has this sort of in it, you'll be able to do those things quicker, right? And be able to turn those things on and off as time goes on. And they even talked about it. Again, it surprised me. I think I've been more shocked in this book by the number of features that Etsy has given out to the world than just about anything else. Because one of the things that they talked about here was they open sourced their feature API. Dude, they've been mentioned so many times in this book for things that they've given to the community.

I mean, it is like a website for makers to give out their wares. That is true. Like a couple of the features that they pointed out that is in this feature API is they have this thing called online ramp ups, which is, you know, as, as the things come in, you get more and more people trying out this new feature. That's kind of cool. And then you could even throttle the exposure to features.

So if you wanted to slow that down and choke it off a little bit more, you can do that. So these are built into the framework. And I think we've even talked about this in the past when we talked about A-B testing. There's frameworks for.NET. I say framework. There's like NuGet packages for.NET to be able to toggle things on and off. I'm sure there are for Java and everything else, right? Well, we've also talked about services like LaunchDarkly.

Right. So that you could just spin out feature flag your portions and control like, hey, what kind of browser, what kind of percentage? You get into like demographics and locations and things. Yeah, I want LaunchDarkly. I just don't have anything to use it with. So if you ever go to codingblocks.net and it's not the most thrilling website, you're in the 10%.

That's right. We really ought to do something with the website. We ought to do some stuff. I mean, not me. Let's redo it and make it look like it was from 1995 or something. Put the little Duke animated logo on it.

Under construction with the guy digging. Yeah, and a marquee tag. If you can help us with the website, send an email to our email address saying that you'd love to help us with our website. Oh, my God. I'm so kidding. He's absolutely kidding. I'm so kidding. That means we're going to have to do work anyway.

Well, that plus we get that exact email, I don't know, 10 times a day. Right. exact email i don't know 10 times a day it's right it's mind-boggling how many people email a day to say they'll redo your website or get you new content or get you number one in google or whatever oh my god are you interested in adobe users oh my gosh oh for a presto users like they know who they're talking to like would you like kafka users presto users elastic search users sql server like every day, it's insane. Insane. And that's the stuff that makes it through the spam filter.

Right. Yeah. We need to work on that. I still need to put that in. All right. Anyways, so there are other products out there besides this feature API, some that you might have heard of, one called Google Analytics that seems to be everywhere on the interwebs. And then Optimizely is another one. I think that Joe,

you actually shared a link from that up there above. I didn't realize you do AB testing with Google analytics. How did I understand that? You can do targeted marketing campaigns and all kinds of stuff and you can, you can track it. Oh, okay. Yeah.

Like we totally have, we have everything we need to do to do cool tests and find out cool things. But targeted marketing, though, that's more about telemetry there, right? I don't know. I don't think so. I think you can actually have it do things like show different ads from Google. And there's all kinds of things you could do with it. I've never dug into it deeply because it just requires work.

I just thought that was a telemetry thing. Well, we don't have anything to sell a lot of it's built around the funnel right so they're like okay well define the parts of your funnel and we'll tell you how far people get and like oh I don't have anything right we need to figure out something to sell as well I'm saying yeah I mean that's why like I mean you're talking about that funnel and and that's why the Amazon uh one click was such a big deal for them right like that that's what was such a big deal for them. Right?

Like that, that was such a game changer because there wasn't, there was no funnel. It was just a hole. You like, you click it and that that's where that's the hole your money went in. Buy with one click done. It was an effective hole.

It still is. That also contributed to me doing like three Amazon orders in a day though, which, you know, I guess they've managed to do OK despite that. Yeah, they're still batching that stuff because things still come in the same boxes sometimes, even when I do that. I can't believe they got a patent for that, though. That was the thing that I was like, OK, software patents are a joke.

They really are. So the next thing we have is integrating the A-B testing into feature planning. So what they're saying here is you just need to tie your feature changes to business goals, right? That sounds like such a simple thing. But how many times have you guys been in meetings where people are like, yeah, we need to add this? It's like, wait, why? Is this more important than this other thing? thing like who's going to bat for this see that's the thing that stinks about this chapter is like this means you have to communicate with like other departments and stuff because they're the ones that usually

kind of have those metrics that are closer to the customer on that stuff so if you're reducing customer service calls or you're increasing sales by whatever like you kind of need to work with those departments on defining those metrics. And then you both need to be able to see those metrics. And sometimes that kind of communication, especially as the organization gets large, is tough. Yeah, it can be. Man, I fell down the hole.

What did you buy? The one-click hole. No, I went to my favorite website, Wikipedia, again, because I was curious about the patent, and it turned out like it was challenged, but they actually held up, but it has since expired. So now we could introduce. That's the only thing we were really waiting for to redo,

CodingBlocks.net. CodingBlocks, yes. We got a funnel. We wanted to do one click shopping so yeah so get on that alan that's yeah yeah i'll be right there oh so one of the things that they're talking about with tying these to the business goals it's very much a scientific approach if you remember the scientific method from back in your grade school days this this is it. Form a hypothesis. What do you expect the results to be? And then at what point do you deem that thing as success, right? So you kind of have to know what you want out of it going into it. And again, we've said this so many times since we've started this particular book is the ability to deploy quickly is what enables all of this to happen.

Right? So just know that you're going to have to put some time into the DevOps side of things. I would like to rephrase that. Okay. And it's not, it's not necessarily about deploying quickly as much as it's deploying consistently,

right? Like having it every time it's a repeatable process. I mean as it's deploying consistently, right? Like having it every time it's a repeatable process. I mean, it's quick because you're not manually doing things, but don't, you know what I'm saying? Like it's not necessarily focused on the ability, like the deployment happens in like 60 seconds, as much as it is that the deployment happens. And every time it's consistently, you know, deploying the bits the right way to the right places, the way they sort of get behind that. But I would say like, if your deployment takes, let's say that you're building something

like windows, like this is an extreme case, right? Like what if it takes two days to build that and deploy it, then you can't react to things very fast then, right? So that's a problem. So I think to your point, a minute versus 15 minutes, like probably don't care that much, right? Days, like then you probably start worried about what your deployment pipeline looks like because a massive mistake could, depending on your size and your audience, could be a big problem.

Well, yeah, but you could still solve that problem in different ways, too. What was that build process? It started with a B that Google had, like Baffle or something where, dang it, it wasn't, I mean, I'm just, I made up Baffle, but Babble, was that it? It was Babble? Oh, I think so. The build process that they had where it was a much more intelligent build system. Yeah.

So when you get to the scale of you're trying to recompile Windows or some massive... Bazel. Bazel. There you go. Thank you. If you're trying to recompile the Google search engine or Kubernetes, then there are other tools that you should be looking at to add to your build chain that can add to that to make that quick. So I will concede with the quickness being important as it relates to being able to experiment, right? But I don't want to take away of DevOps to be that like, hey, this is about deploying quickly either, right? Because it's more than, it's so much more than that. It is so much more than that, totally.

Too much more. It's really about the YAML. Today's episode of Coding Blocks is sponsored by Datadog, the unified monitoring platform for full visibility into all of your serverless functions. Troubleshoot performance issues faster by seamlessly navigating between logs, lambda metrics, and distributed request traces all within one unified platform. Yeah, and Datadog provides real-time screen boards and service mapping so you can get complete observability into your serverless environment. And, you know, we've been talking a lot about metrics lately, and I was just scrolling through the Datadog blog, which is just fun.

If you like visualizations or data or metrics anything you're gonna love it but anyway just looking at the latest things they've added including the ability to do complex boolean filters on metrics and learning and the things they've added for incident management and of course serverless functions and just looking at the visualizations they are killer so definitely want to go and create yourself a dashboard if you haven't already and do it anyway, just because it's awesome. Yeah, they have some amazing visualizations. So start your 14-day trial today. Receive a free Datadog t-shirt after creating just one dashboard.

And make sure you visit datadoghq.com slash codingblocks to learn about how Datadog can help you optimize your serverless environment. Remember, that was datadoghq.com slash codingblocks to learn about how Datadog can help you optimize your serverless environment. Remember, that was datadoghq.com slash codingblocks. All right, so it's that time of the show to where we ask you, if you haven't had the chance or you've been too busy and you wanted to do something really nice for us,

head over to codingblocks.net slash review and click one of those things there. I think we have iTunes and we have Stitcher as two places that you can easily go in and leave a review. It really does mean a lot to us. Like when we get these things where people say, Hey, we get it. I get a good laugh and I'll learn something or, Hey, you helped change my life. It really means a lot when we see those things. So if you do get the opportunity, definitely go up there and check it out. And we always appreciate it. And it just about always, almost always puts a smile on her face.

We sometimes don't get great ones, but, you know, it happens. So, yeah. All right. Well, with that, speaking of laughs, it's time for everybody's favorite new portion of the show. It's time for dad jokes. Yeah. We really should just make it a regular thing. Uh, all right. So I got this one from, uh, Jim on, uh, on, uh, Slack and, and you probably know

him more as the, uh, the design patterns evangelist. Um, but he, he shared with us a tweet that I thought was kind of humorous, and we'll see if you guys are ready for this one. Two windmills are on a date, and one asks the other, so what kind of music do you like? And the other replies, I'm a big metal fan. Awful. Love it. Awful. Love it. That's hilarious. I couldn't see it coming either. It's just great. awful love it awful love it

i couldn't see it coming either it's just great yeah so uh all right in all serious we head into my favorite portion of the show in this show. Survey says. All right. So a few episodes back, we asked just very simply, which one? And your choices were Coca-Cola, because I'd like to teach the world to sing. Or Pepsi, it's hair on fire good. And I dare to like read that coca-cola one and not have the song sing in your head it's always coca-cola that one no oh my god no the smile on your face oh i remember no neither oh we're both wrong about the pickles teach the world to sing in perfect harmony. Oh, no, no. I hated that song.

Nah, it's always Coca-Cola. No, but that's what the whole thing was because I'd like to teach the world to sing. I tried. It was literally a lyric from the song. It was so bad. It was so bad. And it had all those people.

I burned every album that had anything to do with any of those people. Wasn't the Chicago Bears on it? Oh, wait. I'm sorry. I'm thinking of We Are the World. Sorry. Different song. All right.

Okay. I think Alan went first last time. I think. I always forget. So we'll let... this will be really fun. Math of the chicken, you have two choices, sir. Pick one with a percentage. Okay.

Well, obviously it's Coke with 100%, but I know there's some shysters out there, so I'm going to go with 111. Okay. I'll have to take it. The math with chicken strikes again. Wait, Dr. Pepper is not Coke. Wait, you said

duh-duh-duh, but shouldn't it be bawk-bawk-bawk? It should. Alright, so I too also though happen to agree, and maybe it's because But shouldn't it be bawk, bawk, bawk? Yes, it should. All right. So I, too, also, though, happen to agree, and maybe it's because I live here in the south, as my California motorcycle riding dude does. It's got to be Coca-Cola. I'm going to go with 60%, even though we know that we have a very diverse crowd of people from all over the place on that.

Listen to the show. I still think that there's no denying that Coca-Cola tastes better. So it's gotta be that man. And if we didn't say Coke products and we didn't say Pepsi products, right? Like if, if, if Dr.

Pepper and diet, Dr. Pepper and Mountain Dew and all that stuff made it in there, it would be a different world. But we're talking about Coke versus Pepsi. So Coca-Cola is the clear winner at 60%. Okay. I definitely like your reasoning. I would like to know how much Coca-Cola paid you for that endorsement.

I didn't realize we were doing paid endorsements for it. They need to talk to me here after this one, please. For real. So the winner is... It's Alan. Of course it's Alan. Yeah. Yeah, namely because you didn't overshoot the percentage.

But yeah, it was 68% Coca-Cola. nice wow that was very close yeah that's seven to three man that's rough why is there so much pepsi he said it was very close where is pepsi even from there's no world of pepsi is there uh that's a good point sir you know i don't know there's got to be there's got to, wherever their headquarters is, I'm sure that they have like a world of Pepsi. And for those who are like, what are you talking about? World of Pepsi. If you've never been to Atlanta, which is the home of Coca-Cola, there is the world of Coke museum that you could go to. It is a waste of money, but totally go do it if you're here and. And you will drink some of the worst caffeinated, carbonated drinks ever known to man.

No. The best one that you definitely need to take, you know, maybe just buy as much of it as you can. Take it home. You're going to love this one. It's the Beverly. You want as much of that one as you can possibly get. And all those Italian sodas, as much as I love Italy and all the people there, y'all sodas.

Wow, this got harsh quick. Right. We just took a turn. They're not good. They're not good. By the way, Pepsi was introduced as Brad's drink in North Carolina. North Carolina? Well, they're headquartered in New York City.

Well, it's actually quite a bit north of New York, but I just want to say New York City. Oh, that was from the Coke commercial. Dude, this was also created in 1898. I didn't realize Pepsi had been around so long. 1893. Really? It says right here it was first invented by Caleb Bradham in his pharmacy in 1898. No, on Wikipedia it says it was introduced as Brad's drink in 1893 by Caleb Bradham.

And it was renamed to Pepsi in 1898. PepsiStore.com says 1898. That's what I'm saying. I'm up on PepsiStore.com. Well, nobody goes there for their information. It's Wikipedia. So, PepsiStore.com doesn't work for me because half the page is a big Adobe Flash player thing that Chrome won't show me. You can't hate on them for having outdated technology, sir.

Oh, I can. This is that A-B testing. I'm trying to see if it would work on you. Well, this is why they didn't get the 68% that Coca-Cola did. You're right. They do have Adobe Flash. How do you have Adobe Flash? Well, I only know because it's blocked.

Right? I just didn't. I just see something the other day that said, I think it was Chrome was saying that they were stopping support for it in October this year. Oh, really? Yeah, I think it's officially gone. I mean, this is what bothers me. Okay, now, like, it's bad enough that they lost out to Coca-Cola, you know,

math them a chicken quick. What would be the math? 33%. Yes, so close. With the percentage. I was looking for 32, but whatever. So, it's so bad that they lost with 32%. But then they have the wrong information on their webpage to say that it was 1898.

And they have Adobe Flash. And it's like, dude, what year is this? No, in all fairness, that was PepsiStore.com. I don't believe that's the official Pepsi. No. Oh, is that the case? You know what? You're probably right.

Yeah. Yeah. I mean, we were totally raining on your parade with an out, not a real site. So that's, you know, we were cheating. We cheated. I was like, I thought you were being serious there. But yeah. Okay.

That makes so much more sense. No, no. The Pepsi site's actually decent. So, all right. Okay. All right. What's our next survey? I apologize for everything I said, Pepsi.

But Coke Zero is the bomb. Okay. So, where did we leave off? Oh, so then we'll get into this episode survey. And hey, Joe, Recursion Joe, sent us this one. And I thought this was pretty good because it's also really timely because iOS 14 just came out. And so his idea for the survey was, hey, when a new mobile OS update comes out for iOS or Android, do you update as fast as possible? I can't get the bits, the new bits fast enough or no. Wait, why? Wait, no, that's not right. Why did you guys like, uh,

override my thing here? Oh no. Okay., sorry, I'll fix that. But so update as fast as possible or wait for the release to stabilize or just never update. All right. So I'm going to ask you a question here. This may not lead the audience. Maybe it does. Do you have iOS 14? I was going to ask you that, sir. I do. Because I was going to be really disappointed if you didn't.

Do you? You do, don't you? Of course. So we know our answers. All right. I am definitely in that, I can't get it fast enough category right yeah but but specifically though why i was curious if you had it though was because uh of the new feature with the widgets and i remember how much you enjoyed that from uh android and that was the one thing that a lot of people in it from the android camp hated about ios is the fact that it didn't have the widget capability that you could put anywhere. And now it's there with iOS 14. Yeah. Welcome to the 2010s there, Apple.

Good job. Oh, we're in 2020. That's right. It's that far. Have you been to PepsiStore.com? Nicely done. nicely done i can't stop laughing at that all right so let's go ahead and jump into the next bit that we've got here so we were this is where we are for a second no we weren't i was sorry i don't know what i'm talking about come on oh yeah why uh, this is actually longer than I thought it was.

So, yeah, we are actually going to wrap up on the second way here in this section. So, the next portion we have was create, review, and coordination processes to increase our quality of work. So, this stat blew my mind. I think because it's eight years old and even seeing the stat, it's still kind of like, wow, this is impressive. Because of GitHub's pull request system, they were able to do 12,602 deployments in 2012. How many employees do they have? That's crazy, dude. How many do they have that's crazy dude how many they have in 2012 right yeah that's 34 almost 35 a day on average and that includes weekends and everything so uh was there 180 business days a year or something like that um that's probably not right

no that's gonna be more than that that's's not at least full days, I think. I mean, just watching the math in the chicken work is just a treat all to itself. Seven divided by ten. Carry the one. Seven minus five. Divide by five. It's roughly 250. Let's just call it 250. 250, yeah.

There's like 100 days a year. You're looking for the number of business days. Couldn't you have just done like 50 times five? Well, there's holidays. That would have given you a holidays. That was taken two weeks off. Yeah.

Yeah. So anyway, uh, somehow I got, somehow I got 50 per day. That's about right. And yeah. How many employees?

I mean, obviously more than 50, but,, but 2012, I don't know, man. There's some weeks where I'm lucky to get a single pull request in a week, and usually even then it's crap. Yeah, I mean, that's truly impressive. But the important part of this that they were trying to bring out is they tried to eliminate the need for approvals from people that are not tightly or closely involved with whatever that changes. Right. So, yeah, don't don't go to some other department for approval. Don't go up the ladder to somebody who's not familiar with the code and the change that keep it close to the people that are making the changes. And then that way you're sort of guaranteed

that there's a decent level of confidence in whatever that release will be. Okay. Yeah, that's good. Yeah. And then also with smaller changes, you know, we've talked about all the goods, like smaller changes merging in every day, all sorts of stuff helps with that. And let's just keep everyone in the loop uh yeah there there was one whole thing about this section this whole section is having had

because we have already done the google review uh because we already did the episodes on related to the google engineering practices and how they conduct their, uh, poor, uh, their code reviews and, you know, for put together a poor request and all that. Like it's kind of interesting because there was portions of this chapter, not the whole chapter, but portions of it that they did, uh,

coincidentally talk a lot about Google's processes for it. And it was like, Oh yeah, yeah. I remember that. Yep. Been there. Remember? Yep. Done that. We talked for it. And it was like, oh, yeah. Yeah, I remember that. Yep. Been there. Remember?

Yep. Done that. We talked about it. Yep. Got the book about it. Yep. They do go in here and they talk. The next section is the dangers of the change approval process.

And I actually like this title a lot because they like to call out that, hey, bad deployments are usually attributed to a couple of things. Maybe there's not enough approval processes in place or there's not enough good testing procedures in place. And the irony behind that is they said, well, they've actually gone and done some studies on that. And the finding is often that these command and control environments where you have to get so many levels of sign-off and everything, those usually increase the likelihood of bad deployments for any number of reasons.

I totally believe it. It just slows the whole thing down. And, man, when you know that a change just went out and this is like the only change. There's only been two changes, now things are broke, it's pretty easy to figure out what went wrong. When it was six weeks, it includes six weeks worth of changes and something's broken now, good luck. Well, you're just like, because you know it's going to be such a hassle to do it, it's discouraging. Right. do it, it's discouraging. Like the three of us, we worked in an environment where long ago where we used to do deployments. Like it was no big thing. We would do three a day.

Like, you know, according to this, these numbers, that's not even a big deal. But, you know, for us at the time, you know, we thought we were doing pretty good because we had it, you know, automated three deployments in a day. That was no big deal. If we had to fail back, we could. We had processes for that. And then management decided they didn't like that metric. And they thought that's too many. You shouldn't be doing that. We need to restrict that. We're going to limit that. And do you remember they started like, uh, like really putting the squeeze

on us to where it was like, okay, well, you're only going to do two deployments a week. And these are the days you're going to do them. And we're going to make to where it was like, okay, where you're only going to do two deployments a week. And these are the days you're going to do them. And we're going to make sure that it was like it, the quality didn't get better because you limited how often we could deploy. No, it actually got worse, right? Because your ability to respond to it and the change size got bigger. So it was harder to test to see if, if the things that you did, because instead of testing two changes, you're now testing 50, right? And that was really, I don't know that at the time

it clicked in our heads that that's what was going on because there was also a bunch of other things, right? Like just the whole approval process and even getting set up to deploy ended up becoming another process in and of itself, right? Like you had to go find all the tickets. You had to time all this. You had to create wiki pages. Like it was insane. But looking back on it, I think probably the real problem was that you were releasing too much.

It was not easy to isolate what those changes were. So you couldn't go in and validate those quickly. Well, I definitely feel like based off of everything that we've learned over the years since then, with all the various books that we've discussed and all the practices, I feel like if we were put in that situation today, we would be able to better articulate to upper management why that would be a horrible idea. Back then, we knew it was a bad more data that you know we've we've discussed and dug into the details about it that we would be able to like say like no that's a horrible idea and here's all the reasons why we should not do that you remember why they wanted it right like they wanted to know okay tuesday something went out and now we know that there's a problem so now

we know that it happened on tuesday so really they wanted to be able to track stuff back to when changes were made. But I think now we would make the argument basically that, yeah, we need to improve our telemetry much more too, but we don't want to slow things up. Right. Know that the thing went out, but be able to track it and be able to do it multiple times a day. Look for those spikes. Look for those common things. So yeah, I mean, what we're talking about here is they were saying, these are some of the problems with overly controlling the changes.

And Joe Zach said it a second ago is you have longer lead times. That's a big problem. And then with that, they say this also reduces the strength and immediacy of the deployment process, right? Like I think you said it earlier, if you're releasing something that you worked on a month ago, your confidence in whatever that change is is way gone. You don't remember it, right? If you made that change yesterday and you're releasing it today, you're pretty confident in what you did because you were just in it and and that's really important yeah and if it does happen to go bad it's fresh on your mind like what you did and how you did it you might immediately be able to spot something yeah yep but that um that definitely bucks old

world practices where you have like a long qa cycle on planning cycle i mean everything like you really need to have the whole business aligned towards doing this. And you can imagine the value. Like, imagine 12,000 releases. Imagine having 50 releases a day. Imagine having all your developers releasing changes every day. Like, that's super powerful. If you do that, you can make changes quickly.

You know, we talked about the Intuit thing. Like, imagine if they did wait until the off-season to do their tests. And if something worked, then they weren't going to profit off it until the next year. And then by then, who knows if it's even still relevant. So I do want to call out something, and this is probably really important for people to know is this, what we're talking about works really well in certain environments,

right? Like if you're doing websites that you control, that's a really good place where you can do that, right? Cause you're controlling it. If you have software that has the ability to update itself by through some sort of, you know, polling or push updates or something, that's also good. There are situations, the three of us have worked in enterprise software where you don't have access to that. The software that is getting installed might be on boxes that don't even have internet access, right?

And so your ability to update that software is minimal at best, right? And so some of these things are really problematic, right? Like you can't even get the short feedback cycles. You can't get any of that stuff because you don't have access to the environments to where these things are deployed. So I don't think this is a silver bullet. It is worth knowing that saying that this doesn't fit every scenario, right? And that's the only reason I wanted to call it out. Is this just websites, though?

No, it's not just websites. No, that's what I'm saying. It's not just websites. Websites are good. Anything that can be updated that you have access to get those updates quickly, right? Like if you can send out patches and all that, but sometimes you don't, right? Like think about if you've got software installed at a government installation, like the Department of Defense or something, you're not getting your hands on that stuff, right? You're not going to know what's happening unless they tell you, right?

And so if you release features to them, you're not going to know if what you did worked or didn't work until you get screamed at, right? And so those situations are really frustrating, and it's kind of counter to this whole DevOps thing. This DevOps thing implies that you have the ability to update and release these changes consistently. I mean, it doesn't even have to be that extreme, though. It could go back to my iOS app store example, right?

Yeah. Like if you wanted to redeploy out to the app store, you got to go back through the approval process for that app. And depending on that process, that process can take a couple weeks. Or at least my information might be a little dated on this. It's been a minute since I've done an iOS app. But I remember that's what it was like. Maybe now it's better.

I know. And I remember, too, that for large companies, then they could buddy up with Apple. Then they could get some special, you know, whatever. The treatment. Special treatment. Right. Yeah, to like help speed that along.

But even then, you know, Apple was like, okay, we'll do it this time. But, you know, don't make this a habit. Right. If you're a video game, like I don't want to be installing multiple updates during a play session. Yeah, it's hard. So if you could do any project, if you have a new project to do, you should try to do it on the web. If for no other reason, then you can iterate quicker. Yeah, I mean, that is true.

But I mean, even talking about the video game thing, turn on Call of Duty. There's an update every time I turn that game on, right? Like every day it's like downloading updates and it's probably some sort of patch file or something, right? Yeah. But that ability to quickly turn those things around is amazing. And sometimes you just don't have that control. And that's worth saying. But going back to this thing with, you know, having too many controls in place.

When you do this, you're adding more friction many controls in place, when you do this, you're adding more friction to the deployment process, which is a problem, right? And we already talked about the reasons why. You're multiplying the number of steps in the approval process. That's always a problem. If you're increasing the batch size of the deployment or what's in the deployment, that's a problem. We talked about the testing. Your deployment lead times. That's another problem. All these things add up to really big problems. They don't seem like a whole lot until they just start stacking on top of each other.

And now you have deployments that are scary. And that's why people don't like to do it. And that's where the Phoenix project came into play. Oh, spoiler. Whoa, spoiler. Wait, wait. It was all about DevOps. I don't think it's too much of a spoiler. Whoa. And I think we could agree on this one too, right? The people closest to the items know the most about them. If you're the developer that did that change,

you're going to know the most about it. You're the subject matter expert in that area right now. It makes sense for you to be involved in whatever that process is. But they do say that the people that are further removed from it, so like, I don't know, the director of your department or somebody that's above him or her or whatever, people as they get further away from it, they don't know, the director of your department or somebody that's above him or her or whatever. People, as they get further away from it, they don't have the context of that, right? And so maybe they don't even know how they are going to go about approving it, right? And so now you're just adding people to the mix that aren't as close to it, and they're not going to be as quick to get in there and approve that stuff.

They might not even understand what it is you're trying to approve. Let's be honest. They might be so far removed from it that you could try to describe it and it won't make sense. But there's also that thing too. As you add more people to that approval process, the communication starts

to get exponential in trying to, trying to, uh, involve everybody and make sure everybody understands. Yeah. Yeah. Yeah. And the benefits are dual.

It's like one, the person who's approving you probably already knows that this change is happening or knows that the larger feature that you're, you're working towards. And also it keeps them in the loop is to the change too. So it's like, not only are you benefit by getting their experience size, but also you're letting them know about a feature they're close to.

That's great. Yeah. You know, one of our friends, I think, uh, John, he was on the show back in episode 100.

I think I remember him at one point saying communication doesn't scale. Yeah. Right. And that's so true. Once you start involving a bunch of people, the meetings start getting set up and people are going to start discussing things that they don't necessarily understand or don't know all the intricate parts about. And it turns into just a big downhill battle at that or uphill battle here that I really liked is they said, as the distance from the person who actually does the work and is familiar with it increases, so does the likelihood of failure. I think that's pretty accurate.

Yeah. So, yeah, I'm sold. Yeah, they also said, too, though, that organizations that rely on change approvals have worse stability and throughput through their IT systems. Isn't that crazy? Yeah. Yeah, you would think it would be the opposite, but yeah. Yeah, I mean – Even with those orgs making changes slower and getting less work done, they're still less stable.

And the real thing there, though – I mean, we've experienced it firsthand, is that the managers that would want to put that process in place, they think they're doing they have good, good intentions. And they think that it's it's the right thing to do. But that it ends up, according to, you know, the numbers, it ends up actually working against them. Yeah. And the gist of it is they're really just trying to get to the point that peer reviews are probably way more effective than the outside things, right? Involve the people. And this kind of goes back to what you talked about a minute ago, Outlaw, with the whole Google review processes. There are certain subject matter experts that have to approve a particular thing before that thing can get merged in, right? Because, you know, outlaw owns the order system. Joe Zach owns the back office system.

If I'm going to make a change to one of those, I got to go to you to get that approval. I got to go to him to get that approval. And that's, that's what it boils down to. So that's, I think it's really strong stuff and it'd be hard for an organization to swallow that pill. If they, if they've always been in the,

in the world where, you know, management is the one making those decisions, it's going to be hard for them to steer away from that. Yeah, totally. One other thing they mentioned here, which is something we've talked about many times from many different angles

is the more loosely coupled our architecture, the less we have to communicate between teams. And in case you're new to the show or haven't listened to all 142 of the backups, when we talk about loose coupling, we basically mean the ability for our code to work in separated modules. So they don't have hard dependencies on the code next door. They communicate through some sort of interface, whether it has a little eye or, you know,

or it's just some other form of contract there. But it just lets us keep stuff more modular. So we can swap things in and out and work on them in isolation and make changes in isolation. And that can happen at the big level, like architecture between, say, services. And it can happen at a small kind of code level where you're just kind of keeping changes in separate files or in

separate modules or classes in order to make those classes focus on single responsibility that makes them easier to change when only that one thing actually forces them to change. I'm sorry. I can't not comment on it because Mathemachicken strikes again.

Oh, what'd I do? If you're listening to this episode, there's only 141 back episodes to listen to. Well, no, there's the one episode that never aired. Remember? Remember? I still have it. Don't make me release it. Did it really?

Do we have one? No. Yeah. Was it number two? No. It was an episode that we recorded and then the next day we were all like, man, that sucks. Oh, we did. That was one of our original three, I think.

And we were like, no, we can't do that. And so we scrapped it and we did it again. Yeah, that was painful. I still have it. You don't believe me. I'm going to release it on Twitter. It's uploaded to TikTok right now. Yeah, we're going to have to rename him to the threat of a chicken. That's right.

All right. So back on this one real quick. One of the things that they say here is this loose coupling that he was just talking about that's so important is it allows you to make changes in an autonomous way, right? Because like he said, you're coding to a contract. So you don't have to go talk. I don't have to say, hey, Joe, what does this method do? Or what method do I call to do this? You have an API essentially that you're talking to. And as long as that API is there, you write your code, right? And that's super important. They do say, though,

that this doesn't mean that you could just throw away communication, right? There are going to be situations where you have to talk to people. And, you know, that's important. Just know that you're not going to eliminate that. And they said this is especially true when you have overarching, like, infrastructure type changes, right? Somebody's going to upgrade the database version or somebody wants to do something crazy like that. But, I mean, if we've learned anything from clean architecture, though, you shouldn't have overarching infrastructure. What would that mean? The database example seems like not such a good fit right so then it's like what would overarching mean that could possibly be a problem in that

regard then well no database is probably a good one right like no because the applicant no because because you know if you i should be providing like you shouldn't have access to my database directly right like i should have some i should front that with something and and you're going I should be providing like you shouldn't have access to my database directly. Right. Like I should have some I should front that with something. And you're going through that thing so that when I make changes to my database, you don't know anything about it. But OK, so that's fine and all. But let's say that you're the application team and you rely on an ORM. Right.

So you've created that abstraction and you're good with that. But that doesn't mean that if the database is updated, that all of a sudden you're not going to have problems, right? Like certain functionality that was being called in a proc or something somewhere that your ORM wasn't directly doing anything with, but that stuff might break. I would argue that that's not, if the separation is just the ORM between your application and the database, then that stuff might break. I would argue that that's not, uh, if, if, if the separation is just the ORM between your application and the database, then you're still tightly coupled. Like,

you know, if you wanted to truly have a clean architecture, then, uh, you know, you wouldn't know about my app, your, your application wouldn't know about my database. I would front that data to you

some way so that you wouldn't, you shouldn't have your tentacles into it, right? You're talking about your own database. No, no, but you're talking about two steps removed. I'm talking about you're an application team, the infrastructure team, the ops team wants to upgrade the database server.

You need to know about that. Well, they're talking about overarching infrastructure here. If you're saying your application and the database are part of the same thing, then that's not overarching. That's part of the same application at that point.

I see what you're saying. Yeah, that's part of the same thing. Overarching would be like, Hey, uh, we are all in the same Kubernetes cluster together and all of our stuff is deployed in the same namespace together. And now we want to like upgrade the nodes and,

but hopefully, you know, Kubernetes is abstracted enough to where you don't care about that. Like your application shouldn't care about that unless maybe you were doing some, maybe you had some applications that were making calls out to the Kubernetes API. And then maybe that's why an upgrade to Kubernetes would be an overarching infrastructure change. So maybe an upgrade to your OS that you're running on would be probably another one too because org say everyone we're moving right ubuntu 19 there you go yeah something like that okay that's fair yeah we no longer support encryption rsa 256 everywhere is going to 2048

yeah there you go those are those are in my opinion that would count as overarching yeah i like that better yeah and what you're saying is you shouldn't have multiple apps that are talking directly to the database because you're breaking something there. But the alternative is you have one service layer that fronts the technology rather than multiple services fronting by feature. So, I don't know. It's hard. Yeah, either way you slice it up. There's always problems, but your point is well taken, right? You should be abstracting your data storage layer somehow if other people are going to be talking and using that functionality. And the people you find yourself talking with are probably the services that you're coupled to.

That's a good way to kind of converse. You can kind of flip around. It's like if I'm in meetings all the time, it's because my stuff is too coupled. Why are we in meetings all the time crap well i mean we did learn that uh last episode joe sends out a lot of meeting invites that was one takeaway he doesn't like to be in them but he likes to send them yeah i mean yeah i don't start the meetings and yeah sorry about that everybody so i will say one thing about this particular section of of the second way is they repeat themselves a whole lot so we'll try and blow through some of it like so this next

section was enable peer reviews of changes we've sort of already said that you're close to the code you should be the one doing it right small changes are easier yeah now this one we've even talked about from the Google engineering practices. Yep. Now this was one thing that I actually liked though is, and we've probably said this in a number of different ways in the past, but the size of the change is not linear with the risk of the change. Meaning it's not,

it's not the straight line going up at an angle, right? Like as the size of the change increases, that risk level goes up tenfold, right? It could be like a logarithmic growth on the risk. Yeah, it's big. So that's super important. Because they also said too, in this portion, where they said that if you give another developer 10 lines of code to peer review, he's going to find 10 mistakes or 10 issues. If you give a developer to review your code, if you give him 10,000 lines of code, he's going to say it looks

good. Exactly. So that goes to your point that the size of the change is not linear in regards to the risk because the larger that change is the more likely it will not if especially if this is like a habit for your organization the or your team rather uh the chances of that code actually being given a decent review and a thorough review is probably little to none. Right. Right. Like it's just going to be rubber stamped. Yeah.

Yeah. I think this one's one of your favorites here. Short-lived branches. And we've talked about a few times about how I'm having like long live feature branches optimized for individual productivity or small team productivity at the expense of of larger integrative productivity. And also, if you've got long-lived branches, you're delaying these integration periods,

which means waiting on the errors to show up later. And as we said over and over again, it's all the stuff that ties together. The sooner you can find the problem, the better. For scarier changes. Yeah, go ahead. More than one reviewer for scarier changes which makes sense i've seen um some teams will do things like if it's got this file extension then it needs to have this kind of person i i don't love that but i mean i i kind of get get the intent there is to

make sure that enough eyes get on it well i think it was part of the google engineering practices too where they talked about like depending on the language they might have like one uh you know code reviewer that's just specific to the language right not even necessarily specific to that area but to the language but i think it was uh also this portion of the book too where all of it's a blur now so so I don't even know where they talked about, you know, doing things like plus the number of people that you wanted to review. So like in your pull request, you could kind of like indicate how many people you wanted to have review the code. And, and you know, you could see where like for the scarier changes, that would be a nice practice to have with your team.

Yep. I really like this next section, which ties into the same thing. Basically, the gist is the more manual testing you do, the slower you are to release. And the larger your batch size, the slower you are to release and the larger your batch size the slower you are to release so if you are an organization that is doing a lot of manual testing whether it's you the developer or even a qa team that's got some steps to go through to kind of certify something and you have a large batch size because you have periodic releases then what we're saying here is that you're you're not going to have very many releases and what we've kind of said in addition to that,

based on the stats and stuff that we're reading in the book, is that the slower your release is, the more error prone. So if you're working at an org that has a lot of manual testing and big batch sizes and infrequent releases, just think about how that's working out for you. If it doesn't seem to be working out for you, then, hey, you should leave a comment on this episode and maybe win the book.

There's that. If it's working out great for you then cool i mean you know going back to the start of this right with githubs uh twelve thousand twelve and a half thousand uh deployments that they were doing or even like remember google's it was like hundreds of thousands that they were doing right like you you're, you're not, uh, you know, they're able to do that because they don't have, you know, manual testing of, uh, large chunks of code, right? It's just little bitty things here and there. And they're like, yep, that little bit works and the rest of it hasn't changed. So it's safe. Wait,

did we ever actually figure out there's a correlation between more changes and more money? Because really what I want to do is I just want to do something once and then just make the money from then on. So I don't want to make a lot of changes. I just want to make that paper. That's what we've been doing wrong. I think I'm reading the wrong book here. We need to do the one thing but just do it a bunch of times. Heck yeah, I'm going to start book here. We need to do the one thing, but just do it a bunch of times. Heck yeah.

I'm going to start a lawn business. A landscape. Why does it seem like every one of those people drive like $90,000 trucks around? I don't understand. They do in Florida. That's for sure. Yeah. I don't get it.

All right. So the next ones we have here is they talk about enabling pair programming to improve all changes. And this one, you know, kind of depends on what side of the fence you land on here. But Jeff Atwood actually had a really good quote that they had in here. Pair programming is code reviews on steroids. And that kind of makes sense because you constantly have somebody looking over your shoulder. So you're catching things before they're ever done wrong, right? Like, hey, Mike, you typed in the wrong thing.

Or, hey, Alan, that's the wrong method call. Whatever, right? Like, that's kind of cool. Well, they also talked about, like, you know, different implementations or, you know, of pair programming where, like, one would have to, like, write the code, one developer would write the code, another developer would write the unit test, or, you know, one's just constantly looking over your shoulder, like in the example, the way you described it.

You know, I mean, it... I would like to be in an environment doing pair programming, because it sounds so like there's so many benefits to it, right? From everything that we've ever heard. But I, I just,

unfortunately I've never been in a pair programming kind of organization. You guys open to it? We've paired on a problem. Yeah. I was going to say, we sure that, but that's as, does that count?

So I'll tell you as much as I do like pairing when there's something, like maybe you're just getting your project started or whatever. I like that sort of thing and certain kind of touch points. But the idea of spending all day in a code review on steroids sounds terrible to me. I've heard good things about it. I don't know anyone who's who's done it all day and said it was awful oh yeah it sounds awful really have um will will actually oh i thought

he liked it he did like it oh yeah you said awful no he liked it yeah everyone who's actually done it said yeah it was good uh they don't seem to rush to go do it again but this was good that's a good point yeah yeah i don't i don't know man i do like what they said here though is they said that having that pair programming forces communication that may never have happened right like yeah for sure ideas will come out that that maybe you wouldn't have had you know somebody else just just having two different perspectives on it i know we've talked about this back in the day but one of the things they found with programmers is if you get people like diverse backgrounds,

you know, people from all over the world, different walks of life, whatever people solve problems differently because they've experienced life differently. And so it adds to the creativity when it goes in. And that's pretty awesome. That's actually one of the things I like about being a programmer is the

creativity. Yeah. Do you keep your ketchup in the refrigerator or not? Right. I don't. Right. Yeah. It's so much better that way.

It really is. I don't like ketchup, so I don't care where you put it. What the? What? What? There's my crazy comment for the night, and I'm done. The only argument there is Heinz or Hunt, and that shouldn't even be a question either.

That should be like Coca-Cola or Pepsi, really. But we won't go there right now. I don't know. I can't explain it. Yeah, so the last thing they said here is also by doing this pair programming, you reduce those bottlenecks of these code reviews, right? Because a lot of times the code review will sit out there and, you know, one of two things happens with that one that has 500 lines of changes. Either that thing sits around for days or it gets rubber stamped immediately, right? And both of those things

are bad. I would be curious though, like while you, dear listener, are going to the website to enter your chance to win a copy of the book, if you are in a pair programming organization, I'd be curious to know, do you constantly rotate from one project to the next who you pair up with? Or are you always paired up with the same person every time? Because it would seem like if you did that, then you could see some pros and cons to it, right? If you were always paired up with the same person, then eventually it would be kind of like hive mind. This is the way we do it, and we're just always going to do it this way. And we're not going to question status quo because it worked for us the last time. Versus what you were saying, Alan.

Like if you were always with somebody new, like from one project to the next, then, you know, you're kind of like learning like, oh, hey, that was a great idea. Yeah. Oh, that's what you guys did on the other team. That sounds pretty cool. Yeah. Well, here's how we did it. And then you kind of like, you know, mix the peanut butter and the chocolate together and out comes something amazing or ketchup mustard yeah you know why not catch up maybe mayonnaise and mustard um so the next section here hey sorry we're actually close to the end here. So they said you should also evaluate the effectiveness of the pull request via processes.

So look at your production outages and see if you can tie them back to a peer review. What did you miss? Did you have a peer review? Did you have a peer review? That's one. Another one that I thought was really good is they talked about the fact that sometimes you'll have a pull request in and people just put a ticket number. And there's no context, right? Like, what am I supposed to be looking at here?

And they say that as simple as that is, that can really add to the effectiveness of being able to do that peer review. Providing sufficient detail on why the change is being made. I think I'd mentioned that I'd done a pull request into a public project, the hoodie project, right? And they actually had a template laid out that was, you have to fill these things in, right? And there was like, you know, six or seven bullet points. And I kind of liked that because it made you think about what you were doing and you had to tell them why.

So it was very clear and concise and consistent for everybody that looked at that. Yeah. You ever look at one of those that's got like 13 things in it and you're like, oh, what if I just delete a couple? I wonder if they'll notice. I don't want to type all this. It is pretty consistent

though on like some of the big projects. I know a lot of the kubernetes ones you know you go looking at their their pull requests and they're all templatized like that like when you're looking at the the well i guess i'm thinking more of like the the feature request more than the pull request though a similar type thing though right i mean yeah you know fill in the os fill in whatever. Yeah. So some of the other things were how the change was made. And if you were aware of any risks that are associated with that particular change, you should probably put that in the pull request as well.

And the very last thing that we have here is fearlessly cut bureaucratic processes. Can I get a name in? Yeah. Right. Yeah, for sure. Yeah.

Then just keep it, keep it fast. Keep it light. Boom, boom, boom. Just like these episodes. Yeah.

The goal should be to reduce the amount of outside approvals, uh, then amount of Joe's meetings and the number of Joe's meetings, and the number of sign-offs that are needed. That's right. Which really, if you think about it, if you're doing really small deployments anyways, because you're just deploying like, hey, this one little change, then it's like, how many people do you really need for that, right? Nobody cares. If all you're doing is you're just like, now the logo's on fire, how many people need to approve that? Right. I think that's the important thing, right? If everybody can buy into that mindset, then people just won't worry about it because

there's not a lot to worry about. Oh, we're just releasing this one fix. We've moved the logo three pixels to the right. We need to go all the way up to the CEO to make sure that this is approved, right? Really? Wouldn't happen. But if you have a month's worth of backlog things, that's a different deal, right? So, yeah. All right. Well, we will have links in the resources we like section for this episode. Of course, we're going to have the DevOps handbook and the Phoenix Project and the Unicorn Project and others. And with that, we head into Alan's favorite portion of the show. It's the tip of the week.

Yeah, you know, we really need to turn this into plural. It's the tips of the week. I got two. I think really for you, Alan, this is the tip of last week. Yeah, this is plural. It's the tips of the week. I got two. I think really, for you, Alan, this is the tip of last week. Yeah, this is unfortunate. So I don't know if you guys, because I haven't had a chance to listen to the episode yet. Last week, I had a bit

of an emergency. My youngest just started yakking all over the place. What a great way to put that. It truly was. It was not pleasant. And a little bit about how the sausage is made here. We're always recording late at night. It's going on 11 p.m. Or no, it's after 11 p.m. right now.

Last time it was well after that. So I'm cleaning that stuff up for an hour. It was not fun. So anyways. All right. So on to my tips of the week. Your dated tips of the week, we should, on to my tips of the week. Your dated tips of the week, we should say. My dated tips of the week.

These are carryovers from last episode, some of them. The first one is the Costco credit card. Now, look, Outlaw makes fun of me because I'm a bit of a Costco fanatic. A bit? I love Costco. A bit. Hey, Joe Zach, I believe you are too, right?

Like you like Costco. I like it. I like it. The meat. The meat is good. And the vegetables. The meat. The fruit.

Oh, and other stuff. Yeah. I just bought a pressure washer from there. See? That's what I'm saying, man. The retirement policy, yo. Okay. Do you ever just find yourself like, do you go to Costco for lunch?

I used to. Yes. See? That's what I'm saying. There's a difference there because you lunch i used to yes that's see that's what i'm saying there's a difference there because that's the line that's the line no man joe's like i like it you know they got some good stuff they got but but alan's like uh-uh no i want the credit card i want to go there for lunch it's all right so so in fairness in, the reason why the credit card is such a big deal, and this is – I actually hem and hawed on this because I hate having credit cards all over the place, right? That's why I can't believe you're giving a credit card as a tip of the week.

Here's why it's so important. To not use it? Don't get it? No, no. You want it, and this is why. So there's a few different things that matter here is so first off i'm recommending this because i used to my amex used to be like where i'd buy tvs or whatever because the amex used to toss in two extra years of warranty on top of the manufacturer's warranty

so just by buying something with your amex you basically got three years of coverage on any device you bought. They killed that at the beginning of 2020. Amex is no longer doing that on a vast majority of its cards. Costco does. So just by having the Costco card, if you decide to go buy a TV and it doesn't have to be from Costco. It could be from anywhere or a device. If you buy it, you get two extra years of warranty. That's huge. So that's cool. Is Costco even overseas though?

I don't know. So, all right. So let me get into some of the other reasons why this is important. Okay. So when I traveled over to London, I did the speaking at, I can't even think of the name. NDC. NDC.

Thank you. NDC London. Everything I bought over there, I used my card, right? I paid so many international fees for transactions that, I mean, by the time I got back, I probably had $100 in fees, which isn't a ton, but it's kind of annoying. Your Costco credit card, anywhere in the world, you use it, you don't pay international fees. So if you travel, and actually one of our friends that we work with is the one who told me about this. And I was like, oh, that's interesting. And then on top of it, it's a cash back card. So if you go buy gas anywhere,

you get up to 4% cash back on gas purchases. It doesn't have to be at Costco gas stations. It has to be at gas stations. So up to a certain amount of dollars per year, you get that cash back. So the reason I bring up the Costco credit card here is because if you're buying electronics and we're all developers, we buy electronics. We do.

It's a fantastic rewards card in that you're getting money back in February every year based on your purchases. Now you do have to be a Costco member. So if you aren't, go find you a Costco and enjoy and revel in the beauty that Costco is. But it is truly like a fantastic card for all around usage. So like I'm trying to switch almost everything I've got over to it because it's like, hey, if I can get money back for the stuff I'm spending and I can get extra warranty without having to pay for those squares. What is it? Squarespace or whatever. Not Squarespace, Square Trade or whatever these these plans are that everybody's pushing on you every time you buy something. You get it for free so yeah

all right so that was that one that's totally not tech or cody related but i had to throw it out there because i was super excited when i found out about all this stuff i really just think you were looking for another reason to uh express your love of costco oh man it's really that's really all this is about. Look, let me give you a perfect example. So Joe's Act said to me, right? I got to tell you this because this absolutely just hurts me to my soul. If you go to a

local butcher shop here to get prime filet or prime ribeye, it's $25 a pound. At Costco, that same prime filet or prime ribeye, it's $25 a pound at Costco. That same prime filet is 1399 a pound, dude. It's better there too.

And it is better. It's better. It is. So like that alone, I could drop a fortune on meat at that store alone and be happy for the rest of my life, except for my cholesterol. So now I can't, but whatever. love it so maybe too much maybe that's the problem i think maybe i think maybe but yeah do you ever realize that you have a costco addiction though that maybe like

maybe there's like an anonymous a costco anonymous that you need to go i would totally sit in on those meetings. I have a problem. There's no question. So, so yeah, man, I mean, I find myself buying stuff there just cause it's there. I'm like,

it's gotta be good. I mean, Costco is carrying it. I mean, for those listening that don't quite still fully grasp the love affair that Alan has with Costco. I mean, he will go to Costco,

not because he needs to get anything. Like he'll go to Costco just to pass the time, just to see if there's anything that he might want. He's not wrong, man. That's where I get my exercise. I don't know. I don't know what the problem is. All right. Okay. So see, so I spent so long on that one. I'll try and blast through these other ones. So one that was really cool is Zach Braddy. We don't love him. He was for a minute the reactionary. Wait, is he not anymore?

Now he's doing Tabs and Spaces podcast. Definitely go check that out. That's fun. But he had a great tip so we were talking about uh you know emacs versus them and all that kind of stuff we've talked about it several times he said that his go-to editor now is emacs because there's an emacs doom thing for it and he said it's the best thing ever so I'll have a link in the show notes for that. So definitely check that out.

And then... Wait, wait, wait. But what is it though? When you say a doomed thing... It's like a full IDE kind of built in. Yeah. I mean, I think thing is the right way to put it. I was looking at it. It looks kind of like...

It's like Nerdtree. It looks like it's Nerdtree built in like a VI plus Nerdtree tree but it's also got like multiple windows that you have open according to the studio code but in emacs also with a really nice color scheme yeah yeah okay there you go visual studio it looks pretty cool i i was hoping that you were going to say that uh it was emacs and when you when the boss isn't looking you could play doom and then when it looking, you could play Doom. And then when it comes back, you could be like,

oh, flip the screen, and you look super smart because you're using Emacs. Well, check this out. Here's one thing. Doom is comprised of 150 optional modules. So apparently they've tricked this thing out to be whatever developers dream, and he swears by it. So, so, uh, this is what happens to Emacs after exhibit comes by.

All right. So the next one I have, and this one's kind of cool. I don't know that it's absolutely necessary, but it's pretty neat. So I believe Christoph Wieg and Raymond Gasper were talking about this in our Slack channel. And there is, we've talked about editor config in the past, which I think is super important. If you want to enforce consistency for developers in terms of how their code formats, tabs or spaces, all that garbage, editor config is a great way to do it because most of the big IDs out there support it out of the box. So all you have to do is commit your editor config file and everybody will be on the same page, right? Well, if you've never set up an editor config because you don't want to have

to go through the hassle of saying, I want tabs versus spaces, there is a Microsoft plugin for Microsoft. Plug in for Visual Studio that will actually inspect your code and create the editor config based off your code. So that's kind of cool, right? Like if you've always had tabs in there, then my guess is it's going to say, hey, prefer tabs. And this is how everything's going to happen. So it will generate your editor config for you. Really cool stuff. And then the very last thing, this has been driving me crazy. So I use OhMyZSH in Mac for work. And I don't know if it's that that started doing this or something else. But you know how like typically you'll type in some sort of command, like let's say kubectl, right? And you hit enter and it'll give you a list of all the stuff that's supposed

to happen. Like, right. All the, all the commands, you can do all the arguments for whatever reason on mine. It does it in a less way, you know, like where it'll give you a page of it and you can hit the space bar and it'll go to the next page and you hit the space bar, go to the next page, whatever. Right. So that's all kind of nice nice because it didn't blow all by me but what drove me crazy is when i'd hit the q button to exit out of it all the command information was gone and i was like well doggone it now i gotta type it again because i can't remember what it was i need to see it on the screen right so I'd find myself going back and forth five and six times like, oh my God, like what was this? So there is a trick that I found out about. So if there is something in your shell that is sort of kind of calling less behind the scenes, which is

what this was really doing, you can, and I've got a stack overflow to it, man. I actually need to vote this guy or gal, whoever it was. There is a way you can basically export your less command. So you export less equal, and then in quotes, do dash X capital X lowercase R and then close quotes and hit enter. And what will happen is it'll still use less when you go to do your command but when you hit q to exit it'll leave whatever was on the screen right there so you can see it so that i swear to you as small of a change as that is that made me five times more efficient in just everyday things okay i gotta ask you a question. Uh-huh.

This is probably going to irritate me because you're probably going to tell me something so stupid and simple that I'm going to be like... Wait for it. Okay. Because I gotta know. I mean, the enthusiasm in your voice behind this less

trick that you've got here. Do you like this better or costco oh costco you gotta be good at math to know that was gonna the bathman chicken got it man come on hey, the chances are you can stack up most things in this world against my love for Costco, and it's probably going to win the vast majority of the time. Okay. I mean, I've been waiting for them to get a good mountain bike.

I'm just saying. Come on, Costco, get one in stock. All right. Well, my turn. And I've got two tips this time. So one is for everybody, and the other is for outlaw. And I've got two tips this time. So one is for everybody and the other is for Outlaw. So I'm very excited about this. I get my very own.

Our friends over from – we had some friends on the Rollback podcast a couple years ago. They ended up kind of splitting up and doing different things. And now two of them are on a new podcast called head in the clouds which is a i would say uh a cloud development focused podcast but it's from uh and i remember saying this about the the rollback last time it's not from uh it's more from a geez i don't even know how to say it like a devops is maybe i would say or like a network admin or a sysadmin or just a different perspective from what I'm used to. So the people on the podcast come from backgrounds like in security or network engineering. These basically came from a different path to get to kind of the same place as a lot of things that we're talking about, the DevOps.

So it's a really good podcast. It's really refreshing for me because they're talking about the things I know and really trying to learn a lot about right now, but it just comes from a totally different way of thinking about things. So it's been really cool and really refreshing. And it's a newer podcast that just came out. I think we've got like maybe 10 episodes or so. And so you got to go check it out. And we'll have a link in the show notes.

Now, number two. Did you know that GitHub had a CLI? I did not. Okay. So it just hit version 1.0. And my initial thought was that's the dumbest thing ever because, like, I mean, Git is a CLI. Like, what the heck?

Why would I ever need GitHub? So let me tell you what I've just been doing uh while we've uh recording the podcast uh you can create repositories from the command line with this tool uh you can create issues list issues assign issues uh you can um pr create pull requests you can review pull requests which will automatically check out the branch that's associated with that pull request and go through those changes looking at this and approving the pull request and merging the pull request all from the command line and of course it works with all the git stuff so if you're using github or github enterprise and by the way it

prompts you for which one you're using when you log in uh which is really nice and you do a gh off you can do everything you need to do to interact with github through command line so you never have to go to the website oh this is pretty slick maybe i shouldn't say everything i should just lock the github actions lots of fancy stuff settings that you can't do but yeah it's actually super easy and it's um it's kind of reminds me of uh i guess it's kind of how a lot of modern clis are doing now where it's like basically gh and then it'll be like down to like issue and then verb so gh issue list gh issue create so after like two seconds i was able to do everything i was trying to do without even looking at the help. Where's the do I just app get install

GH? If you're on Linux, I suppose so. It's also brew. I use scoop for Windows scoop install GH. All right. That is my tip. Thank you. Yeah.

I think you're going to love that. Uh-huh. Just to be able to do everything from command line. Ooh, they got in on Chocolaty, too. Choco install. That's right. For me, it's not. I don't hugely care about it, but it has been annoying.

Whenever I create a new repo the first time, I'll make some new code. Like, okay, cool. I want to save it. So I'll do git init. Okay, now I got to go to GitHub. I got to create a repository. And then I got to go run the thing to set the origin. Like, no, no. I want to save it. So I'll do git init. Okay, now I got to go to GitHub. I'm going to create a repository. And then I got to go run the thing to set the origin. Like, no, no.

No, man. You're burying the lead here. The pull request. It's all about not having to go to the website so you can do the pull request. Well, in GitHub, I do less pull requests than I do more creating repositories and then abandoning them. Well, okay, fine. Your code goes to die. But, okay, let's pretend we weren't talking

about your use of GitHub, though. Individual, yeah. If you're an organization, it's huge. Yeah. Yeah, I mean, yeah. Okay, thank you. That one, I am very happy. And that just was announced today.

Yeah. But, bing, look who's on top of things yeah yeah man joe zach is i got some good tips of the week this time that or costco definitely this github cli no come on man what about you joe oh man uh oh geez come on it's costco i mean i just got a pressure washer it's costco you don't need to go back to costco anytime soon so it's github cli have you seen their salmon you cannot get better salmon not the sockeye don't get the sockeye but the regular whatever salmon yeah the atlantic salmon yeah don't get the sockeye don't get the sock eye from anywhere. It's got bones. All right, I can't take you serious right now.

Clearly, the right answer is the GitHub CLI, sir. Okay. Well, then it's time for my tip. Or unless, were you done? Because you did say you had more than one. Are you even going to bother? I mean, I should have gone last. I'm sorry.

I don't know how I follow that up. I mean, that is a big one for sure, for sure. So I will share this one, which is definitely not from today. This is a couple years old. But if you're not using WSL2 yet, if you're still forced to use the other one, then install Ubuntu and set that up to where you can use your Docker and Kubernetes and Helm all within Ubuntu and everything work fine. Because if you've ever noticed, it's kind of annoying that some of the commands are, there's, there, some of the commands are more significantly faster in the POSIX environment than they are in Windows.

I don't know why that is, but it's observable. You can see it. And some of the commands you'll be like, I don't care. I really don't care. Like, okay, fine. It saved me a whole second. But then some of them, like some of the helm commands would be like, you know, a six second difference, you know?

And it's like, you're like outlawed six seconds. But, but if you're doing scaffold, for example, and you're scaffolding up an environment, that's nothing but helm commands. And, you know, you could see it take, you know, a longer period of time to do all the Docker builds, then all the helm commands, uh, you know, necessary to get that environment up and going. So it can matter. So where am I going with all this is there's an article on medium that talks about how to configure your, uh, your WSL environment so that you can bind the Kubernetes and Docker instance inside of Ubuntu to that of Windows so that it's the same, it's sharing the same thing, the same instance. And no matter where you run the commands, you're going to have the same stuff available to you.

But you can get all the benefit, the speed benefit of running it through Ubuntu instead. So I'll have a link to that article. And then I kind of wanted to mention this one again. We mentioned this one back in episode 107, the kubectl cheat sheet. And I couldn't remember if at the time the reason why we mentioned – I think we were more mentioning it, if I remember correctly, cheat sheet are a couple of lines for you to use to where you can get the bash completion of not the kubectl letters itself, but of the other things. So like if you typed in kubectl de and then you hit tab, it would finish out describe. Or more if you did like kubectl pod and then you started to type in the pod name and then you hit tab and it finishes out the pod name for you, right? I don't know that we really called that portion to attention last time. So I wanted to highlight that here.

And that's a game changer. Not having to type in that hash at the end of a pod name. Yeah. It's awful. Yeah, it's awful. Or as I would do, KubeCuddle get pods. Okay, copy paste KubeCuddle describe pod paste. So yeah, total game changer. Like Alan said, yeah, you should,

you should take advantage of that. Uh, so with that, uh, I think we finished up the second way. Finally, we finished up the second way. Uh, and yeah, with that, you know, subscribe to us on iTunes, Spotify, Stitcher more using your favorite podcast app. Uh, as Alan mentioned earlier, uh, we would love to hear, uh, any of your feedback and,

and read your reviews. We do greatly appreciate those and, and really enjoy reading them. They put a smile on our face. You can find some helpful links at www.codingblocks.net slash review. Oh, and I see, uh,

while you're at codingblocks.webs, and I see, uh, while you're at coding blocks, dot website, dot whatever, uh, check out our show notes and examples in the discussion and the other stuff that's there. Okay. And,

uh, wait, wait, something just happened there. Uh, yeah. While Joe didn't want to have to talk anymore. So Joe's like,

here, let me make this over there on the desk. Um, yeah. Well, ever since you told him that you could just bell talk anymore, so Joe's like, here, let me make this all for him. He's asleep over there on the desk. Yeah. Well, ever since you told him that you could just bell midway through, he's like, get it up, done, I'm out. Yeah. If you have any questions, comments, or whatever, you can send those to our Slack channel. Definitely go over to codingblocks.net slash Slack and get on there and interact. Good stuff.

And this is you, joe make sure to follow us on twitter at codingbox or head over to codingbox.net we can find all our social links at the top of the page and uh you could also uh somewhere buried hidden on this website is a link to the spreadsheet where gremlins have gotten in and are changing the names on who has to say what. It's awesome. I don't know who it is. That's weird. Must be on the website. Totally weird.