Coding Blocks - The DevOps Handbook – Enabling Safe Deployments

Episode Date: August 31, 2020

We learn the secrets of a safe deployment practice while continuing to study The DevOps Handbook as Joe is a cartwheeling acrobat, Michael is not, and Allen is hurting, so much.

Hi, you're listening to Coding Blocks, episode 140. Subscribe to us and leave us a review on iTunes or Spotify or Stitcher or wherever you prefer to find your podcasts. And visit us at codingblocks.net where you can find show notes and examples and discussion and more and you can send your feedback questions and rants to comments at coding blocks dot net now that i have a slight case of anxiety you may also follow us on twitter at coding blocks or head to www.codingblocks.net and find all our social links there at the top of the page with that i am alan underwood i am so confused and i am michael outlaw oh my god i'm sweating man we're doing the whole show like this right And I am Michael Outlaw. Oh, my God. I'm sweating, man.

We're doing the whole show like this, right? All right. Anxiety gone. This episode is sponsored by Datadog, the cloud-scale monitoring and analytics platform for end-to-end visibility into modern applications. And SecureCodeWarrior. Build your security posture and defend your organization from cybersecurity threats by empowering your developers with the skills and expertise they need to write secure code from the start. Okay. So, uh, yeah, obviously we're having a little bit of fun and, uh, uh, if you haven't already guessed, uh, we should probably say something too, along the lines of like this episode is sponsored by NPR and no, just kidding. Uh, yeah. So we're going to be talking about the DevOps Handbook still.

There's just still so much to learn from this book. And before that, hey, well, I guess there's no reviews, really, since the way we ended up recording this one was, like, really close to when we recorded the last episode. So, yeah, not a lot of fun there, unless you had some reason why you put in the crying emoji no it wasn't that yeah the sad face is hurt yeah that's all okay yeah so we do have one little piece of news and it's not really news but it was uh a little i'm gonna call an

article of just a little thing that micro G posted up on, on our Slack channel. And it's really good. I mean, we've talked about this stuff before, but it's really just what this guy called Rob Pike's five rules of programming. And so we'll have a link up there.

Basically in a nutshell, it's don't prematurely optimize, right? And don't prematurely optimize, right? And don't get clever with your algorithms. The more clever you try and get, the harder they are to reason about, the easier they are to break. So I thought it was really good. It's truly just a text page on the Internet, an HTML file. But, you know, go check that out.

It's a really quick read, and you'll probably be nodding your head the whole time you're reading these five things. You know, Rob Pike, by the way, the guy behind Go. Really? Going, yep. Really cool. Didn't know that. Are we not going to discuss the five rules ever?

Do we want to? We can. I mean, we got no reviews. We got no reviews. We might as well do it i feel like we could get through them really quick like i'll start us off with the rule number one you can't tell where a program is going to spend its time bottlenecks occur in surprising places so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is you know that reminds me of a really long tangent I'd like to go on. That never happens.

Right. Rule number two, measure. Don't tune for speed until you've measured. And even then, don't unless one part of the code overwhelms the rest. Some of these are like kind of consistent with the like the clean code, I think. Right. Clean code and also kind of what we're talking about with, you know, metrics, telemetry, all that kind of stuff. Number three, fancy algorithms are slow when N is small and N is usually small.

Fancy algorithms have big constants. Until you know that N is frequently going to be big, don't get fancy. Even if n does get big, use rule number two first. Measure. That's rule two, in case you forgot. Really good. Rule number four, fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures. And rule number five,

data dominates. If you've chosen the right data structures and organize things, well, the algorithms will almost always be self evident data structures, not algorithms are central to programming. So good. All five of them are so good. Yeah.

It's kind of the same one, which is basically what you said, which is, you know, measure twice, then cut. Yeah. Yeah, no, I like that. Thank you for sharing that. Yep. Mike RG. Yep, yep. All right, so let's continue this dive into, I think we're on the 85th way now.

We're pretty far, right? We're on part eight of the second way. Still the second way. It's a big way. Dang. You know what's so funny? It is a big way. I think I even said last, all right, so we're done with the second, but then we went and looked at the rest of it, and it was actually really good.

So it was like, okay, well, maybe they'll forgive us. Maybe. Maybe. Yeah. So, yeah, this first one is enable feedback to safely deploy code. And we sort of talked about this a little bit in the previous ones, but this kind of goes into some more of the details of it. So what was really funny is when they start this off, which by the way, I did pick up the audio book as well, and it is an easy listen. So I highly recommend it as

Mike did previously. If you want to hear this stuff and be able to do it while you're driving. Or you can listen to us talk about it way longer. But either way, right? So you'll be good. Listen, we cover in three hours what they do in 45 minutes. Exactly. Exactly.

I mean, bang for buck. You're getting your money's worth here. But you will not laugh listening to theirs. I will say that you will not laugh. But here was one thing that is really interesting. And this is true. And I've seen this happen in many places. Operations, they don't like deploying developer code, right? Because stuff goes wrong. It's frustrating. But then developers like to complain about operations not wanting to deploy their code imagine right yeah how many times you ever said to like someone who's uh asking about a bug or something for on-prem product and you're like yeah but what version are they running you should try

upgrading first and then don't then bother me if you have to right right because that's or how many times how many times have you worked with a developer where you or the other developer have been hesitant to be like, hey, what version of.NET are you running? We should just upgrade the framework on the system. And you're like, I don't know. I don't want to do that. Yeah, yeah. Nobody's tried that. That seems dangerous. I'm out next week. You should do it next week.

Yeah. So they go into this little story about, hey, given a button that anybody can push to deploy, nobody wants to push it. And if you think about that, that makes sense. Nobody wants to be the person that breaks production, right? You don't want to be the developer who did it, and you don't want to be the operations person, so nobody wants to do it. And that makes sense. You know, that is so true. Like, I can actually give you an example. Nobody wants to do it. And that makes sense. It's so true.

I can actually give you an example. I have lived through this. And I think we may have mentioned this in the past where in a previous environment, in a previous gig, we had our environment set up to where everything was through TeamCity. And you could literally just click a button in Team team city to deploy the code and to test it. And, you know, if you want, you're ready to push it, you could click that button. But, uh, you know, we had set that up so that we wanted the, the, the testers to be able to kind of like have a part of the process, right. And, and, and have some ownership and have some stake in it. Right. And, and when they wanted to test them, they could just be like, yep, we test this and oh, it's, it's done. It can

move on to the next stage. But they, they were off, like it was a common thing where they're like, I'm not, I don't want to push the button. I literally, I need you to, I need you to do the deploy. And it's like, what? There's a button. It's just a button. It's a button. You got to click it. You got to log in and push it. Yeah. Yeah. But the thing that they get to that is really good, and you only see this once it's kind of happened, is the solution is these deployments need quick feedback loops, right? If you get that in place, now people are not as scared to do this stuff because they can see if things went right or wrong almost immediately. Which goes back to our previous conversation, right? Like this is where you need that telemetry so that you can know what's happening, right? Because without it, like how else are you going to know, right? Like you need to have some metrics and to see like what's happening with your application.

Yeah. And what they say here is if you do deploy something and it still breaks for whatever reason, then what you do is you fix it fast and you add the telemetry necessary to help you identify it so it doesn't happen again. Right. So it's almost the same type thing that we've talked with, with test driven development or things like that is if you find an edge case that you weren't testing for, because let's be honest, you're not always going to cover everything. The goal isn't to just put the fix out there. The goal is to put the fix out there along with that unit test so that you feel safe in the future, right? And by doing the same thing with this telemetry,

treat it like the test, right? This is your test to see that things are going smooth when they get out into the field. Yeah. I mean, we've described this before. I like to create a unit test that reproduces the problem and then make the fix until that unit test passes. And then that unit test gets committed. And then that's knowledge that the system learns and keeps, you know, for the life of the system. Right. Yep.

Right. Cause like, you know, a hundred percent code coverage does not equal a hundred percent of every possible scenario ever thought of has been covered. You've covered all the, those are not the same thing. Right.

You've covered all the methods, but you haven't covered all the use cases for those methods. And that's two different things. Not even, not even all the methods. Those are not the same thing. Right. You've covered all the methods, but you haven't covered all the use cases for those methods. And that's two different things. Not even all the methods. Like you've covered every line of code, cover some test. Right. But that doesn't necessarily mean that you covered all the different permutations of data that could flow through it. Right.

Right. Right. So interestingly enough, we're talking about unit tests here. What they say is when you get this information in front of everybody. Right. And they called it the radiator thing. Right. Where everybody can see it on screens and all that. This also encourages the developers to write more tests and to write better code because they want to release a successful deployment. Right. Like if you were the guy that releases something that breaks, you're kind of going to be feeling a little shameful about it. And next time you're probably going to do a little bit more, go that extra mile to ensure that yours is a successful release. Step one, get the metrics. Step two, use the metrics. Yeah, totally. And here's one thing that I thought was really good.

And I never thought about this. It's almost like the shame thing, though. It's like a public shaming, though. That's the downside of that statement, is it almost sounds like it's a public shaming. Like, hey, I'm going to encourage you to write better code so that it's never your fault again that production gets broken. Hey, however it has to work, right? But in all honesty, though, it makes us more thorough. It makes us more thorough. But here's one thing that I've truly

never thought about that just made a ton of sense is they said one of the side effects of this is developers will be willing to write smaller chunks of code. So, I mean, we've all been there where we're working on a branch for weeks, right? or a month, or other developers doing that for a long time. And the problem is, as soon as that thing comes back in, everybody's nervous, right? Like, you've been gone for a while, and you're bringing your mess back into the big house. What's going to happen here, right? by going with this more test-driven approach with the telemetry and all that, developers are more willing to put in little tiny bits of code and check those in and then use things like feature toggles and stuff to make it dead when it deploys, right? And then that way, you have a much more releasable branch all the time.

And it's such a, I mean, can we just stop for a moment and just say from like a developer experience, when you have tests set up, it's a much more pleasant environment, right? Oh, yeah. It's just easier. You feel comfortable making your changes. I'm like, okay, well, I guess I have to spin up the environment, run 8,000 different permutations of data that I've thought about. But then there might be another 8,000 that I didn't even consider or didn't even know about. So, yeah. It definitely gives you confidence.

But sadly, they don't. Right. Right. confidence but sadly they don't right right and you know like the number one or maybe the number two a reason people don't write tests is because it's hard to write tests because the code isn't set up in a way to be testable and like by having tests you kind of fix that yeah it does require some pain you do yeah but the but you know depending on like what type of testing you're talking about like when you start talking about uh test driven development like that's one that i've always i mean we've talked about this like when you start talking about test-driven development, like that's one that I've always,

I mean, we've talked about this, like it's a struggle for me personally to like try to think about like what I want to do first before I write the code. So like I, that's a test pattern that I struggle with, but you're absolutely right though. Like if you go into an environment, like a brownfield application

where there isn't the test and now you've got to go and put it in, you'll find so many dependencies that were baked in that nobody realized like, oh, that's a dependency. And, you know, if you're not using any kind of like tool to do that static analysis, you might not realize it, like how tightly coupled your code actually is. Yeah. So going along with this, going with smaller chunks of code and these releases and the testing and all that kind of stuff, this is what will give you the constant, super quick type feedback loops, right? As you do that before it even deploys, right? If a unit test fails, it stops it. It's not going to go out. You can't even deploy it.

If you do push this thing out with a small chunk of code, the telemetry in place, you get these quick feedback loops, both from operations, development, everybody, right? And they do go in and say, just automating deployment isn't enough. After that deployment happens, you need

to be monitoring everything. You need to be able to see what happened as soon as that thing went out the door. I mean, we've talked about... Go ahead, Joe. I was going to say, is there anything developers hate more than slow feedback cycles? No. There's some things that we hate a lot. So I've been doing a lot of more DevOps stuff lately.

And sometimes there's just things that you do and you have to wait for the build pipeline to see if it worked. And then it kind of stinks and some little thing was off and you got to do it again. Oh, my gosh, it can be brutal. I mean, just reloading the page sometimes. Like I've been doing Angular development lately and just reloading the page sometimes even though it's like you know you can just npm run you know your your environment and you have this like always listening server it's like every time i hit save i'm like you you gotta wait for it to recompile and and then load the page and i'm just like oh man like it takes too long like and that's a prime example like that's actually really quick compared to where we were like 10, 20 years ago, right?

Yeah. And yet I have the audacity to complain about it. Right. Like what's wrong with me? Well, I don't have the attention span I had 20 years ago. Now if something takes 45 seconds, I'm like, all right, let me go check the chat. And I have like a billion messages over there and then I get all torqued up about that. And then what was I doing?

What's going on in Slack? I knew I was going to check something. Yeah. So the next section that they go into is saying, hey, use telemetry to make your deployments safer. That makes total sense. So this is – go ahead. I wanted to add one thing, though though to that automate and the deployment process wasn't enough comment that she made because they,

they also talk about in the book, like even, and we've talked, we've talked about it too, as it relates to the book in previous episodes that like doing all the automating, all your performance testing and automating all your your user acceptance testing, like doing, uh, all the automating, all your performance testing and automating all your, uh, your user acceptance testing, like, you know, uh, so you have this idea

of like, you know, test the, you want your test to fail as fast as possible or you want your bugs to show up as fast as possible. Right. So you want them to show up in the faster test methods, but eventually you get to this scenario where like you're spinning up an environment and you're, you're testing it at scale for user, you know, real, you know, real use case, like user acceptance kind of testing things. Right. And so that's an example of where like, you know, the, the deployment process itself,

just automating that isn't what we're talking about. Right. So we mentioned this a second ago. You always have to be monitoring your telemetry on a production release. Always right now. Here's what this adds to you is if you are doing that, if anything goes wrong, you should see it fast, right? Like you should know that something is off.

And this is something that I think Joe Zach loves is he's been, he's been mentioning it a few times over the past several episodes. And even in our work life is don't call something done until it's done because everybody has their version of done. You know, you ask a developer, oh yeah, I'm done coding it. Okay, is it tested? Is it working? Is it right? Like done can mean so many different things. And their definition of done here was, it's not done until it's operating as expected in production.

And I'll tell you, so doing more DevOps works, I keep mentioning. Outlaw, I have to apologize to you for even even though I mostly agree with you, but I still, I debated. I still, there was a shadow of doubt in my mind that maybe DevOps should be a role because I couldn't expect everybody to learn about all the ins and outs of DevOps type stuff. It was just unreasonable to expect people to be able to learn that. So we need people that are going to be specialized in that. And yeah, sure, other people should learn about DevOps.

But I still kind of felt like maybe 25% of the DevOps was maybe a job and not a culture. I am so against that now. I didn't understand at the time how much work you were doing and how much crap you got stuck with when you're doing so much more of that work. And now that I'm doing a little bit more of it, I see people chuck stuff out. And it's not people I don't like. It's not people being rude. It's just when people are detached, when developers are detached from the deployment process, it's so easy to stay detached.

And so people check stuff in. It looks good. The tests pass. Then they disappear. And then when you deploy the thing and it doesn't work you are stuck holding the bag and unless you're an expert in their stuff and you're not then it is just miserable to track down and a lot of times the developers aren't around that it was something they broke weeks ago or whatever and it's just this nightmare for whoever ends up

being the push and push that button and And yeah, so now I hate it. Now think about this, right? Think about like how, like even in our day-to-day work life environments, right? Like how much our world has just like this explosion in automation because now we've kind of like forced ourselves to live in a world where we don't let one person be in charge of DevOps.

It is like, you know, it's more of a team effort now and it is more baked into our culture now. Right. And you know, it's been an awesome explosion of, of things that we're doing now. And there's still, we're by no means done.

There's still a lot that we want to do and a lot of things that we can improve upon. But, yeah, I mean, by having it be part of the culture and everybody's a part of it, then it's a lot easier to iterate on at you know, at a much grander scale. And we're actually going to come back. We're going to come back to this here in a little bit further on in the episode, because there's some good stuff that comes out of it that Google does that we'll talk about. So, here's one more comment that I wanted to make, though, like related to this always be monitoring thing. Because, you know, like for, I guess, the salesman thing is always be selling like the abc's always be selling you know so i

guess i guess the new developer motto should just be like always be monitoring it should yeah totally so here's the one thing that'll be our motto when we talk about this it sounds like it's a magic bullet right just like unit tests sound like they're like the silver bullet for everything. Telemetry sounds like, oh yeah, that's the silver bullet. Like everything's going to be perfect from here on out. No, right? Having the telemetry in place so that you can see it, monitor it, just allows you to get that faster feedback loop. So if something does go wrong, you can react to it, right? That's as opposed to what we talked about, I think in last episode, where it was like, I don't know what's wrong. Just restart IIS.

You know, like this gets you into a place where you can make more well-informed decisions quicker, right? And that's what you're going towards. One more gripe I got to make too. Every time that, and by the way, this is mostly me me i've ever said or someone else has said the builds broke or the deployment broke uh or production is broke now i want to scream because it is so rare and so unlikely that it was the build server that broke right or that it was a production environment thing that broke it It's almost always code. It's almost always smiles.

But just the way the vocabulary that we have around this stuff implies that it's a build problem or it's a production environment. It's somebody else's problem. And what that means is the buck starts with that person. So a deployment, someone who set up the deployment ends up looking at the build or looking at production or looking at the whatever. And I think that's backwards.

So you've got to have those tests. You've got to have the telemetry so you know the right person to start with because otherwise you're starting with the last person in that chain which is exactly wrong yeah i agree with that i can't tell you how many times like people would come to me about hey the build server is broken uh i submitted this uh pr and it's not building my thing and like you know so you got to go fix it and then i'm like well okay i don't know why it's on me to go fix why your thing isn't compiling and then you know i'm a good guy so i'll go looking into it and then i'm like oh here's the reason it's because we agreed that like this is the version of like the, for example, that we're going to use, and you decided to use another version.

Yeah, you up there. And, you know, yeah. So, I mean, if that's what you want to do, then we have to agree on that. But, you know, the build is set up for this other version. Or, like, whatever random thing. Similar types of things like that have happened to me or like, you know, somebody will have like a cached package that works fine for them, but it's not actually included in the project. And it's like, yeah, you could easily

reproduce this if you just like deleted your local cache of, of packages and then tried to recompile and you would notice like, oh, Hey, yeah, that thing actually isn't being included in the source code. Let's be fair here. The problem is when you, like in this case specifically, you went and automated a lot of this stuff. It's magic to everybody else, right? So when something goes wrong, there's something wrong with the magic. As opposed to when you write code, you're intimately familiar with that code you did. And so if something goes wrong with it, you know where to go look at it, right?

And that's the problem when everybody's not involved in this whole DevOps flow and this whole notion that they're a part of the process is it's too easy for people to just dust their hands off and be like like this isn't my problem i didn't touch it but that's wrong because you probably did do some code that does touch this and so you are as familiar with it as the other person so and and you know i still think that devops can be a role but but i do believe that everybody should be playing a part in this and it shouldn't be, hey, the build server is broken, right? If you got to log into that build server, it's almost upon you. If it was your pull request that broke, you should be the first person to go in and at least try and figure out what's the error that's throwing.

Does this sound like something familiar? And sure, if you don't know, okay, hey, outlaw, can you help me with this? I know you're familiar with this setup. Can you hold my hand here so that next time I don't have to call you, right? Man, to me, that's so important. Here's my gripe about it, is that it's surprising how often people will make a change. It breaks the build or a unit test or whatever. And the first assumption isn't, oh, it must be something I just added. Right.

No, it never is. It's never related to the change you just made no of course that system's broken and that's why because it works fine on your system so so and and that's the that's the go-to right so it works fine on my laptop so it's got to be yours right yeah and that's like an old school kind of mentality too right that that's joked about so like that's why like having all these unit tests in place, for example, unit tests, uh, you know, are so great because you can like verify things early and unit tests don't require any kind of, uh, you know, set up on your environment. There's no like dependencies.

You could just run them and, and, you know, see if there, if you can reproduce a problem or not. Right. And that's like your first level, there's your fast feedback loop right there if you can reproduce a problem or not, right? And that's like your first level. There's your fast feedback loop right there that you can perform locally, you know? You know what really stinks too about kind of being on the front line there and being like that person to go to for escalations is the first person who gets like the emergency, it almost becomes like your responsibility to prove that someone else needs to look at it. Right. Because no one else wants to touch it, right?

So it's like you have to definitively find the problem to go get someone, get the right person to fix it. But the thing is, half the time, by the time you find the problem, you know what to do to fix it. So you just fix it. And there you're just like cementing yourself as the go-to escalations person by doing that stuff. And then people don't know what you did and yada yada.

And so remember the Phoenix Project, there was a guy like that uh wes was the person oh it was brent wait it wasn't yeah it was brent oh crap you're right i told someone wrong the other day uh yeah so so brent would be the one to fix everything everyone always run to brent whenever there's a problem and brent knew what to do but nowhere to go when brent wasn't around or brent became a bottleneck because there were multiple things broken, it was a major problem. So eventually they're kind of fixed with that to separate Brent from the keyboard and have other people learn how to do that stuff and start taking

ownership. And that's really what it needs to be, right? It truly does need to be everybody taking ownership in it and learning the process. It shouldn't be all in one person. And that's why it's a, that's why it's not a role.

That's why it's a, uh, you know, a job responsibility. It shouldn't be on one person. And that's why it's not a role. That's why it's a job responsibility. It can't be a role. You can get paid to do it. Because everybody needs to be a part of it. Everybody does need to be a part of it. It can still be a role. Alright, so the next part. This is important and it's so obvious

that you'll go, oh yeah, totally. The biggest cause of production issues are deployments. That shouldn't be a surprise, right? Like, if you're not deploying anything, unless there was some sort of server patch or something like that that happened, chances are it's just going to continue running. Unless, you know, all of a sudden there was a spike of traffic or something, right? There can be some anomalies, but for the most part. A server patch is still a deployment. It's just a deployment of someone else's code, right? That's fair. That's fair. That's a really good point. So, but yeah, unless it's a denial of service,

you know, if you're getting like a DDoS or, you know, uh, you know, if you're, if your system just can't legitimately keep up with like Christmas traffic, for example, then yeah, okay, fine. Yeah. So that's also, they wanted to point out, that's also why it's really important to overlay these deployments on your metrics, right? Like it should be plotted on that same chart where you're looking at the number of incoming traffic and all that kind of stuff. It needs to be there. You know, if the developer is able to have the telemetry and to see the stats on their change and it goes into production quickly after they push it then they should be able to verify that it's working in production and be able to step into that so it's all it's like several failures have happened by the time some you know a devops person gets kind of that

you go fix it that that reminds me of like the Facebook workflow where like, I think we've talked about this before, but like, and maybe it even comes up in the book, I forget now, but where like the developer who's writing the code actually is the one who will push it out to production and they're not done until they've deployed it. Like that, what a beautiful thing, right? Like it's so simple, but you know, that idea of like defining done, right? And empowering the developer to actually deploy their little bitty bit out into production, right?

And they're set up in such a way that they could do like hundreds or thousands of deployments in a day. And you know, it's fine. Right? Like that's the kind of automation I want. Yeah. I want to live in that world. We're getting to the utopia soon. So this next one is kind of interesting. This next section is talking about pager duty, both devs and ops working together, right? So this is good because this hits on the business case, the actual overall business case, not just the application that's deployed and all that. They talk about some problems, and we've seen this, all three of us. I know we have. Every developer that's

been around for a minute has seen this. There's some problems that go on forever, right? Like, there's a problem that, you know, I don't know, every night at 2 a.m. something goes bad. Right. Like you get errors and that just goes on for three, four or five months. Right. And everybody's like, I don't know, it's two o'clock in the morning. Something's wrong, but it seems to recover. So moving on. Right. Or maybe it's not even that consistent. Maybe it's a problem that happens like on some random pattern that we haven't figured out when it happens.

But when it does happen, it happens for like a 30 minute block of time. Right. Or, you know, some, some significant, like it has significant impact,

but it's inconsistent. And, you know, you have difficulty trying to track it down. Yeah. So what they call out here is sometimes that'll get sent to you, right? Like you're a development team and you'll get sent this bug, but you got some features that need to go out, right? Like you're a development team and you'll get sent this bug,

but you've got some features that need to go out, right? Like you got some features that the marketing team or the product team wants you to get done. And so those bugs or whatever get deprioritized, right? Like they're sitting on the bottom of the heap somewhere. And that can be a problem, not just because it's an issue for the application, but because you might be putting a stress on the operations team, right?

You're sort of decoupled at that point. The development team's making their stuff. The operations team's trying to keep it alive. You as a development team may not even be aware that they're going through this problem, trying to resolve this and figure it out every night, right? And that kind of stinks. And what they said about this that I thought was really good, and this is where the business use case comes in, is this is what they called

self-optimizing for various different centers, right? Because you as a development team, you're optimizing to try and get features out. The operations team is optimizing to try and keep the thing running. The problem is you as a business overall may be suffering because you're not trying to get the entire group working and functioning properly as a machine. Right. You know, go ahead. Well, I was just going to add to it. Like those two goals are at odds with one another, right? Like you mentioned the developer team wanting to, you know, they're trying to optimize their team to like get new features out.

And the operations team is trying to optimize their team to keep the application up and running and error free. Go back to what you said before. Most production issues are related to production deployments. So therefore, the operations team has an implicit desire to do as few deployments as possible. Right. Right? Because to them, all the code is just a black box. They know nothing about it.

Right? This goes back to what you guys were describing before with the build server issue. Like if you don't know anything about it, then you just treat it as like an unknown, right? And that's the way the code is to them. Yeah, I mean, so the way they worded it here that I thought was really good is they said,

when it's done this way, when you kind of silo these things off, you are reducing the performance for the overall value stream of the business. And that's a really good way of putting it, right? Like it points out the problem. So what this means and what they say about it is everybody that's a part of that value stream should share the responsibility for operational incidents, which gets back to what you guys were talking about where a developer being involved, right? If you were the one that coded it, yeah, okay. So operations is the one that has to keep it

running. But if there's a problem, you should also be in that loop, right? It shouldn't be the operations person trying to figure out what in the world happened with this black box. Have you guys, I mean, this is less common these days now because we have things like DevOps in place and whatnot. But I mean, I can remember, you know, a while back, you know, like years ago, where, you know, you would see the people who were responsible for maintaining your code. And every now and then you'd be like, oh, I'm sorry, man. I didn't realize this was like keeping you up. You know, I didn't realize that this was a problem. Like nobody said, like, you know, it's an easy fix. I apologize, man. And then they would be like, so relieved, like, oh God, this is, this is behind us now. I don't have to worry about it. Like you fixed it. Thank

you. Thank you. Thank you. And you're like, you know, like you feel bad. Like, you know, cause that was, you know, 10, 20 years, you know, one hand wasn't talking to the other. That's the way these things were, you know, managed and dealt with, right? Right. And at least now as an industry, we're starting to like recognize like, no, we need to get the developers talking with operations and operations. Like, you know, let them be involved in the process of building the thing, you know, not just the deployment of the thing.

One of the, the next things that they said I thought was really funny in here is new relic. A company may have heard of, they said that, Hey, when developers would get awakened by a pager at two o'clock in the morning, issues never got fixed faster. I mean, think about that.

It makes sense. Like operations over there are struggling. The developer is like, oh, God, I know what happened. Right? So they get out there and fix it. So does that mean that you should always call the developers at 2 a.m. then to fix every bug? You're not going to find my phone number. I work in DevOps, so I didn't change the code.

I know where Alan lives. Hold on. That's right. You have to come find me. But fortunately, I sleep like a rock, so it ain't going to help anyways. So this is, again, going back to the whole definition of the word done. It's not done until it is out there and running as expected so you know this was all really good stuff and we're about to pick up on

some some more interesting things that have touched on things that both of you guys have said leading up here in a minute today's episode of coding blocks is sponsored by datadog the unified monitoring platform for full visibility into all of your serverless functions. Troubleshoot performance issues faster by seamlessly navigating between logs, lambda metrics, distributed request traces, all within one unified platform. Yeah, and as we've been going through this whole section on the DevOps, we've talked a ton about telemetry and Datadog definitely gives you that in spades. Datadog provides real-time screen boards and service mapping so you can get complete observability

into your own serverless environment. And Datadog's big conference dash just completed and they had a ton of new announcements including the incident management and compliance monitoring. So I definitely recommend checking out their new announcements, including incident management and compliance monitoring. So I definitely recommend checking out their new announcements. They've got some posts on there for it. So just a lot of really cool stuff that is still coming out today.

Yeah. And they just recently put out a blog article on, you know, we were talking about like monitoring before and the importance of like, you know, knowing when, you know, keeping track of that telemetry and metrics and knowing like, you know, when your releases are tied to it and whatnot. They just released and put up a blog article about like real user how are your mobile users, which, I mean, come on, let's face it, that's just a huge part of any application today, that you need to understand how they're using it, because how they're using your application is going to be different from the desktop users, right? So start your 14-day trial today. Start monitoring and find out. They've got 400-plus plus integrations so they've definitely got you covered whatever your technology stack is they've got ways to monitor uh sign up today

get that 14-day trial started start your first dashboard create that first dashboard and receive a free datadog t-shirt and make sure to visit datadoghq.com slash coding blocks to learn more about how they can help you optimize your serverless environment and don't forget about that trial and t-shirt and you can get there at datadoghq.com slash coding blocks all right hey it's that time again to ask for a review because it's so helpful and we were so sad today so we can use a little pick me up just a little bit if you could just go to cuttingbox.net slash review and hook us up that would

be phenomenal and Outlaw said he would do a cartwheel on show if we got two reviews so that would be just great man I'm going to break something if I do a cartwheel. Not in the office.

You're talking about on you. Both. Both. I'm probably going to break something on me and something around me at the time if I try to do a cartwheel. You made a bad deal. I mean, I do it all the time. It's okay. Have you guys ever tried to do a cartwheel. You made a bad deal. I mean, I do it all the time. It's okay. Have you guys ever tried to do a cartwheel?

Oh, yeah. Yeah. I do it all the time. Successfully? Yeah. Yeah. Every day. I do it downstairs.

All right. You guys, I can't even trust you guys. I do cartwheels to the mailbox. I'm going to get the Visual Studio tattoo instead. I went and picked up five guys today. Yep. I cartwheeled get the Visual Studio tattoo instead. I went and picked up Five Guys today. Yep, I cartwheeled into the door. Someone held it open for me. Maybe I'm doing it wrong.

All right, I'll try again. All right, well, with that, we head into my favorite portion of the show, Dad Jokes. Gotcha. All right, so I got this one from Klaus on Twitter. jokes. Gotcha. All right. So I got this one from Klaus on Slack. I think he said he found it on Twitter. But what's the difference between a hippo and a zippo?

I don't know. Uh, one is very heavy and the other is a little lighter. Nice. All right. I love dad jokes. They're so silly. All right.

So, uh, now it's time for my real favorite portion of the show. Survey says, all right the show, Survey Says. All right. All right. So a few episodes back, we asked, or Google Play Music, CDs, or vinyl. All right.

Who went first? Alan, you went first, right? Last time? I have no idea. Okay. We're going to go with Joe. Joe, you go first. Tell me which one and by what percent.

I think the radio came in dead last with 0%. Okay. That's not the way we play this game. You can't change the rules. Unless you're just going to go through all of them and work your way up to the winner. It's going to take a while. I think vinyl came in second last.

I think Spotify won with 15%. Spotify at 15%. Okay. All right. So I'm going to put on my... A lot of developers are really cheap. And so I'm going to go with YouTube. And we'll go with 20%.

20%. All right. So Joe says Spotify with 15%. Alan says YouTube with 20%, correct? Yep. Also, Spotify can be free. It's got to have ads. Has ads.

Yeah, that's true. That's a good point. I missed that. Too late. Yeah, that's true. That's a good point. I missed that. Too late. Yeah. And the winner is... Mathema Chicken! What the fuck? What?

Alright. Yeah, who's bad at math now? What's up? Mathema Chicken. Got your number. What's up? Yeah, except you were way off on the percentage. It is 55% of the vote is Spotify. Wow. Yeah, that was a bit surprising. Y'all got to send me your playlists.

Yeah, your claim for last place, you were wrong. The radio, my friend, was not dead last. Dead last was a tie between vinyl and iHeartRadio. I almost said iHeartRadio, but I'm trying to get them to sponsor the show, so I was kind of... Actually, I didn't trying to get them to sponsor the show, so I was kind of... Actually, I didn't expect to see them... YouTube music was second place at 15%. Oh, okay.

And that was actually kind of surprising. Yeah, major drop-off. Because here's the way it went. It went Spotify, YouTube, Google Play, Amazon, and then Apple. And I was like, really? Apple's number five? I get the whole Android, Apple versus iOS thing.

But I kind of expected that Apple would have had a better showing than it did. Spotify is really good, man. Yeah, and Apple sucks at this. I wasn't surprised to see Spotify in first place. I'm not arguing that part. I't surprised to see spotify in first place i'm not arguing that part i just i was surprised to see apple so far like amazon ahead of apple agreed yeah that's true but but isn't amazon you get some of that for free do you get any music for free with apple uh that's a good question it's been so long since I looked at it. I mean,

that might be it because I mean, I can totally see people just turning on. If you got Amazon prime, you can listen to a lot of music on there. So, yeah, but I feel like that's fair. Cause now it's like,

you know, you're paying for it. Right. It's not really free. Well, you get your shipping, right? Like, come on. Yeah. You get a lot of stuff with Amazon prime. So it's It's not really free. Well, you're getting your shipping, right? Like, come on. Yeah. You get a lot of stuff with Amazon Prime, so it's all just kind of free. Yeah. You get a lot of stuff. So, you get a lot of value out of it, but it's not free.

Yeah, that's right. Yeah. I will say. And, you know, they keep raising the prices of Prime, so it keeps becoming less free. That is true. I'll give you that too. Spotify's interface is amazing. Straight up amazing. Yeah, totally not discrediting. I wasn't surprised to see Spotify in first. That's pretty good. I didn't expect it to be that big of a drop off to number two, though, honestly. That's great. That's cool. The thing that got me is I expected Spotify in first place, and I expected Apple in second. And my reasoning, my rationale behind that was because I figure Spotify, A, it's wildly popular.

B, it can attract both the Android and the iOS market. But for the rest of the iOS market, I assumed that Apple Music would have been the dominant player there. And that's why I'm so surprised that Apple fell back as far as they did. What was its percentage? I mean, in fifth, it was less than 10 somewhere, right? Yeah, it was like 5%. Yeah. Ooh. Ooh. Yeah. That's crazy. Yeah. All right. Well, for this episode, uh, this is, you know, a follow-up to a conversation we recently had and it was on, on, uh, discussed in, on Twitter and in Slack. And I thought, you know, Hey, this would be a fun one. Um, so that the internet can prove me wrong. Do you prefer that your laptop keyboard... includes a numpad?

Because I'm a number-crunching accountant. Tax season ain't got nothing on me. Or does not include a numpad. Just center the 84 and be done with it. I don't need carpal tunnel and just one wrist. I'm genuinely curious on this one. It just bugs me so much. The laptops, they include the numpad because they're off-center to the display. I know we've talked about it before, but I just...

Oh, come on. Stop making them. Or at least make it an option. Like, make it an option. Why can't you make it an option? Especially, like, open, like, kind of platforms, you know. Like, you can already decide if you want, like, a U.S. English keyboard or, like, other languages. Like, how about numpad or not numpad?

But then, of course, that would require that they, that they put the trackpad in the center as well. So whatever. This episode is sponsored by Secure Code Warrior. Secure Code Warrior's gamification lets you learn how to write secure code from the start and identify bad code already present. Hey, so as a developer, as somebody that has to go through trainings constantly for companies that I've worked for,

I think we can all admit that usually these trainings are pretty boring. It's usually a video sitting there and at the end you have a quiz you have to take, right? What if you could actually take some training that not only you learn something out of, but you're fully involved in, I think we could all agree that doing something fun like that is something valuable.

So if you're a developer and you really care about making sure that you're improving your security footprint at your company, maybe this is something you should tell your boss about and be like, hey, there's a great way to get training that's not just listening to videos and clicking some buttons at the end. Imagine you put your time in and you actually learn a bunch of stuff come out of it. I'm just browsing the languages right now. I'm looking at Go API, Java Spring, JSF, Java Enterprise Edition. I even say they've got one for COBOL here. There's a good chance that they've

got you covered on whatever kind of technologies you're using so you can actually get in here and learn about how to make your code better and what's better than beating your friends and being able to have metrics to show that you've bettered them and you can show them the numbers because i'm just saying like you know i mean alan got to step up your game, buddy. You're in last place. My accuracy destroys you guys. Wow. Yeah.

But you know what? I mean, if your kill-death ratio is awful, who cares that you, like, that one time you aimed the weapon, you got the person headshot. You know, that doesn't matter, right? So, yeah. So So that's the fun. That's what's so great about what Secure Code Warriors has done is they've made it fun to try to go through these things. And it's no joke, the amount of languages and the depth that these things are covered. I don't know if you guys are aware, but Joe Zach has expressed a certain recent love affair with Kotlin. They have 405 challenges just on Kotlin Android SDK alone.

Just the Android SDK alone is that deep. Every language you want to look at, they have a ton of different challenges for you to go into. And then you too could be like, hey, Alan, you got to step up your game or whoever your Alan is, right? And that's honestly, it's fun, right? So listen, head to discover.securecodewarrior.com slash coding blocks to start your next game. And if you score 5,000 points, you get a cool t-shirt. That's discover.securecodewarrior.com slash coding blocks. All right.

So now we're talking about, oh, wait, let me. Got to do them cartwheels. That's right. Okay. Yeah. Just got done doing some cartwheels here. So now we're back to talk about developers following work. Whoa, whoa.

Sorry. I was trying to do a cartwheel and I got hurt. I believe it. I believe it. It was very convincing. The walls were shaking. It was crazy. I told you.

All right. So this gets into kind of what we were talking about earlier with developers looking over people's shoulders, right? Like we've all done this in the workplace. Like, hey, go find out why customer service is having a problem using the system right you go sit there and you look over their shoulder and you're like i'm so sorry just like you said earlier right out for sure like man i apologize they actually had some stories like some real life stories in the book that were like um yeah there was one thing that this person was doing it took 63 clicks or something like that, right?

They were like, man. And another one, 1,300 steps to do something, right? It was absolutely ludicrous. And hopefully you haven't got anything that bad, but maybe you do. And there were tasks that these people had to do multiple times a day. Right. That they were expected to do. Right.

Which absolutely stinks. And as a developer, if you see that, you automatically want to help fix it. You're like, oh, man, I know that I can improve this for you, and it really wouldn't be that hard. Let me go back and see what I can do. So their whole point in going through this is we as developers have always enjoyed doing that, right? Like making something better for somebody. You should also do the same thing for operations, right? As a developer, it should be built into you to want to try and improve their life because they're the ones that are trying to keep this thing alive. And so remember, next time you call customer service and start to give them a hard time about something that got messed up,

that it was probably a developer's fault that it got messed up in the first place. And it's probably a developer's fault that the customer service is having a hard time finding and fixing your stuff. Because, well, someone like you wrote the code for it and they're not as perfect as you are now. Someone like you years ago you're

right you're first starting yeah totally it would never be you now right here's a here's an interesting here's an interesting take on it i don't know if you've ever thought about it from this point of view but uh i've kind of had this opinion that installers are the old school way of DevOps. Like that's the old school scripted, repeatable, like, let me lay down the application, right? Like that's the OG of DevOps. But nobody like really thinks about like installers in that way. But like, if you have an application that requires an installer, like that's kind of how it's done, right? Like that, that thing lays down the application in the way that it's expected and, you know, upgrades bits as necessary

and, you know, is supposed to handle any migrations or whatever. That's absolutely right. And if it was headless, that was even closer, right? Oh, yeah, that was the best. Yeah, when you could script the – when you could like pass in the installer like a silent file of options. Yep. Yep.

So what the whole point of this thing was, was to, if the developer is working with operations on it, they feel the pain that operations feels, right? And when you do that, just like if you watch over the shoulders of the people in customer service who are using the application, now when you go back to write your code,

you're going to be aware of the pains that they experienced. And so you're going to make better informed decisions when you're writing your code. And when you're figuring out how this thing is going to work when it gets deployed, you'll know about those things, right? And that disconnect is a big problem. If you bring that and close that gap, now you got something to where everybody's happier, right? Yep. And then you can like, you know, go back with your head down, you know, tail between your legs.

Like, I'm sorry, I didn't realize it took you 1300 clicks to do that thing that you have to do five times a day. Right. It's such a simple thing. It's amazing that just these little things of just getting people talking together and working together on these little problems, what might be a massive problem for operations turns out to be a 10-minute fix for you because you just weren't aware they even had to deal with it. Yeah. That's why it's just, it's that feedback.

Who would have ever thought that collaboration would lead to better things? What a weird concept. Yeah. But it's kind of like we talked about when we first started talking about this book. We talked about Pat Flynn and the wedding invitations and how you might think one person should fold and one person should do this. But all it takes is one little mistake that can lead to a really big bad outcome. But if you just do things one at a time, it ends up faster and better because you stop when you make one mistake and that mistake that you made probably is going to inform how you do things from then on i do want to point out one thing here though because i think

this is super important there is a big difference between watching over somebody's shoulder or working with somebody on something versus somebody telling you what's wrong with something because inevitably things get left out and and so you might think that you're going to help solve a problem but you didn't get all the information you actually needed or you might have been given information from a perspective that really wasn't how it was supposed to be used like it's important that you actually work with the people and not just take notes from what somebody is telling you because the two don't necessarily meet up.

Well, I mean how many times have you found yourself with a ticket to like fix some problem and you just have to go back to the, to the business user and be like, okay, just show me. Right. Yeah,

totally. Yeah. I see all these words. I got nothing. So yeah. So here's the part that Joe, you hit on earlier that I thought was just absolutely amazing. So this particular section is developers self-managing their production service. All right. So this sounds kind of scary, but sometimes deployments break in production because we learn things too late in the process, right? Like, oh, we didn't know that worked

in production like that, right? Like maybe there's any number of sorts of reasons that that could have, but having the developers who actually created the thing, both monitor and manage the service when it first launches could be incredibly valuable, right? You're the person that created this thing. You kind of know how it's supposed to run. You get it out into production and now you're watching it, feeding it, making sure it's healthy. You now know you're intimately aware of the things that

are going on. So when there is a production issue, you probably have a better idea about what you need to do to solve that problem. Yeah, for sure. I know like after you've been working for a while, everyone has a situation like this where something went wrong. You didn't pay proper, you made a mistake. And you then, you know, a little while, went to lunch or went home early or did something and you got a call while you're there with someone having a major problem that you could have prevented. It's such a horrible feeling.

And we know everyone's been there. It happens. But man, I like to avoid that. Yeah, everybody does. And so here's the cool part about that. If you're the person that's managing this thing when it's out there, this doesn't mean that you as a developer are going to own this thing for all time. You're going to get it out there, make sure it's stable, make sure it's running,

make sure it's done, quote unquote. Then you hand it over to operations because now they've got something. They can feel pretty confident that, hey, this thing's up and running. They've worked out whatever kinks, you know, we'll keep it alive. And they say that this is practiced by Google. So that's really interesting. I mean, maybe Merle knows that he can give us some feedback on that. But, you know, I imagine like you guys remember when Gmail launched or any of the Google Labs products. Like, I have a feeling there's probably a group of developers there that are like, yeah, OK, we think this is good. All right.

You guys are going to get this thing out into production. And then after some time, it'll be turned over to the ops team yeah i don't know though because in the book they were talking about how like there was a period of time there at google where the deployments were just awful and nobody wanted to be a part of them right yeah i remember at the time that gmail came around that i think was around the time frame that the book was describing those deployments being painful that's probably so maybe nobody did want to be there so uh you know that reminds me of something else i heard about google recently where um someone who started uh at google when it was you know still early so like early 2000s maybe uh and they said back then all new employees

had a like a orientation where they explained how search worked and how google worked and at the time it was small enough that people could explain to you how Google worked and what Google had to offer customers and how Google made money. But now Google has grown so big that it's unfeasible to even describe how they make money, what their business model is to new employees in a reasonable – you can't have an hour-long orientation to describe how search works and how Gmail works and all this stuff and how it all plays together. It's just unfeasible. And part of me thinks it's like, well, maybe that has something to do with them, you know, getting their heads wrapped around DevOps basically, and being able to grow and figuring out how to just scale as an organization, both with people, but also with the products that they offer. Well, I mean, we've talked about in the past that like the more people that involve like the the more uh that that matrix of communication becomes like just impossible to scale right and so that if

you can communicate things through code then you know you're better for it right and so maybe as part of their their uh adoption of devops and everything. Like now those things that they used to have to communicate in person, now they don't have to communicate that way. They've got other mechanisms that can be used for like, hey, you want to know how to deploy this? Well, there's go get the mono repo, and there's a readme in that section for it. And you can see how the DevOps portion works for it.

I'm just guessing or speculating, but maybe that's part of it that's really like so much of devops is just about figuring out how to automate and scale your organization like not not talking about like scalability in terms of how many you know users or bytes you can process whatever but just in how much you can offer and the only way you can get to that point is by basically automating and, uh, you know, getting this stuff going so that you can move faster. It's hard to the book.

Go ahead. Well, I was going to say the book, the book makes a strong case that like, you know, DevOps is more than just automation, but I still have a hard time like separating it. Like I get,

I get the metrics and the telemetry are part of it, but you know, and it's definitely something that you want everything, but I still kind of view that as like, okay, you know, that's part of the code that you're, you're,

you're, you're going to automate and deploy. Right. You know, like, so it's, I have a hard time separating this to those things, but you know, they, they make the case that it's so much more.

Well, the thing that the thing that stinks and the reflux when it comes to the thought of taking their developer and spending two or three weeks trying to automate something right well i mean i can make you go in and do the build and just give me the the compiled assets and and hand it off and it'll take 10 minutes right and so they have a hard time around that. But what you'll find is if you can ever push that through, once you have it, as soon as you don't for even five minutes, people lose their minds because you've enabled people to do things that they've never been able to do before at a speed and a repeatability that you just can't match. And it's really hard to get people to open their eyes and buy into that if they've never been a part of it before. I'll tell you, so... I mean, Parts Un...

You go ahead. I was just going to say, like, Parts Unlimited had a really hard time. They didn't used to be a believer in DevOps. And then they learned their lesson. Yeah, with the Phoenix Project. Phoenix Project,enix project yeah all right go ahead so i was gonna say you know back a long time ago uh when i was doing web development work uh i mean i still am but uh a lot of my job at the time was uh at a boutique web shop was spent doing things like text updates for uh on images for example because i

sometimes went out of text and and one of the sites looked good and we had the fonts and stuff, so I would do that. And I also had batch jobs that would run every night or every week that I would kind of shepherd. I would make little changes to it. I would check it out the next day. I would build and publish and get it signed off on. And so much of that work now are things that have been turned over to users,

either through better tooling or just better practices or whatever. And so so so much of my job and so much of my time spent all those years ago was doing work that was easily automatable and what that means is now instead of having a person that you keep on staff and pay a bunch of money every year to monitor jobs and make text updates and publish to production. Now you can have a developer come in, spend two weeks, set that stuff up and then move on and then turn the buttons over to the business to make the text changes and to check on the jobs and have the monitoring. So you're only, you know, when you're alerted, when there's a problem. So it just frees that developer up that headcount to do something else that's more valuable and more specific to your business you know that's an interest it brought up like an interesting uh

idea that um what you were saying because have you ever like especially from a consultancy kind of point of view notice that like when when you go into a new customer or whatever, and you start talking about like automating things and whatnot, like immediately somebody gets afraid for their job, right? But just like in your example, right? Like there was still plenty of development left to do. Like, okay, fine. So you could automate your job. That just meant that there were other bigger things that you could go and then work on. Right. And oftentimes that might mean building upon the things that you previously automated.

You just keep building these layers on top of it. Yeah. The people paying for the services didn't care whether I was hand editing an image or if it was updated by them. Right. So we could have charged the same amount of money and I could have been working on stuff for new clients. clients could have been getting new work in. Yeah. There's always the value add thing, right? If you can automate something,

you probably should. So, Oh, and, and that ties in, like, I didn't even realize you had this here, but like ops can act as the consultants to assist in the,

in this process. Right. So like, yeah, you know, get, get there, get there, treat them as the subject matter expert, right? And they can help out with it, right? Even better if you can actually have

them sitting alongside you writing, you know, the code or, you know, configuration to assist in this process. Yeah, totally. One of the things that was interesting about this is it was kind of like a punch list that they said that you could have, right? So these consultants from the operations group could basically just give you like sort of a checklist, like, hey, we need to know about defect counts and severity, type and frequency of pager alerts, monitoring coverage, system architecture,

deployment architecture, production hygiene. Those are all things you have to be aware of and know about so that when this thing is in production, we know how to keep it healthy, right? And so you as a developer, if you have these people consulting with you before you ever even push this thing into a production environment, you have all these things in mind, right? Like you've, you've coded to these, these particular philosophies that they have or these needs. And then that way, everybody's in a better spot when it actually happens. And then they actually said, if you haven't met these, then they should be addressed before being deployed. So your consultant there says, Hey, no, you know, I like most of what you

have, but you need to get some more telemetry in here. That saves you a ton of time in the future. And then the last few little pieces here that I thought were interesting is another thing that they can consult you on that maybe you didn't think about as a developer. Are there any regulatory things that you need to be worried about? I know several of us have heard of HIPAA and GDPR and all those kind of things, right? Maybe you're not aware of it as a developer, but these operations folks manage a bunch of different systems that all have to be compliant. Maybe it's FIPS compliant or whatever. There's just tons of different things. Let them help you, right?

Let them tell you the things that you need to be aware of. Yeah, I remember when HIPAA came out, as an independent contractor, I was just doing development on the side. It was gun for hire kind of thing, right? And some of the customers I did work for were, you know, like medical-related type fields. And then HIPAA came out, and they were like, oh, we have this whole other slew of requirements. And not being, you know, the person that was, you know, the subject matter expert on the regulatory process for that, it was a whole task for me to be able to even keep up

with everything that they, the requirements that they had to where it was like, at some point it's like, okay, no more medical customers because it's just too much. It's easier to just not focus on that for right now. Yeah. You know? one thing i

like here they mentioned is having a service handback mechanism which is basically a way for ops to hand back the code if production service becomes too difficult to manage so they can take a look at that and it'd be really cool to not only i mean i think you have to have this like it can't just be something that's extraordinary. You have to have a way to be able to say like, if this has too many tickets or you need to kind of just define some objective, some service level agreement or objectives

to be able to say like, if this has become too much of a drain on our time and operations, if we keep having to go back over and over and over, look at the logs, send you the logs, get the fix out get those uh get the pagers going off then we've got to be able to turn this back over to deb's because it's obviously not ready and i it's i don't know how to do this without it being insulting and maybe it should

be insulting frankly but um i think that it this is important to kind of have that as an option because otherwise it's just a really really rough rough time for ops. And it's not fair because getting paged is horrible. I don't know that. I don't know that it has to be. How did you put it? Insult. Yeah, an insult. I don't think it's insulting if you do it like what you said.

If you have some sort of metrics that are measured, right? Like if you get x number of tickets or this thing's unstable for a number of days then then you guys own it back right it's no i don't necessarily think that should ever be a slight or an insult on anybody because like we said man like we've been doing this long enough and the three of us have worked at enough different technologies and stacks to where we know that most of these things just really aren't that easy right yeah like i've had friends that are like oh man with machine learning and ai it's going to take over all your jobs like man nothing works the way it's even supposed to work so yeah so luck i you know

computers like ones and zeros if this works like this this, right? If it worked that way, then we'd probably all be writing better code. Nothing is ever that simple. So it shouldn't be an insult. Just know that there's going to be problems that happen. And I think it's actually really good if operations could be like, hey, we're going to need you guys to figure out what's going on before you give it back to us. Yeah, I think having those metrics really helps because if you can say, you know what, you rolled this thing out three weeks ago. Since then, two-thirds of our new tickets have come from this system,

and the rates aren't going down. In fact, it's getting worse. So you've got to take this over. You have to give us some people to rotate in on that on-call schedule or whatever because we just can't maintain this load from development. So welcome to ops. And they did say in the book also, this is also a Google practice, right? If something like this happens, the ops team can turn it back over to the developers.

So again, these companies that have sort of been forced to mature their development process over time, if they're doing it, it's probably not a bad idea to take a look and see if it might benefit your organization. Yep. Yeah, so with that, we'll have links in the resources. We like, of course, the DevOps Handbook, the Phoenix Project, the Unicorn Project are all going to be there. And with that, we head into Alan's favorite portion of the show. It's the tip of the week. Yeah, baby. So here's the thing, man. my life so much more amazing that I don't know how I ever lived without it. Or I'm so mad at the world would try to do something that when I come across

something cool, I'm like, I've got to share this because maybe it'll save somebody else hours. So I don't know about you guys. Mac OS can be really frustrating. It, it, it's an amazing development environment.

You do almost everything there, right? It's, it's your,-y stuff. You can do your Windows type stuff. You can do everything there. But when something doesn't go wrong, it seems just so hokey. So here's an example. You mean when something does go wrong? Yeah, when something does go wrong. I'm sorry. That was so conflicting. I was like, like wait a minute when it doesn't go wrong so yeah that might be the problem is even that goes wrong so here's the deal right like i we've talked about this with languages even dot net like dependencies are always a pain in the butt regardless right i think the one language that seems to have their act together is Java. Ruby, oh my goodness, I ran into some problems with Ruby.

Python also seems to be sort of a train wreck when it comes to dependencies and versions and all that kind of stuff. So in a nutshell, I was trying to do something that was doing a pip install and had some other stuff, right? And everything was working fine for me inside Docker. Docker was great. And then I go to do this inside JetBrains PyCharm, and I have it do what I think is called a virtual environment or something like that. I don't even remember. Yeah, we talked about the virtual environment module

as a tip of the week before. Okay, so I'm using this thing, and it blows up. It blows up. Every time I try and do this thing with the pip install, it fails, man. There's so much hidden in Mac OS that relies on X code extensions that will just drive you crazy. So I go out, I Google, I try and figure out why it's not working. I can't find a good answer. Everybody's like, well, you need the latest tools. I go and get the latest tools.

They don't work. It's still breaking. And I'm just mad at this point. So this led me to PyCharm has the ability to run Docker as a remote interpreter. That is so amazing. So you can take your Python, all the stuff in there, and I'll give you the gist of it. So typically, if you're going to do a pip install, if you're running Python on your local machine, that's going to go get the requirements from whatever you're trying to install.

And it puts them into like your bin, right? So that you have access to these things. Would fail every time. What you can do when you do it as a remote interpreter. So I created a Docker image that had the pip install that I needed in it. So basically when this Docker container would run, it already had all these things installed and then it can launch this application and run it in that Docker container, either in debug mode or in a run mode. So that's amazing. I didn't have to worry about this garbage that was going on on my Mac OS system

to find this command line utility that should have shipped with Xcode that wasn't there. It all just worked in Docker. Absolutely amazing. I have a link to it here. So, which this also is why all three of us love Docker is because it has all the dependencies you need in it. I don't care about what's on my developed machine, is why all three of us love Docker is because it has all the dependencies you need in it.

I don't care about what's on my developed machine, whether I have 3.7 or 3.8 or 3.9 or whatever, right? It's in that container. Whatever I need is in that one container. Done. You are doing any development locally with it, though. I can't recommend enough to use the virtual environments. And that was my tip of the week back in episode 102.

So I'll have a link to that episode as well. But yeah, definitely virtual environments in Python makes it so much easier because then you can upgrade whatever you need to upgrade or have like whatever packages and you don't have to worry about overriding your global, I don't have to worry about overriding your global

I don't want to call it a namespace, but global configuration with specific versions and then have different code that relies on another version break, right? You could have each of your code bases have their own dependencies available. Yeah, I was excited about them until they didn't work. And then I was like, man, I don't have the energy. You weren't using a virtual environment though.

I was like, no, I was, it still was trying to rely on the Xcode built in command line utility stuff. That's the part that drove me crazy is I went with the virtual environment. So it would be sandboxed and it still was trying to reach out to some stuff that should have been installed Mac OS and he couldn't find it. And I was like you know but did you upgrade did you did you but you then upgraded

in your in virtual environment no i started i created a new project with the virtual environment and it wouldn't work yeah no it was yeah so the way that works i'm sorry i'm trying to like make sure i follow along but because the way that works is the virtual environment is going to be based on whatever your current system is and whatever the packages are available. And then if you do nothing, then you basically have a clone of whatever you already had globally. So then once you have that virtual environment and you source it, then you can upgrade or whatever you want in there, including installs in there, by the way. Yeah. And it'll be specific to that environment. So that's what I'm asking is like, did you do that too? Out of curiosity?

No. So the virtual environment I started was with like Python 3.8 or something. I don't even remember now. And so it did, it cloned it. Right. But then when I go to do a pip install, it would fail going and looking for some Xcode thing. So even though it was in the virtual environment, it was trying to reach out to OS-level type utilities that weren't there. The wrong version was there. I don't know. But that's what I'm saying.

A dumb question, but you did source that virtual environment once you built it? I don't know. maybe not so i i we we probably have to look at it you have to like there i'm saying that like i'm just curious you know yeah we'd have like we could all like backseat drive this thing but the cure because there is like a like once you create the environment you know it's not enough to just create it because like what i'm trying to call out is like, if you were in say like a, I think an NPM has this concept of like where, yeah, cause it has its own node modules right in the, in there. And so like, you don't have to do anything. Like it's just there for free. But with Python, that's not true. When, once you create that virtual environment, it's there, but unless you like, I'm saying source the environment,

cause I'm thinking of it from like a shell, you know, like a bash shell kind of environment, but I forget the exact way to do it off the top of my head, but you have to then like actually switch your, your executed command is specifically say like, use this environment as the source for all of my packages. So if you don't do that, then it's never, it's never using it.

It's still using the global. So that's what I'm trying to figure out. But I was using it inside PyCharm, which would have made me think that it would have already sourced it for you because that's how the project was set up. So maybe I wasn't. No, VS Code doesn't. VS Code doesn't work that way. Yeah, I don't know. Okay, whatever. The cool thing is that PyCharm has a way of reaching into a running Docker container, and that's really the coolness here. Yes.

Right? Yes. I'm just off in curiosity tangent land. That's fine, because I will go check that, because I'm curious about it now, because I wasn't aware that I needed to do anything. So I will look at that tomorrow. But the next thing I want to bring up, because I saw this while I was doing some of this Python stuff, is I didn't realize that YAML had so much stuff built into it. Like, I mean, let's say what BS you mean? Yeah, no, seriously. Like, yeah,

totally. Because I mean, typically, would you guys agree when you think about YAML, you're basically just thinking about space and then at things with some colons for some key value pairs and some dashes for some lists. I mean, yeah, it's offensive. Yeah. Okay.

So here's, I, I was looking at a Docker compose file and, and I'm seeing all these things in there, and I'm like, what in the world is going on? So I'm bringing this up because I'm not going to tell you all the crazy stuff that's in YAML. Believe me when I say there's more than you probably know. But one thing that's interesting is if you ever look at a Docker Compose file, lot of times you'll see environment variables or something that that might get repeated across multiple services. Well, if you have those, you can do something that's called, I think, anchoring. But basically what it is, is you can ampersand and create a variable like let me call it environment.

Right. So ampersand environment. And then you can have a list of 20 environment variables that need to be basically copy and pasted across a bunch of different services. So after you call it ampersand environment and you define it the first time, everywhere else where you need that stuff basically copied and pasted, all you have to do is asterisk and then environment, if that's what you named it. And it basically copies that reference and includes that stuff all over the place. That's a YAML feature. If anything, I would have thought it was like a Docker Compose feature, but it's not. It's YAML. So YAML itself has all these features built into it that you probably have no idea. I have a link to the 1.2

spec in here, and I actually have a link specifically to that anchor feature, but man, I'll put it like this, man. The page is so long, like you can scroll forever and you'll probably have tendonitis after you're done with it. There's just so many features on here. So, um, you know, check that out. If you live in YAML land, I never realized that, uh, YAML was like PHP, that it's a recursive acronym. Oh. But according to this link you have, YAML stands for YAML Ain't Markup Language. And I got to tell you, there's something about my OCD that's just bothered by recursive acronyms like that.

I can't. No, just tell me what it stands for. All right. no just tell me what it stands for all right so uh my turn oh i just want to say earlier so i gotta say my worst days with docker are still not nearly as bad as the worst days i've had without it so oh totally and it's just like we said earlier like so many of my problems with quote-unquote docker have really been with whatever i was trying to do in docker which was usually my own stupid stuff and if it wasn't my stuff it was whatever i was trying to put in there that was messed up um but uh yeah so hey uh great tip thank you uh alan for turning my into this tip from justino garcia uh which is actually really nice i did do a little bit of Visual Studio-ing today.

And so this is very timely. So there is an extension called Tweaks from Mads Christensen, which I got to imagine we've mentioned at some point in the show. They do a whole lot of stuff with.NET and Visual Studio, really great stuff. And they've got this plugin called Tweaks. And you can find it in wherever plugins are found. And it does a bunch of really cool stuff. it adds shortcuts it adds some really nice features one of the things in particular that alan brought to my attention was presentation mode which is basically you can

right click down at the bottom and say go into presentation mode and it will pop your visual studio into like a really basic presentation mode. It kind of hides Windows. It hides different plugins that you have, and it sets it back to basically almost like a default. It's not quite default, but like a simple layout and default colors. And you can configure this too, of course. But what's really nice is if you have a font that you like, and you like the dark mode, and you like to have 52,000 plugins, and then you go do a presentation and a user group or something, and you want to flop it into light

mode with a bigger font and hide all those crappy windows, then you don't have to go and do that before your presentation and then go set it all back the next day when you go back to work. You just go, you right-click, say, Presentation Mode, do what you need to do, and then flop it back to Developer Mode. And all your stuff gets back. You're right back to where you were. So that is really slick. And that is just one of, like, the 30 things that this plugin gives you.

The ability to reopen closed files, auto-saving. There's a command to restart Visual Studio. I mean, it's just a ton of really good stuff. Build time statistics. I mean, it looks like maybe even, I don't know, I can't count, like 20 items that are all just really good. I definitely recommend checking out Tweaks. It's, of course, free, works with Visual Studio,

and it's awesome. Is it just me, though? Because, like, I can't hit this. What? So it's not just me. Yeah, Visual Studio Marketplace is currently down. What? I got it. Hold on. Let me...

I got a bad link there. Let me call up a developer. Well, you know what? It's not 2 a.m yet i'll wait yeah right so uh yeah it's really good should get it i'll take your word for it because i can't see it so until this link loads then i don't think you have given us a tip of the week, sir. Uh, Hey, it just came up. Uh, you got lucky. Uh, yeah. Okay,

cool. So, so you could have your, no, why would I want my, but this is if I want, this is if you're doing like presentations, why you might want like a presenter mode for visual studio.

That's what came to mind. Just cause I know what it's like to kind of have things set up how I want it and then go do a presentation and I shut all the windows, I shut all the plugins, I try to basically get it as simple as possible so you can just focus on the text. That stuff could just be really busy.

Like I've mentioned, like IntelliJ, I love it. My gosh, it's so busy. There's literally text climbing up the walls sideways. There are menus climbing up the walls sideways in IntelliJ. And I think they've got a similar kind of mode for that too where basically just do presentation walls sideways there are menus right climbing up the wall sideways intelligent and i think you've got a similar kind of mode for that too or basically just do presentation where it's gets all the stuff it gets all the chroma anyway and i forget what they call it it's like nirvana mode or something but um so really this is so really this is like a a plug-in for all the

busy bodies like you and alan that go and do talks. But for the muggles like myself that just sit at their desk and do their job normal, would I get value out of that? You totally would. Who's the muggle here? Because I heard his presentation from you. That's how I go. And you don't even remember that drop of jaws. I don't know what you're talking about.

Drop of jaws. You totally can, man. So there's other things on here. It's got run code cleanup on format. Don't copy empty lines. Don't copy when there's no selection. Format on moving line down. It's got a ton of features.

There's the real. And I put the link back in there. If you click it now, it probably should open. But yeah, like Joe said, there's a ton of features. It just so happens that Visual Studio in general stinks for presentation mode because they don't make it easy for you to do stuff. So being able to launch another one like that easily is really nice. Well, in dark mode.

Because part of what I was trying to remember, too, is that I remember I used to have different presentation – or not presentations. You could save window layouts in visual studio. Yeah. And so I would have like a visual studio layout for like, here's my three monitor layout, or here's my laptop monitor layout,

you know, and, and I would just flip between those layouts and then not have to worry like windows would just magically appear where I wanted them in the size and whatever that I want. So here's a feature that you would love outlaw that's in this is in the output window for builds. You could set the verbosity level. So, you know, like an MS build, you could do it when you actually call it.

They added a feature individual studio to where you could actually go down there and you can say, Hey, give me diagnostic, minimal, normal, whatever. So yeah, there's, there's a ton of like little features in this that are really good. Cool. And last episode I learned about all of those different, uh, there's a ton of little features in this that are really good. Cool. In the last episode, I learned about all of those different levels of logging. So now I can understand it. That's right. Hey, this was going to drive me nuts.

But I found it. And I'll include this in the show notes for this, though. But when you create that virtual environment, so there's a script, a line that you would do where you do like Python-M to specify the module, V-E-N-V, and then whether or not you're in like a POSIX environment or a Windows shell environment, an activate script. And it's either going to be like an activate.bat or.sh or whatever. It's going to vary depending on your environment. But inside of that bin or scripts directory, you have to then run that script to activate that virtual environment. And that's what I was getting at before was like, if you never did that, then what you were doing is you were creating the virtual environment, but you were never using it.

And that's why things kept going back to your global config space. That makes sense. Yeah. I'll give it a shot tomorrow and see if that works. Yeah. I'll check back in on that episode. I'll include some, some notes in that one,

this one, because I was looking back at like episode one Oh two and the show notes that I had for that didn't include that. So I'll include it in this one. And then that way people can follow along and see what we're talking about for those that haven't ever used virtual environments in Python, because it is quite awesome once you do it, because you can truly have like a, you know, isolated little environment, much like other, you know, things already have, like, you know, we mentioned the Node modules for NPM. Okay, so I got a couple tips of the week. I channeled my inner Alan and came up with three.

So the first one, well, actually, I'm going to skip the first one for a moment, and I'll start with Visual Studio Code. And I think I might have hinted at it earlier or mentioned it briefly earlier related to some Angular development that I've been doing lately. And if you, like us, use Visual Studio Code and love Visual Studio Code, then there are a couple of plugins that are really nice to have. So one of them is the Angular language service.

I'll include a link to that, but it'll include like all the type ahead and tell us since you know, sugary goodness, except for your Angular bits. Right. So it's, it's quite nice. But then there's also another one that I'll include that is specific to your

language version, but it's called Angular Snippets that'll add like, you know, some support, snippet support for your TypeScript and everything. So I'll include links to both of those for your Visual Studio environment. But then the also one that is super killer to have along with this Angular development is a plugin for Chrome developer tools called Angular State Inspector. And what this will do is once you open up your dev tools, you could click on various elements in the DOM and in your elements thing'll, you'll have like when you click on it and you see like, here's the styles and the computed styles and the event listeners and things like that,

you know, those tabs that you'll see along with that, you'll have a new tab that's called state and you can see whatever you were using to like Pat that was passed into that thing. And I found it to be like, like for example, um, uh, for charts, you know, like to be able to see like, Hey, what data was I actually passing into this chart? Like maybe I'm not passing in what I thought I was passing in. And, you know, it's one thing to step through, like, and see like, you know, what you're building and passing into it. But sometimes it's like, okay, I know what I think I built and what I think I passed in. How did you actually interpret what I said? And so this is, it's, it's really, uh, helpful for that. So I'll include a link for that. And again, obviously that, you know, that's a Chrome, uh,

dev tool, which I wonder now, like, how does the Chrome, uh, web store work now that with, um, edge now that edge is based on Chromium. I wonder if you can install. I haven't ever tried that. Have you, guys? I've tried it with Brave. I know like with the Brave browser, you can go to the same Chrome web store and install stuff in Brave. So I would imagine if it's based off Chromium, you're probably pretty good.

All right. Well, so that's it. That's All right. Well, uh, so that's it. Uh, that's the show. I hope you've enjoyed it. Uh, that's all folks.

Uh, yeah. And, uh, you know, now we'll let our, our, uh,

NPM, our, uh, sorry, uh, NPR overlords take over and tell you, uh, please subscribe to us on iTunes, Spotify, or Stitcher, or wherever you find your podcast apps.

And be sure to leave us a review by visiting https colon forward slash forward slash www.codingblocks.net slash review for some helpful links. This hurts so much. So much. What's my favorite part? Hey, while you're up there, go check out our show notes. Examples, discussions, and more. And send your feedback,

questions, and rants to slash slack. Make sure to follow us on Twitter at CodingBlocks or head over to CodingBlocks.net where you can find all of our social links at the top of the page. Thank you for listening.