Coding Blocks - The DevOps Handbook – Enable Daily Learning

Episode Date: October 12, 2020

We dive into the benefits of enabling daily learning into our processes, while it's egregiously late for Joe, Michael's impersonation is awful, and Allen's speech is degrading.

You're listening to Coding Blocks, episode 143. Subscribe to us and leave us a review on iTunes, Spotify, Stitcher, and more using your favorite podcast app. And check us out at codingblocks.net. We can find show notes, examples, discussion, and a whole lot more. And you can send your feedback, questions, and rants to comments at codingblocks.net. Sounded like we got professional here for a second. Follow us on Twitter at Coding Blocks or head to www.codingblocks.net and like we got professional here for a second follow us on twitter at cody blocks

our head to www.codyblocks.net and find all our social links there at the top of the page with that i'm alan underwood i'm joe zach and i'm not that'll teach you next time you say we got professional that's right that's right you learned to be, didn't you? Nailed it. First try. Who are you? Not. You're not. Okay, good. This episode is sponsored by Datadog, the

cloud-scale monitoring and analytics platform for end-to-end visibility into modern applications. And Teamistry, a podcast that tells the story of teams who work together in new and unexpected ways to achieve remarkable things. And today we are talking about the third way, which is the third DevOps way, chapter 19 if you're following along in the DevOps handbook book. So with that, let's hop into the news.

So as we like to do, we have, we like to say thanks to everybody who left us a review, you know, in iTunes and Stitcher. I wish other places would let you leave reviews too, because then we could read all those too. But from iTunes, we have John Rowland, Shefodorf, DevCT, Flemmen001, Ryan J. Caldwell, and Aseum. Oh, hey, that's me. And for Cesar, thank you very much to Helia. Very nice. And I just remembered this, and I did it about a week or two ago, I think. Now, if you are one of the people stuck at home, working at home, and your bum is uncomfortable from sitting in whatever chair you have,

we've got a couple of reviews up on YouTube for office chairs, and I just did one. It's the one that I recommend to most people. It's actually called the Baby Miller or the Herman Miller is kind of what it's called. If you're interested, we'll have a link in the show notes here. Definitely go check that out. If you've never seen one of Alan's reviews, talk about professional.

Super good. Was it 100% thumbs up on YouTube, which is unheard of? Oh, man, you just. I jinxed it. Yeah, no, no, you didn't jinx it. Somebody listening is going to be like, oh, man. Sabotage.

I'm going to be number one. Yeah. Yeah. It doesn't happen that much. Anybody that's ever posted a video up on YouTube, you know, like to have a video up there for any amount of time that doesn't have thumbs down is impressive. So I think I might have broken a YouTube record for two weeks straight without a thumbs down so far i don't know we'll see well until until this episode airs yeah until this episode airs you're welcome destroyed yeah thanks thanks joe all right so this is actually part number 85 that we're doing on the DevOps handbook, right?

Yeah. You think it'd be like a DevOps encyclopedia set. I like the look on Outlaw's face too. He's like, oh, deer in headlights. Oh, sorry. No, I was looking at the, I was trying to remember because you said the baby Miller. And I'm like, wait, which one is the baby Miller? And I went like Googling for baby Miller and that came up as a worthless search. Then I was like, okay, baby Miller chair. Surprisingly, that didn't help. So then I was like, okay, which one is it? Because I was trying to not click the YouTube video because I knew it was going to like autoplay and you know, that would be part of the, know recorded forever and it was and uh but yeah so i

had to go to the youtube to get to it and then clicked the link there to see like oh what what chair are we actually talking about see i'm the point is the point of all of that whole rant that tangent is yes i was paying attention yeah good job awesome before we dive into part three or the way, I want to go ahead and mention what the first two ways are because I always have a hard time putting them into words. So I just Googled it again and see if I can remember them this time. So the first way was all about flow. You remember that was us talking about kind of like CI, CD type things and basically getting a moving pipeline. We talked about the Numi plant, doing things one at a time and doing things continuously rather than batch. The second way was all about feedback, which is like measuring metrics, using that to make informed decisions. And then the third way,

and experimentation and testing. And the third way is all about continual learning. That's what we're talking about today. And experimentation. And experimentation. Yep. So in the little intro section, really the only takeaway to this was when we talk about learning, we're talking about learning quickly, frequently, cheaply, and as soon as possible. That's really what it boils down to. And then we're going to dive into what each one of these means.

That's my favorite kind of learning. Right. And the way I've kind of internalized this is like the first two ways are really about getting your stuff together. You got to get it buttoned down. You got to get it right. You got to get your ducks lined up. And then once you've got things going so they're not tripping on the little tiny crap that drives us all nuts, once you finally got things moving, that's when you can start taking the big swings for the fences. Yeah, I like it.

So the first thing that we have here is what they talk about is if you really want to get good at working on complex systems, that you need to be good at detecting problems, solving problems. They call this multiplying the effects by sharing the solutions within the organization. And we'll talk about that in a second. Or actually we can talk about it now because I was last of the bullets. But so what they mean is when you learn something, share your learnings with the rest of the organization.

And then that way, everybody benefits from whatever you gained, right? And that's what it boils down to. And granted, anytime you're telling everybody else in the organization, maybe not everybody listens, but some people are going to listen. And so that knowledge is now starting to spread throughout the organization. Well, they specifically gave an example of Google and Google sharing their lessons learned back in the repo so that everybody could take advantage of those lessons. I don't remember any examples. Oh, God. You probably skipped them all, the best parts of it. They also gave an example of Netflix.

Did I did I move this in the wrong place? I don't think I did. So I think it's right. Yeah, they were talking about how some of these learnings that they would get. They used Chaos Monkey across like availability zones to to to create these problems. Right. availability zones to create these problems, right? And this, what they would do is they would take these failures as an opportunity to learn rather than to punish anybody, right? So, as they're going through this, that's really the whole sort of idea behind this particular chapter

is embrace failures as a way to learn how to avoid them in the future instead of trying to put you know throw somebody under the bus they they actually they're you know the netflix and the chaos monkey are going to be like a common theme throughout this whole uh third way i mean it gets referenced a few times because they talk about how you you know, uh, I think it was like 2009 where Netflix started re architecting their, uh, um, their infrastructure to be in AWS and then their approach to try to, uh, withstand random outages. And that was like, you know, what became of the chaos monkey. And, uh, you know, there's a, there was an example in here. They were talking about like, you know, they wanted to be

able to withstand an entire region going down and, you know, you know, spoiler alert, a little foreshadowing there, guess what went down an entire region in AWS. And like it, it, uh, took out like a lot of companies. It was a big deal back when it happened at the time. I forget what the exact, what year it was that happened, but it was a big deal when it happened. And a lot of companies were impacted by it. And Netflix was the only one that wasn't impacted because they had the chaos monkey

and they had already like tried like, oh, hey, what if we lost everything in this region? Can we withstand that outage? Oh, yeah, we'll make this change, make that change. And, you know, they would like learn some little things here and there. And then when an actual problem did come up, they were, you know, already able to, you know, withstand it. And it why Netflix didn't suffer an outage was because Netflix had some kind of special in with Amazon or Amazon was treating them special. And that was why everybody else went down except for Netflix. And it wasn't the case at all.

It was just they had architecture around AWS. They had forced themselves to go through artificial pains like that, which put them in the place to where they could handle it when it happened. You know, so Chaos Mesh is like another kind of item in this field that just came out. It's a CNCF project that does the same kind of thing. It's Kubernetes first focused. And I was same kind of thing it's kubernetes first focused but i was just kind of curious like chaos monkey comes up all the time like everyone knows the story of you know the chaos monkey and but uh i figure that that's pretty old now so i was just curious to see like what you know what kind of newer stuff that people were doing with that and

so uh you know i'm really excited about chaos mesh and that looks like something i can easily plug in but it looks like there's a company called a gremlin too that has a product that's kind of similar and it's paid service but other visualizations are really cool so it's just kind of cool to see like even though we talk about chaos monkey it's really just kind of a like the the poster child for a family of products and services you say that everybody would know about but i don't think that's fair i don't think that's true though man because i mean imagine like when netflix created that you know we're talking about a dozen years ago somebody could have been in like a middle school gone all the way through high school all the way through college the first they're into

the first couple years of their career and could have like never known about it like you know just taking it for granted like oh that that's a thing. Whatever. You mean they haven't been reading DevOps articles for the last 12 years? Right. What? There's so much else going on. I did add those links to our resources so those will be on this particular

episode. So if you haven't heard of these or haven't seen them, definitely go check them out. So I guess the next section we're going to jump in, and this one is really interesting. I like this one a lot, as a matter of fact. So the name of this one was Establish a Just Learning Culture. And so what they say is by promoting a culture where errors are just, basically meaning they're okay, right? It encourages learning ways to remove and prevent those errors. On the contrary, if you create an unjust culture where people are basically punished for having these failures, you promote an environment that is about bureaucracy, evasion, and self-protection, right? So people are going to try and hide if a failure comes up.

And basically what it boils down to is this doesn't help anybody. And the sad part is this is how most companies and management work. They put the processes in place to prevent or eliminate any possibility of errors, right? Any failures. And, and you'd, you'd think that was, that would be beneficial, but ultimately it hurts because now nobody wants to admit to what happened. And that means that you've created a culture of, of trying to sweep things under the rug when they happen, right? Instead of sharing what you've learned so that other people can benefit from

it. Or, or place blame. Yeah. Yeah. And I know the three of us have been in environments where blame is thrown around like crazy. And outside of the fact that you don't actually get to trying to solve and fix future failures that would follow that same thing,

it destroys morale, right? Like it, it, it's a road, it's a roadblock to progress because now people are afraid to push out more things because the more you push out, the more possibility there are failures. Like it's just, it's, it's a train wreck of things that, that go in a negative direction when you have these type of policies in place. Have you ever worked anywhere where a person was fired after making a mistake? I mean, not like a software developer mistake. HR mistake. Define mistake, sir.

Yeah, me either. I was just kind of curious like when i was uh you know like younger or whatever i always kind of assumed like people got fired because they would mess something up but uh and i've messed up some pretty big stuff but um in my experience like whenever someone has really made a big mistake um even with places that have had like nasty kind of blaming cultures uh i've never really got like let go for something like that so i was just curious you know just because i always kind of had that impression yeah i don't think so i can't think anybody that ever got let go because of doing something that caused an outage or anything i mean there's a difference between like you're trying to do your job and you made a mistake you

know because that's what software development is right like you know you're doing your job and like okay you made a mistake while trying to do your job like you know that that's different than you know where i went with it but i don't know though i mean like maybe you know i don't know like remember there was that uh you know the aws outage where like it was just like uh like a routing table or something that got yeah it was a configurator you know i don't know maybe i you know i would hate it for that person if that's what happened but yeah maybe that kind of level where your mistake is like that costly then maybe it did millions per second but there again i i'd hoped that person wasn't fired right like i think joe you're about to go there with that too yeah totally yeah and i you know i think if if that person had like a long track record and had been spoken to several times and had on you know on

their reviews that they need to be more careful with something and then this happened then yeah totally but it wouldn't right in my opinion that wouldn't have been because of this incident it would have been like because of the incident and the history yeah so along those lines i probably have it somewhere in the notes here but that's one of the things that was interesting. This is a story of somebody at Netflix that had brought down Netflix twice within like a, it was either eight or 18 month period. Oh my gosh. deal with this guy. He did that and it stunk, right? Like nobody enjoyed that, but this was also an individual who was responsible for really moving their platform forward. It like in massive ways over time. So yeah, he had a couple of failures, but he also had some major successes. And the only reason that, that these things happen is because they were allowed to iterate

and experiment and do all these things. Right. So, so you would hope that with some of the bag comes a whole lot of good, right. And you want to, you at least want to weigh on the side of that. I just feel like bigger companies are better at kind of, uh, averaging out that sort of stuff. Like with,

if you've worked in a small company, there's lots of great reasons to work in a small company. If you work in a tiny start, small company or startup, like if your boss just gets mad, they could just fire you and right that's harder to do when you get into kind of bigger corporations like i mean bad stuff happens in bigger companies like that boss might try to force you out and just make your life miserable for months but it's less likely you're just going to get fired on a whim i think yeah so this next part that they say here is when these failures

happen take that as an opportunity to improve the systems that have the problems. Right. And and they went on to say, like, the psychological part of this is not only is that improve the system because you're doing that, but this also improves the relationships between team members. Right. And and I know all three of us have been there. Like when, when people aren't pointing fingers and people are trying to jump in to help and resolve issues and you do that, everybody feels better about it. Right. And especially if you take those steps to ensure that it doesn't happen again, instead of everybody feeling like, you know, getting the imposter syndrome and being like, man, I'm terrible. Now it's, Hey, we're all just trying to do the right thing. Right. Like we're all trying to make sure the systems are good. Imposter syndrome was specifically mentioned in the book.

You don't want to enforce that within your team and within the culture of your team. You want people to be willing to take a chance. And if they make a mistake, it'd be okay. You want people to experiment and, and, you know, not feel, feel bad about it because otherwise, like, like you said, then if you, if you have this environment where you can't take a chance and you do make a mistake and you're going to be blamed for it, like, you know, you're going to feel bad and you're going to feel like an imposter because like, oh, why did I mess up to config? And even if you're not, even if people aren't pointing fingers at you, almost all of us take that guilt on anyways, right?

Like you release something that you know was, oh man, how did I miss that? You know, like you already have that in you. I mean, countless times times like it doesn't even have to get to production it's just something as simple as like you know you you you committed something and for whatever reason it it it didn't break the build necessarily and it got past the processes and then you know maybe like a random deployment happened where it's like oh well if you if you deploy and then you do these three steps i can't even do my job anymore it's broken like what happened right i mean that happened that happened to us this week right where it was like

you know one of our teammates was like oh man i tried everything like how did that happen and it was like oh yeah if you if you happen to do like this order of things then you know and you gotta you know whatever you take that as a lesson learned and and learn from it and that's where it's like uh if you any any of those lessons that you can incorporate back into your process or into like a unit test or whatever like if you can trap that kind of thing it's so much better. And one thing I wanted to mention too, like when it comes to major outages and like major services is like usually one failure,

there's no such thing as one failure if it takes out a whole big thing. Like not only did the problem happen, but nothing was able to protect itself or recover or move on from that that so it wasn't just one failure that led to a failure it's always at least two now if you're you know if you've got like a small jamstack website or something and the website's down then you know i guess i don't know you should have had a backup but uh no uh there's always more to that but you actually just reminded

me and i wanted to go ahead and get this i'll get the link so remember uh was it last episode of the episode before we talked about an article um basically my unit test being overrated oh yeah so um we published the episode put the twitter link out and i don't know if you you all saw this but um the author uh listened to the episode and wrote back and basically disagreed with some of the stuff we had to say and i mean it was all really good like um so i i don't want to give commentary on their opinion just because you know like i kind of want people to be able to just kind of read what they had to say about what we had to say we already kind of had our piece so i thought everything he had to say was like you know really good and really relevant and legit and so i think it's all really good so i have a link here uh you

know ultimately i still feel like it it's almost like good so i have a link here uh and you know ultimately i still feel like it it's almost like we kind of came down to like whether or not we felt they were literally overrated overrated or underrated and so i still feel like i'd like to see more of in the world and uh it feels differently and that's okay but just the discourse was really good so i'll get a link here yeah and it is funny to the the link that'll give you for Twitter. Like the dude spent a lot of time replying back to it because as you know, Twitter's not the friendliest place to try and reply because you're cut off on characters. But I think more or less what he said is he thought that we misunderstood or, you know, kind of misrepresented some of what he was saying.

But he said that was also a common problem with what his article was mainly because of his title, right? He said overrated. I didn't mean it in the negative connotation that it was taken in. He said, but that's actually the word that I was trying to use. And so he goes on to explain a lot of it. Yeah. That's why I kind of want to, I don't want to like debate what he had to say a third time. Cause it's like one of those things like where we got the microphones and so it's not fair to just kind of blow out our side so you know like if if you're interested i would recommend going out and checking the tweet and you can see his original tweet to us and then

there's a link to uh the bigger thing that he wrote so it's just really good and it's all just really interesting so go get it yep totally um so this next piece that we had here is they said when developers do cause an error, they are encouraged to share the details about the errors and how to fix them. Because if you do that, then everybody benefits from it, right? That's what we talked about a minute ago. And by doing this, what they actually suggested is after any time there's a failure, you have what they call a blameless postmortem. So I think all three of us have been part of postmortems where basically, hey, how did this happen? What can we do to stop it? Right. And most of them I've ever been in, they're pointing fingers. In this case, they're like, no, you don't have that. You have this blameless one. And it's interesting. Another

thing they say that you need to do is you need to have controlled introductions of failures into production, which is really interesting because this is their way of basically finding out in a way that you control what shutting off an availability zone would do or making some other things. We'll talk about that a little bit more here in a minute, but it's really weird to hear that. But on the blameless postmortems, let's dig into that real quick here. So one of the things that they say that you need to do is when you go into these meetings, you need to create timelines and collect details from many different perspectives, right? So if the three of us were working on something, hey, when did X go wrong? Okay. The first time I noticed it was this time. This is what I saw happen. And then outlaw, what did you see happen? And Joe, what did you see happen? Right?

Like get it from everybody's perspective so that it's not tunnel vision on any one person. Right. Like, oh, well, uh, I was not here today.

So, uh, I don't know. I think it was probably his fault. That's right. I thought it was blameless though, sir. What happened to that?

Hey, we didn't say that we're good at this. Oh, we're still learning. And that's, that, that was our fault. Our bad. Oops. So one of the other keys to these, these postmortems is empower the engineers to provide the details of how they may have contributed to the failure. Right? So again, we're not placing blame, but Hey, I, you know, I reviewed the code and I didn't see that this was over here, right? Or maybe I wrote the code, but I didn't know how to test for this one thing. Like who knows what it is, right? But basically put the details out

there because by identifying those things, you know what you might need to look for in the future. Anybody want to take the next one? Sure. Encourage. Oh, Nope. Yeah. So as I was saying before I was so weirdly interrupted, right? No, I'm sorry. This is a blameless post-mortem. I wasn't, no, no fault there. Encourage those who did make the mistake to share with the organization and how to avoid those mistakes in the future. And, you know, don't dwell on the hindsight, like the coulda, the woulda, the shoulda, you know, like there was a problem, fine, you know, figure out what caused

the problem, address it and move on, right? Like, oh, my God, but we could have had this, or because of your stupid mistake, we lost, blah, blah, blah, blah, blah, blah, blah. It's like, okay, I mean, we can get mad about the past, or we can just move on and try to make progress. What are we going to do? Right. Also, proposing countermeasures to ensure that similar things don't happen in the future it's also a really big part it's like what could we have done or what can we do in order to

mitigate or just fully stop this and uh setting a date i think is a really big part of this one like it's one thing to talk about like yeah we should do this or even like writing a ticket and not like assigning it to a sprint or whatever it It's basically casting it into the abyss. So I'll put a date on that sucker. Yeah, man, that is so big because I know all of us have come out of meetings where there's action items, but there's no priority. There's no, hey, when are we going to have this done? And if there isn't any of that, the abyss that we're talking about is the backlog that everybody is so fondly aware of in Agile or whatever, right?

Like it just disappears. And then tickets become like CYA. Oh, we got a ticket for that, so you can't get mad. Or what about, you know, you do come out with a bunch of action items, but it's like, yeah, but my current queue is still my priority. So, I mean, sure. Go ahead. Add it to my queue.

It'll be six months before I get to it. And I think the sad reality is typically when there's a failure, it's all hands on deck, right? It's the fire drill. You get this stuff knocked out. And then as soon as it's done, then it's almost like in many, many, many, many cases, it's like almost discarded, right? Like, hey, hey, we got past it.

Moving on to the next feature that we got. Web sites back up. Yeah, right, right. Now we're done you want lunch let's go so yeah put a date on that thing commit to it i thought you were saying put a ring on it commit to it that's a whole nother set of problems you could be a cat or it could be bliss right it's either wow i'm surprised you went negative with it first. That's the funny part. Well, I mean, I was putting on my comedy hat. Oh, okay.

Yeah, it makes sense. It makes sense. Obviously sarcastic. Right. Yes, totally. No, I personally love my bliss. So, all right. So the next thing, the stakeholders.

CYA, they just happened. I don't like a post-mortem in real time. My wife listens to these podcasts. Oh yeah. Yeah. Love you. That's hilarious. Totally, totally not. I was played. I was totally kidding. All right. So, um, yeah, the next one, the stakeholders that should be present at the meeting, this one's kind of interesting. So the people who were a part of making decisions that caused the problem, they should be there. Um, the people who found the problem, the people who responded to the

problem, the people who diagnosed the problem, those who were affected by the problem. Now, this is interesting because this starts stepping outside of maybe the engineers, right? This is, I don't know, were you writing a customer service app? They should probably be there. And then how about this one? Anybody else who might want to be there? Yeah, that one was crazy. Right?

And if you're thinking like, but I work for a small company and all those first ones just like, okay, all of those are going to be me and my boss. Who was responsible for making the decision that caused the problem? Me. Who found the problem? My boss. Who responded to the problem? Me. Who diagnosed the problem?

Me. People who were affected by the problem? My boss. me people who were affected by the problem my boss i like uh people who were affected by the problem are often left out of this sort of thing too i really like that a lot of times i think um the reason why they want the people in the meetings is because they have different perspectives and sometimes like they might be kind of surprising to the people who like found or fixed it like yeah especially um i want to hone in on the people who found the problem you might think like well that was just some person in customer service who got a phone call. It's important to have the person in the meeting because they might say, well, we get calls from time to time, but most of the time I don't think it's serious.

Or whenever I get a phone call like this, I need a bat signal to be able to push up because it's almost always something very serious. Any customer has seen it and thought to call. There are probably thousands of customers that didn't see it. So that kind of alerting is really important and also um people who are directly uh affected by the problem so if marketing was down because the server went down you might get them in the meeting and then realize like hey why the heck do you even care like why did you go down if the mail you know smtp went down or something and you might be able to kind of realize that maybe there's just some like some dumb dependency that you didn't even need to have in the first place that you could just totally prevent

that problem in the future. And you may not have ever realized that this whole thing wasn't even needed if you didn't have the people in the room to say like, I wasn't even trying to send an email. What the heck? Right. Yeah, that's a really good point. And anyone who might want to attend, that's kind of valuable too, because maybe those people are just trying to get insights into how things operate, right? Or maybe you can provide insights to them on the kinds of things that you do. And they can be like, oh, you should never do that, right? Like you might get valuable feedback and you might provide valuable feedback that nobody would have been aware of. Or if you're a pessimist like me, you're just're just thinking like oh everybody just wants to see who was the dummy that made the mistake yeah it's voyeurs

but nobody's gonna know nobody's gonna know it's like hey i'm here for the arena fight the pit fight what are you talking about that's right they're gonna be sitting there with their popcorn like that that's how you know like who's just there to attend that might want to they're they're the ones in the corner with the popcorn it's like are the interns here because they just don't feel like working or are they here because they're actually interested in how we figured this out and what we did about it? I don't know. Yeah. So now let's talk about the meeting itself and what should actually go on in it, right? So we now have identified that everybody in the company can be there if they want to. Um, so what do you need to do? You absolutely have to be rigorous about recording

all the details of your findings, how you diagnostic, what you did to fix it. Like all of it, you need to be just hyper detailed about it, which I'm going to be completely honest with you guys. Like I am terrible about that because I, and I, and I hate it, but usually when I'm problem solving, I'm, I'm constantly clicking through trying to find, okay, this is what happened here. I what's the next piece. What's the next piece. And then I don't take the time until I get to the end of the trail to say, okay, what did I do? Um, and then I try and rewind in my mind and say, okay, this was a, and I'm sure that I missed some things along the way. You know what I mean?

I mean, we're not, you know, with this, these meetings that you're going to have for this, we're not talking about like, oh, hey, there was a slight bug. And, you know, there was, it was a slight CSS bug. So the logo was a little bit off center. I mean, we're talking about the type of bug where like, you know, there was, it was a slight CSS bug. So the logo was a little bit off center. I mean, we're talking about the type of bug where like, you know, money was lost because of a mistake. That that's the only time that I've ever seen these types of meetings happen.

Money is lost or customers are mad. It's one of the two, right? Like that's really what it boils down to. In my, I'm just saying like, I've never seen it because customers were simply mad it was because money was lost and maybe customers were were mad and so therefore they didn't shop but yeah like you could don't directly see like oh hey a deployment went out and uh all of a sudden we started making less money right and this is a super important like uh and you imagine a marketing department uh was uh facing outage because of something that happened in it and say they missed some deadline or they missed some you know goal for their bosses and so they have to go explain like why they didn't meet their quarterly target or whatever and they say uh it was down three times

this month what the heck like we can't do our jobs then it's really important that it be able to say like we were down three times it was down for this long and here's what we did about it yeah i mean that's the only time i've ever seen like these kind of postmortems happen where you would like document like what even happened and and you know you think about like uh think about things that happen in like real time like uh like a broadcast news or live sports, right? If you have something that causes you an outage during that, it's, it's a lost opportunity, right? You can't, you can't go back. Like if you, if you're unable to play a commercial at a time that you were supposed to play it, you know, not only you're going to have to like make it up. So like you've lost that

opportunity and, and you know, it's only in those kind of that type of live uh situation where i've like you know seen uh where where where we've had these you know because it was a big deal because you know money was involved yeah that's true i've seen an ulterior motives too um there was once uh there was a case where we went down and the boss was strangely worried about how much money he thought we lost. Like when he wanted to make sure that we like had a good estimate on how much the, how much bottom line was affected by the revenue.

I was like, doesn't this kind of look bad on us? You know, make a higher, uh, people want to shop. He was like, he,

he seemed to have a, an interest in getting that the dollar value lost hired. I would, I thought that was the weirdest thing. But then next, you know, he's like, he turned that into a play for a more headcount. So kind of like a chip we were going to say like hey look

we lost 500 million dollars because you didn't give me that extra three headcount that i asked for last quarter good job uh nice nice reversal yeah so one of the other things that needs to happen in these meetings is you are not allowed to say could have, should have, would have, whatever. Again, it's counterproductive. And I really like this rule, right? Focus on what happened and what you can do to fix it. That's it. Yeah, because I actually, I love this point too.

Because they say that if you use those kind of terminology, you're focusing on what could be rather than what it is. Right. And instead, that's not helpful. It doesn't matter what could have happened or what should have happened because that's not the system you have. And by using that kind of terminology, you're thinking about some dream system that you wish you had instead of focusing on this is the world we live in. This is the system we have. Here's how we deal with it. Yeah, I totally agree. Another one that I like here that I don't think has probably done enough justice in most

places, most of the time, reserve enough time to brainstorm the countermeasures that you can implement to address these. And I know that seems kind of rough, right? Like how long are you going to make these meetings? But if your goal is to truly improve the systems that you guys are delivering, then it kind of makes sense for people to be able to put those ideas out there and spitball and get some of those things down. I would love to see one of these meetings, the way that they're describing it in the book, you know, because like I've never,

like what happens is at least an, a minimum of an hour is set aside and, you know, my, my, uh, I guess just unfortunate, like where I've,

where I've seen these go, they, they are not, you know, there's definitely blame going on. It's not this blameless environment that they're describing in the book. There's a lot of, you know, finger pointing and could I, should I, you know, conversations happening. And, you know, you probably spend at least the first half hour of the, uh, you know, or the first half of the meeting, just getting through all the emotions of like, well, so-and-so did this and

it shouldn't have been blah, blah, blah, blah, blah. And then you're like, by the time you actually get to like, okay, well, you know, let's try to be more productive. Then, you know, the rest of the time is just focused on, uh, you know, here's how we found the problem and here's what we did to fix the problem. Time to reserving time to brainstorm. No, no, no. Never seen the first. First third of the meeting is you coming at me with the should haves. And then the second third is me coming back at you with the I would haves.

But and then the third third of the meeting is actually productive. Yeah, that's about right. It's not terribly far off. But again, that's a culture thing, right? I mean, this goes back to Outlaw, my favorite part of the book, are all the case studies. And they talk about, I believe Target was one of them that was in there, to where they focus on, hey, what can we do to make this better going forward? Etsy, another one, right? Etsy is huge on this whole process of let's work on how we fix these failures moving forward, right? By identifying the things that we could have done better. So the thing that they say though is

these things that you brainstorm, again, you need to schedule, you know, prioritize the ones that you think have a lot of, have some good teeth that you can put out there and then, and then put it, put it on the schedule, make sure that there's a time to get it done. Um, now this is where I thought things got really interesting is now you need to publish what you learned, the timelines, all the stuff from that meeting to your entire organization. So we kind of joked about this in the past that this stuff typically goes in like a wiki, right? Let's say, and if we, at our current. I don't say this, you say this. So many people have taken this and sort of coined this the wiki is where information goes to die you say i don't do that yeah i don't believe you which is great when we talk about like

post-mortem and morgue yeah right so what stinks about it is and what we mean by that is you spend a lot of time writing wiki documentation, right? Like how do you do this? How do you set things up or anything, right? So the problem is if somebody goes to do something, they'll be like, hey, does anybody know how to do this? Like, hey, did you search on the wiki? No, I didn't think about that. And it's like, really?

There's like 1,200 pages of information out there. Why did nobody decide to search it? And that seems to happen more often than not. But along with this, they actually mentioned that Etsy had this problem where these postmortems and the that people can easily and quickly go and search for these findings from the past failures and whatnot. So again, Etsy throwing some good stuff out to the world in terms of open source software that they've created. Today's episode of Coding Blocks is sponsored by Datadog, the unified monitoring platform for real-time observability and detailed insights into Docker performance. Yeah, enhance visibility into container orchestration with a live container view and easily detect clusters that are consuming excess resources using an auto-generated container map. And you know how we're all about measuring those metrics and service level objectives?

Well, I was just cruising the engineering blog, as I'm like to do for Datadog, and was reading about how someone was using custom metrics and open source software to, and Datadog of course, to monitor their boat and to use them, help sail the seven seas. And it's just a really cool article. And you can see how they use Datadog and all the visualizations. And, you know, of course, I was on the blog too. I had to see what else was going on there.

So the Istio D visualizations are really cool. And of course, I've mentioned several times the Kubernetes Docker serverless. It's just crazy. And then the list of integrations is just insane. So you just got to go to the website and see what people are doing with this and just be inspired. Yeah, it's awesome. Out of the box, Datadog collects critical metrics from each Docker container so you can get immediate visibility into aggregated and disaggregated service level traffic.

Yeah, so we love Datadog and you will too. So you should go ahead and try Datadog today by starting a free 14-day trial and receive a Datadog t-shirt after creating just one dashboard. Visit datadoghq.com slash codingblocks. Again, that's datadoghq.com slash coding blocks to get started today. All right. So it's that time of the show where we ask you, if you haven't already, please, if you want to give back to us, take the time to leave us a small review. It truly puts a smile on our face and it lets us know that people are appreciating it out there. And it actually

helps us, right? Like, no, we probably don't get more downloads, but it is nice when somebody goes and looks at a show and sees that there's not just one person that likes it, right? So yeah, if you get the chance and you're listening to this and you get back in front of a keyboard, head to codingblocks.net slash review. And we've got links there that will allow you to either do it for iTunes or Stitcher. So pick whatever place works for you, but we greatly appreciate it. And we thank you for everybody who has done it.

All right. And with that, we head into my favorite portion of the show. Survey says. All right. So a few episodes back, we asked, what's your favorite mobile device? And your choices were an iPad, the tab father of tablets, or an Android-based tablet, great hardware specs without the hassle of long-term support. Or a Kindle. It was on sale. Or Chromebook for the win. Not quite as portable as a tablet, nor as useful as a laptop. Or a two-in-one laptop, a giant bulky tablet that can run Docker. All right.

I can never remember who went last. So how about, I think maybe Mathema Chicken went first last time. It seems like Mathema Chicken went first last time. I think it did, yeah. So, Alan, with your new hairdo there, you're going to have to see the video for this. You like? Yeah. It's good. i'll let you go first what's your pick and by what percent do you think it won so i can't help but feel like these answers were slightly biased just a little bit biased but

i personally i would have loved it if people had said the two-in-one, but I know that's not going to be it. Anybody that has any self-love will say an iPad, the tab father of tablets. So we'll go with that, and I'm going to give it – I think this is a healthy win here. I want to say 51%. So you said iPad? iPad, yeah i've had it 51 ipad 51 so mathemachicken what you got uh i'm gonna go with um the sixth option none my phone is bad

enough with 70 uh no i guess i mean i guess i'm gonna go for laptop just to be different with uh geez um it's 10 percent 10 to win all right so so alan's gonna go with with an iPad at 51%, and Joe's going to go with two-in-one laptop at 10%. Is that right? Yeah. Yeah. All right. Well, now drumroll, because this is going to be shocking.

Alan wins. And I know you're thinking like, how? The Mathema Chicken clearly had this one. And yet, you're so far off. But... Who was closer on the percentage, though? Alan, you did overshoot the percentage. Oh, did I really?

What was it, by the way? So, you know, I guess because you overshot, that does mean that Mathema Chicken wins. Of course it does. No, I picked the wrong answer. He didn't win. Of course it does. I was closer.

So, iPad was the top choice at 30% of the vote. That's it? Wow. Oh, a second. Oh, no. The suspense is killing me. Yeah. Oh.

I'm having network issues over here. I said the iPad was the top choice at 30% of the vote. Yeah. What was number two? I mean, 30% is not as healthy as I would have thought it was. I mean, this one's going to be a shocker, but but of course it was an android-based tablet as second at around 26 of the vote which makes sense because you know we're talking tablets and those are going to be the top two tablets yeah any android tablets have good screens now

oh there are some with great screens the problem is they have terrible updates and they have terrible longevity yeah as an owner of several not cheap Android tablets, I will never buy another one. It will only ever be iPads going forward. That's the problem with them is that like the hardware specs are great on them. That's never the problem. Yeah. I mean, I'll put it like this for anybody that. We also, if you remember, uh,

go ahead. Oh, sorry. You came in and out again. So yeah, no, go ahead. You're going to put it like this. Oh, okay. So I will say I have had many Android tablets and the max shelf life on them is two years. I've had iPads and maybe you

say two years is fine, right? But, but that drives me crazy. I've had iPads. I have an iPad that is seven years old that you can still use. It's not as fast as a new one, but you can still turn it on and use it. The Android tablets after about two years, you can't even really use them. Like they just, they're there for all intents and purposes, a paperweight at that point in time. And forget about a Kindle. I mean, you can't even get through the holiday season with it. Still be nice.

Those things are awful, man. But if you want to give something to your kids as a screen, you don't care if they break it. The Kindle is a perfect option i'm having some technical difficulties they're very funny for us though um i just stood up he's he's killing up like his kids tv okay so uh yeah i had some some networking issues there, but hopefully, hopefully that'll be the end of that. Uh, you know, cause I mean, with the pandemic coming to

an end and like everybody going back to school and work and everything, like, you know, all the global internet can, can settle back down to what it was before. I think we're done. Um, we's the takeaway. That's the real takeaway. Okay. Um, we wish. So if you recall, uh, in the same episode that we had the, uh, uh, survey about your favorite mobile device, Go's super secret surprise survey of Go or Rust. All right, Mathema Chicken. This one is all you. You got two choices here. Give me your favorite pick. And by what percent do you think it won?

Go or Rust? I think Go won with at least 49%. You think it had a 49% lead over Rust? Oh, no.

You want to know how much the lead was? Yeah, the lead. Oh, point. I think it, I it i'm gonna say 25 lead okay so you're saying that you think that go had 75 of the vote no go had 49 of the vote with 25 lead over russ get with the program man i'm sorry'm sorry. My math isn't, you know, your skills are far superior, sir. So, yeah, okay. It might take my head a minute or two to get wrapped around that concept. So, you know, forgive me if I mess that up again.

Alan? Right, right. I already looked at it. Oh, Alan. Yeah, I can looked at it. Oh, Alan. Yeah, I can't answer this. We're going to have to go with what the Mathematica chicken said here. Yep. That's what it is.

Fine. That's what it is. The confidence is brimming over. All right. I'm going to win this one. Well, let me say that, Joe, you won because you were the only one and you didn't overshoot. So, you know, hey, good. Good job.

I like your strategy there. It was go at 64%. Now, here's what I find interesting about that, because if you recall, when we discussed the stack overflow, uh, 2020 survey, uh, you remember how I had like this, this streak of like, I kept picking the right answers. Right. And this was one of them that we were talking about, like, you know, what was the top language or something, if I recall correctly. And, and I had selected rust as being the top language. And according to Stack Overflow, that is like the most loved and most wanted language. Well, no,

not most, most wanted. Sorry. That was, that was Python was most wanted, but most loved was Rust. And by 86% of the Stack Overflow survey. Yeah.

By all three developers who use it. Yes. No way, man. Like that's Stack Overflow survey. Yeah, by all three developers who use it. Yes. No way, man. That Stack Overflow survey, I mean, I forget how many thousands they said responded to it, but it was a pretty respectable survey they had there. And so I found it interesting that it was like, you know, when we asked the Go or rust that go beat out rust so

yeah no wait wait where did go beat out rust though rust is showing still on that link 86 and go is showing 62 yes but but in our the people that responded to our survey. Oh, ours, ours. Oh, is that what you thought? You looked at the Stack Overflow link? Oh, yeah, yeah. Oh, Alan, you could have participated in the fun. I was totally wrong.

Yep, too bad, so sad. Dang it. I lose. Fine. I lose. You know what? Fine, Alan, fine. We just can't lose. Fine. I lose. You know what? Fine, Alan. Fine.

We just can't have nice things. That's right. So we'll just move on and we'll talk about this episode's joke. I got to trick you up every time. I got to figure out a new way every time. So this one comes from super good dave and uh he he wrote this one in that he found on twitter what did yoda say when he saw himself in 4k

yo that's a lot of your hair okay now keep in mind this is very this is very timely because like mandalorian season two is coming out soon i don't know if you've seen the trailers for it no spoiler alerts don't tell me because i purposely didn't watch it i have no idea what this answer is going to be oh really joe nothing how about i'm not going to try i'm not even going to attempt to do a Yoda impersonation, but I will say that when Yoda saw himself- No, you got to do it. Come on, man.

All right, come on. Let me clear my throat. Yeah, right. Can I even do a Yoda? How would Yoda even sound? He would say something like, do or do not. Okay. So when Yoda

would see himself in 4K, he would say, H-D-M-I. Alright, thank you, Daveave i like it all right so uh for this episode's survey we ask how often do you change jobs and your choices are job why would i do that when I can boss myself around? Or, I don't want to. Interviewing is awful. Or, every three years, like the Stack Overflow Survey tells me to. Or lastly, about every five years after I've built up enough embarrassments. This episode of Coding Blocks is supported by Teamistry, a podcast that tells

the stories of teams who work together in new and unexpected ways to achieve remarkable things. Teamistry, if you haven't heard it, each episode of Teamistry tells a story, and in each story, you'll find practical lessons for your team and your business. And it just so happens that I got a, we all got a super sneak preview of episodes of season two coming up. And I got to tell you about my favorite one, which will be out. It's episode four. So it's coming up pretty soon here, but it's not been released yet. But it's super high quality. It's like, It's like listening to a documentary. But this episode was about using machine learning and AI to track animal populations like zebras and whale sharks.

And so not only was it about the technology, but also about how they got the right people involved and were able to use that technology and put those teams together in order to make it all work. And it was just really fascinating to listen to and i cannot emphasize uh enough just how great the quality was i was like entranced to you know we heard of those driveway moments where you like don't want to shut off the car because you want to hear what they have to say next so uh that's that's the type of uh listening that we're talking about here and uh it's just really inspiring and ties into a lot of the things that we talk about like you know why population control and estimates are important and why and how to get people involved and how to just organize something that's so big and so important so definitely recommend checking

it out yeah i too also got a sneak preview of season two which i think this episode actually is out and this was about the Seiko dueling factories, which was a fascinating story. Like truly again, like what, what Joe said is the production quality, super high. And,

you know, it's, it's the right amount of information plus, you know, takeaways that you could do. And I thought this one tied in really well to this episode, which is where Seiko basically encouraged their different factories to experiment and almost have like competitions, right? Which, which helped grow the company by saying, Hey, it's okay to fail because it's

through these failures that we're going to find the way to do things. Right. And so it was just, it was a perfect parallel to this. And the story was told extremely well. I highly recommend go checking it out. If you like, if you like those type of episodes where you learn something and you walk away with useful information, this is a great one. And, and you know how Alan and I both love the case studies and the Seiko one was just a great, it was such an entertaining, great case study of like what they did. So yeah, these are stories that entertain packed with business cases you can actually use. Season two of Team History is already out as Alan said, like with that, the Seiko episode is already available. Search for Team History anywhere you listen to podcasts. I will include

a link in the show notes and our thanks to Teamistry for their support. All right. So now we're talking about finding more failures as time moves on. And the deal here is that as you get better at resolving egregious at finding real bad errors, the errors become more subtle. Egregious. At finding real bad errors. The errors become more subtle. E-Greg errors? Greggy Aries. All right. It's too late.

It's too late for big words. Big words. It's 1025 PM, my time zone, yo. It's bedtime. Jeez. 1025 p.m. my time zone, yo. It's bedtime. The errors become more subtle and you need to modify your tolerances to find weaker signals indicating errors. And I hadn't really considered that very much. So that's really interesting.

And you know what kind of stinks too is sometimes those subtle bugs that are harder to find are also harder to fix. Oh, yeah. No doubt. too is sometimes those subtle bugs that are harder to find are also harder to fix oh yeah no doubt what what was cool is in those white cases that are those uh case studies that you don't like is the story about nasa that they were talking about why the columbia blew up coming back in it was a loose piece of foam and and it was basically one of those things where they're like oh yeah this stuff happens all the time like they would look look at it. They, people had noted it and said, Hey, that's up there. And they're like, eh, whatever. It's not a big deal. Right. And that's kind of what we're saying is once you start getting really good at fixing the big failures, you need to start focusing on

the little things that can be problematic because that can actually start paying some dividends for you in the future. And that's where you'll find like comments in the code from Alan where it'd be like, I don't even know how you got here. True story. This should never happen. Oh, this is, so I know we marked this as it was done, but here's the one thing that they wanted to point out is NASA at the time was very focused on, um,

these stringent, um, compliances and standardization, like making sure that you checked ABC and D, right? Like they have these checklists. That's what you needed to check. And that's what you stuck to.

And the whole point of this finding the weaker signals is don't get so rigid and process driven in what you're doing. Always be looking for things, always be experimenting, always be trying to find the next thing that it is instead of, Hey, I have this checklist.

That's all I need to focus on. Right? Like don't get stuck in that rut. All right. So looking here now, uh, redefining failure and encouraging calculated risk taking and uh you know we talked a lot about failure on this episode and how important it is to create a culture where people are comfortable um talking about mistakes and uh surfacing and learning from failures because if you are nasty with people when they find problems and

you're encouraging not to find them or to rather uh to fix them uh without really uh letting the appropriate people know and having the the appropriate response and so there's an example here that i didn't read so i'll talk about that it's the one we talked about earlier with the the guy oh right who brought down the system twice in 18 months. So yeah, I mean, it happens, right? Like it's, it's not,

but yeah, it's going to happen. You got to give it up for, for Joe's honesty though. There it is. And the show notes, he's like, I didn't read it.

So I don't want to talk about it. Case study. I'm like, yo, I believe you. I know you wrote this book around them anyways so it i mixed up white paper and case study earlier which i don't know why i do that all the time like i'm always thinking white papers no what's the difference that's the technical bits

white papers are like the technical bits right like a case i Actually, I don't know. Maybe there's not a huge difference. Can a case study be a white paper? I Googled it, and your Google results may vary, but the top result came back. It says a white paper provides the benefits and rationale for the implementation of a proposed solution, while a case study provides actual examples for how a solution has fixed a problem.

Okay, okay. So the case study is actual examples for how a solution has fixed a problem. Okay. Okay. So the case study is a real life example, whereas the white paper is more like, this is what we believe will happen. That makes a lot of sense. I've read a bunch of like machine learning kind of white papers that are all about like, hey, I think if we do this, then you could see these kinds of benefits. Right.

So it's like the scientific hypothesis approach as opposed to, hey, this is real world what happened when it crashed and burned. Although the interesting thing is that the explanation then goes on to say a case study typically offers greater detail than a white paper with the exception of technical white papers.

Right. So. Yep, makes sense. All right, so I don't feel so bad about mixing those up in my head. They're fairly close. All right. So the next one we have is this whole thing that we briefly said earlier, this inject production failures. Right. Like that Netflix has their chaos monkey. And the whole point of this is you make sure that you introduce failures in a way that you can at least predict, right? Like you want to do something. Let's say that maybe you turn off a data center.

You do that so that you could then find what all goes wrong when that happens so that you can start trying to implement things to fix that, right? That's really what it is. And so the more controlled you introduce these planned failures, the more you can take these steps to mitigate those things so that if they ever happen unexpectedly, like the thing that outlaw mentioned earlier when an entire region went down netflix had no issue right because they had done this to themselves over time and prepared themselves to be able to handle it yeah and uh so someone here mentioned the comparison to a car the car crash test remember the the crash test dummies uh in the 80s i think they even had an album they had a cassette tape which they did yeah that's that's pretty awesome i'm this many years old wasn't there a band called that i mean

like it was a legit band that's what that's separate so there were the crash system that means yeah i'm sorry i still once there was this girl who yeah uh got into an accident and couldn't go to school and then when and then yeah it was such a such a terrible terrible good song um yeah but uh yeah so the but there were actually there were rap songs by like the crash test dummies which were literally the crash test dummies that they would put in cars that had like they were shaped like humans they were like weighted like humans they had like different zones kind of marked for like that the things that were most critical to human you know life like foreheads and stuff and they would take these cars and they would crash them and then like video play the tapes

back and see how the bodies responded and some of the things that they figured out then other than uh that people weren't really comfortable seeing that kind of stuff uh is that um they ended up designing cars that were uh would have things like crumple zones and so the car would actually uh wreck in a way that would keep passengers safe so the car would get a lot more damage in some cases because more would get destroyed, but it would crunch up in such a way that it would be safer to the people driving.

And cars are much safer than they used to be now in part because of these really big, great tests. Yeah. Except when you get like a little fender bender in the parking lot you've totaled your car now it's an accordion you hit me at three miles an hour what yeah now but they're saying that you should actually design your system or or try to make your systems work in a way like these crash these crashes right like certain parts of your system might not do as well but your core

things the things that you care the most about need to be resilient, right? So think of those pieces of your system as the people in the car, right? If everything else crashes and burns around it, you don't want like your primary database to go, right? Something like that. I'm going to throw a link in here, by the way, to the 1987 Crash Test Dummies rap. Now, this is not the Canadian band. This is the literal Crash Test Dummies that did a rap about how you should wear your seatbelts. And, I mean, the dancing is phenomenal.

This is like super 80s. So, I definitely recommend you check out those show notes. That's awesome. How do I not click this right here? I mean, I get so bad. I'm actually waiting to the end of the episode and then I'm going to do it. Just the video alone is pretty good. Oh man.

I can't, I want to click it so bad, Joe, but I know the computers. I can't hear you over the rap. Yeah, exactly. Okay.

So, so the next piece in here, cause we need to hurry through this now so that all of us can watch this. Yeah. Is the, you know, the, the graceful degradation, right? So if things start to fail, you want them to fail in the most graceful way possible, right? Like you don't want things just crashing and burning, if at all possible. Oh, and this was a really cool one too, right? Like we haven't gone into all the case studies in the book.

Like we purposely skip over them. Otherwise, these would be 12-hour episodes. But another one that was really good is because like Netflix truly embraces their chaos monkey, right? Like truly, they? Like, truly. They use it for everything. There was a point in time when AWS was going to upgrade the nodes that Cassandra runs on. There were over 2,700 nodes that needed to be upgraded, like the underlying hardware, right? This all happened because Netflix had planned for or down nodes with Cassandra.

They had zero downtime because they had gone through this with Chaos Monkey, run through the simulations and fixed all this stuff. So would it actually had to happen? Even though some of the Cassandra nodes didn't even come back up, life was fine. They didn't, it was like not even a blip on the radar. Super cool.

Hey, 2,700 Cassandra nodes. Like I feel like as we have like seven, eight, nine nodes, if I was working there, I'd be like, I don't know.

That's a lot. 2,700. Jeez. There was also another part to that story there though, where at first they were like, oh, man, no, I don't want to do that. And it was going to be done over a weekend and everything. And they were like, I just can't. No.

And then they were like, wait a minute. We have the Chaos Monkey. No, forget it. No, we're good. Go ahead. Yeah, we don't care. That's exactly how they put it. Yeah.

Oh, wait. I'm going to the bar. I'll be good. Uh-huh. Yeah, you guys do your upgrades. We're fine. it no we're good go ahead yeah we don't care that's exactly how they put it yeah oh wait i'm going to the bar all these i'll be good uh-huh yeah you guys do your upgrades we're fine we're fine yeah it'll be fine so the last section we have here are game days and i've never actually been a part of one of these even though i think we were going to be at some point. The three of us, when we were working at Amazon for a while, they had started talking about introducing these game days. And basically what this was, it was a term coined by Jess Robbins. And it introduces a large scale fault injection across your critical systems. Like, hey, we are going to turn off your network

here. How does everything respond? Okay. Right. Like that's- Let me know when you turn it back on and I'll tell you how things are going. I think the lights are still blinking. I don't know. So one of the things that he said here and i love this quote and this is from jess robbins he said service is not really tested until we break it in production i mean i don't know if i've ever heard truer words yeah i mean because let's be honest, right? Like you typically, and I say this, not being facetious, I say this in probably the most accurate way. You typically don't design your systems thinking they're going to be broken all the time, right? You sort of, if you're creating a website that talks to a database, you're just going to assume that it can talk to the database, right? It's not until that database goes down that you're like, oh man, I didn't have anything checking to see if the connection would work. I didn't have

anything checking to see if that query would fail, right? Like you just assume they work until they don't. I mean, I assume they're going to work in the first couple of weeks of a project. By the time we get to the end i'd be amazed uh that's awesome yeah it's all about how how you plan to use something too like have we ever talked to ever talk about like that the the raid story did i ever share that here yes yes it's a good one i mean like that that's one that just comes to mind you know it's like the way that the way the raid system was you know quote tested by by the people that built it like the way they striped you know like if you're talking about like like when i talk about rates i'm talking about like

one where the entire rack is nothing but hard drives with maybe a controller in the middle you know or you know controller somewhere but you know, a controller somewhere, but, you know, uh, and, and the way they had tested their rack, you know, for performance was, you know, all they cared about was performance. So they would like stripe the, the rate arrays going across, uh, you know, you, they would say vertically, cause you know, you'd have like one, one bank of discs stacked on top of another stack on top of another and each one of those were like physically separate chassis right and and to get to maximize performance you know you were using uh you know each chassis for one drive right to like maximize that performance which would just be blistering fast. It was awesome. But it was like, well, hey,

what raid level are you going to use? Because how important is that array to you if you lost the entire array? Because it's like, if you're in raid five, you could lose you know, which means like, you know, either one disc or like one of those, you know, the, what did I call it a moment ago? The tray that had the discs? The controller? No, no, no. The controller, there's a controller that would literally control all the discs. But, you know, whatever that tray is that would have the individual discs, because I forget what I just called it a moment ago, but I'm going to go with chassis now. The chassis, there you go.

I knew it was a C word. Yeah, like you could either, you know, maybe in a good world, you only lost like a single disc out of that chassis, you know, fine. But in a bad world, like maybe you lost the entire chassis, right? And then what happens? And, you know, if it's a RAID 5 for a RA what happens then you know if it's raid five for raid six you know maybe fine but what happens if you lost like a few chassis because of maybe like hey i don't know random power outage that could happen you know like maybe you like you know you

lost the bottom half of the of the rack and i've seen it happen right and and uh you know that can be catastrophic to like you lost every array that was that rack held but you know if you striped it your arrays to where you're going across the entire chassis for example and each volume was only going across a single chassis you weren't maximizing for performance but you're maximizing for resiliency so So it's like, what, what, you know, what matters to you, right? Like, what are you trying, what are you trying to protect against? And, you know, you think of like with this game day thing, like I bring it up because, and you know, at the time, like people were saying like, oh no, you should definitely go, go vertical, not horizontal until it was like, you know, if you've never tried this with this game day idea of like,

hey, let's break it and see what happens. Like, let's just see like what would happen in this situation if you lost multiple of these chassis, right? So. Yep. Hey, let me get this straight. We're not supposed to test the production, but we're supposed to break stuff in production.

Right. Isn't that crazy? Hey, so check this out. We actually had a conversation today that was interesting. And this is where some of these game day things could come into play. So like I said, you typically design software with the notion that you think that, hey, this is going to work, right? And we had a conversation today, at least several of us did, to where there were services that relied on external APIs and services, right? If you've ever worked with something like Amazon's AWS APIs, like their marketplace APIs and stuff,

they will throttle you, right? Like you can have X number of requests and if all of a sudden you bump up over that or you're issuing so many at a time, they'll actually kill the amount of data that you can get back. And so you start getting throttled back, right? Maybe you didn't even realize that. Maybe you never tested that in development. Maybe you had no clue that that happened, but you assumed everything worked, that it gets up there. And all of a sudden you start hitting these throttling things and things

start backing up. Well, think about this. If you're hitting a bunch of external APIs, what if you start having things like this happening all over the place, right? This is something that if you introduce a game day and you say, hey, what happens if this API goes down? Let's force it to not work, right? Like let's say that we kill the DNS on even being able to get out to that thing. Now you can sort of plan and figure out strategies around it. Whereas if it just happens to you now, it's a fire drill, right? So that's the whole point of these game days is, is force these controlled failures. And, and what's interesting is, and we haven't even led up to this is the game day is the day that you go do it. Like they're going to say, Hey, we're killing all this, but you prepare for it, getting up to it. Just like you would anything,

right? Like if you're, if you're going to try and run a 5k, you're probably going to build up to that over a few months, trying to get, you know, your lungs up and all that kind of stuff. You do the same thing, preparing for game day. You, you know, internally shut down parts of a network and you say, okay, what happened? Let's evaluate it. Let's see what we can do to get around it. Right? So you keep building up to it. And then that way, everything that goes wrong, you take notes of it, you fix it, you retest it. Game day comes. Hopefully you're good. Probably you won't be completely because there's going to be some stuff you missed, but it makes you stronger because now

you'll take the learnings from that and take them back, implement them, put the fixes in. And then the next time you'll be more prepared, right? So that's the goal of the entire thing is to create these, these more resilient, these more fault tolerant, these more graceful degradating degradation or degradating system. Degradation. Yeah. Was it a serious? It was egregious

degradation. Degradation. Wow. Thanks. Thanks. So what's on now? Oh,

the episode's over. I guess. Nevermind. Yeah. I'll get you next time. Degradation. That's, that's a hard one.

All right. So we'll have some resources we like, uh, as usual. And, uh, now we're on to, uh,

I can't say it. It's outlaws. I was going to say it. Oh, uh, it's, uh, Alan's favorite portion of the show. It's outlaws. I was going to say it. Oh, uh, it's, uh, Alan's favorite portion of the show.

It's the tip of the week. Yeah. Come on. All right. So I've got only a couple this time and these aren't so much development. Yeah. Only a couple. I know.

Right. Um, well, I got to make up for that episode where I had to run because my kid was yakking. So, um, I hate the way you describe it. I mean, that's what it was.

So, this first one... Isn't yak an animal? Isn't that an animal? It just sounds amazing, right? It sounds right. It sounds like the right word. I'm pretty sure it's an animal. Yeah, it is. It is an animal. Yes, it is i i think they must vomit a lot i don't know a large domesticated wild ox yeah there you go you learn many things on cutting blocks yeah all yaks are domesticated i never knew super important detail all right your kid had a yak in the house that is crazy no wonder you

had to leave in such a rush he got kicked out the house oh my gosh so all right back to this one i think i've mentioned melanator in the past like i hate giving my email when i go to a website and they're like hey you know if you want if you want this, you know, free, how to write code, whatever you're going to stick in your email. And I'm like, I really don't want to. So mail inator typically works. Um, there are some sites that catch it. Yeah. I mean, there are times where they're like, Hey, put in a valid email address. And I'm like, Oh man, they found out. Okay. So Firefox actually has a feature. And this came from Ollie Navard over in one of our Slack channels. It might've been episode discussion or in gear, two of my favorites. Um, but at any rate,

Firefox has a feature called relay relay, and you can go to relay.firefox.com and they have the ability for you to plug in an email that will be at, I think relay.firefox.com or something like that. And you can have it forward to your emails. So you don't even have to do mail-in editor. You could actually use basically a trusted domain. I'm sure that this thing will get blacklisted at some point, but you can do this without having to go through another site. So I love this as a tip, if you don't want to have to give out your emails. So have you tried speaking of, cause kind of similar to that, then, uh, have you tried the sign in with Apple where you can, where it'll create like a fake, you know, for you? I, I think I, anytime I see sign in with Apple or Google, I almost always say no, because that's usually using OAuth and behind the scenes, they're requesting more of your information. Like they, Hey, if you, if you sign in with Google, then they can have access to your profile, your contacts, whatever. And so I just don't do that. Like if I have the option to sign up with an email or something, I always do that instead of

using one of the OAuth methods. Yeah. Except it's not like that for Apple. That would be true for everything else. Right. But, but with sign in with Apple, and this is why I was curious if you tried it, cause it kind of similar to your relay Firefox, except, uh, you know, obviously the place, you know, the, the destination has to support it in order for it to work. Whereas the relay for Firefox is a little bit more generic. But, uh, when, when you, when you use sign in with, I haven't tried it, so, uh, you know, but just from like what they, what they said about it, like, uh, when you, when you sign in the apps and websites can only like ask for the, the names, uh, and an email

address set up. I haven't, wait, let me restart this. You can hide your, your email address and it can create like a, uh, like a spoof one, kind of like what you're saying with the relay that they can't, um, you know, track back to you. So like each time, like from one destination to the next, it's going to be a unique one. If I recall correctly, like it's not going to, it's similar to your relay at Firefox. Right. Yeah. Um, but, but it's secure and, uh, you know, private from the, from the get go. Right. And because you, if you have two factor turned on with your, your Apple ID, then actually you might have to have two factor enabled with your, your Apple ID in order to use sign in with Apple.

Now that, well, this is cool. I found the webpage and I put it under your tips of the week here, but this is awesome. So yeah, I mean, they've got the bullet points here saying exactly what you said. And so it looks like they kind of hide your identity for you. Like you can sort of put in what you want, which again, I know we've talked about this in the past. It's the reason why I'm on an iPhone.

It's the only reason I'm on an iPhone is because they care about my privacy. So, um, yeah, it's pretty cool. I mean, I, like I said, I haven't had a chance to use it because like you, anytime I see something like that, I'm like, no, I just don't want to sign in at all. You know, forget it. Yeah. I'll find another way. So my second one, and, and I think we've mentioned, uh, Martin sale, take it a while. So back in the day, he was the one that helped us

improve our audio quality on our podcast. Like, you know, massively like back in episode, I don't know, 10, it's been a while. Um, but he's a constantly, uh, an awesome member on our Slack and he's always contributing good stuff. Right. Uh, well, one of the things that he shared with me at one point is we were talking about different podcasts and he listens to several as well. If one of the ones that he mentioned that I am an absolute fan of now is how I built this by Guy Raz. If you love business and hearing people succeed stuff, how I built this is probably other than Cody blocks. It's probably my favorite other podcast out there, period. But the reason I bring it up is I know people that listen to this podcast, they like to learn.

This is, I mean, otherwise, why would you listen? I mean, yeah, we are funny and we are, you know, amazing and all that, but you know, you want to learn something. So there was an episode about Khan Academy. I'd never heard of this and I felt kind of silly after not having heard about it because apparently it's a massively big deal. But the gist of it is Khan Academy will have a link here. It is a free learning website. If you want to donate, you can. It's amazing. But the reason I bring this up is a lot of us have kids that are working, that are at home doing school and stuff right now. The way the guy started this, I forget his whole name. It's something con. Doggone it.

I cannot remember. At any rate, the gist of it is this. He ended up helping out like his cousins or his nephews or something or nieces with like some math. They were struggling with math. And he's like, look, anybody can do this. And he took the time to start tutoring them. Well, they loved it so much that they started sharing it with friends and family and everybody else. And so he was like, man, everybody should be able to learn. So if you come to this website, you go to the courses dropdown, they have stuff for every grade level. They have topics from science to math, to chemistry, to all this kind of stuff. They have economics, they have life skills, they have all this. This is all free. And from what I understand, it is like the highest quality stuff. Bill Gates, he was doing an interview somewhere. And if you listen to the

episode, you'll hear this. It's actually amazing. Bill Gates was asked on some like AMA type thing, like, Hey, what are you excited about? And he said, Khan Academy. Like, yeah. How, how unreal would that be? You know? So, uh, yeah. So, uh, so I can't, am I wrong then to think that like you, you were new to Khan Academy? I'd never heard of it. Yeah. Okay. So, so I kinda, I kinda got that impression. Um, uh, so my kids use it a lot.

Like it's actually like a great, like anytime they need to study something or whatever, like they go, they go back and, you know, they'll find courses or, you know, subject matter related to Khan Academy. So it's an awesome resource for that. Um, and I've actually heard that like one of the, uh, when your kids get into SAT age range, that like one of the strategies that they, that they recommend and they talk about for like, let's take just the math portion of the SAT, for example, as a study technique for the SAT is they go to Khan Academy and then just brute force your way through all of the math, starting back at like, you know, kindergarten, first grade or whatever, go back from the very beginning and then just blast your way through all of it as your study guide, you know, to, to, before you go take that SAT. So I got a tip for you on that too,

because he actually talked about this on the show. And again, I can't highly recommend it enough. I'll get a link and put it in the show notes for it. But he was actually approached to create an SAT testing course. So there is there's actually three courses here on how to take tests for the SAT, the LSAT and practice core. So there are three courses on that as well. college, or if you're already in your professional career and you know, you have family members or family, friends, whatever, to where there are, you know, young kids that are getting ready to go to college, point them at this, right? Like practicing and getting this, getting a high score on an SAT can really change your options for where you're going, the kinds of things that open up for you. So here's absolutely amazing. Here's another parenting tip that we've used over the years too.

And now that you know about the Khan Academy, you can use this one, Alan, and you can thank me. And since your wife is listening too, then she knows now that you can't take credit for this. That's right. Uh, you heard it here first, but, um, like one thing that like, if you ever want to, you know, like sometimes your kids will want to like, uh, do something to like get a reward, like, like, like if you limit, um, you know, like how much time they can play like video games or whatever. Right. You know, but maybe you want to like, give them like kind of rewards or incentives to like, okay, fine. Yeah. I tell you what, if you, if you do this, then you can, you know, play another hour of whatever. Um, you know, we've used Khan Academy over the years as a way to like, Hey, you know, go,

go work on, uh, this subject in Khan Academy for, you know, 30 minutes or whatever long, however long it takes, go do that course, and then we can talk. That's awesome. Yeah, I love it. Yeah. And again, all three. Because let's face it, parenting is nothing more than it's just all about bribery.

That's all parenting is. Which becomes illegal as things progress in life. But while they're at home, bribery is fully enforced. So those are my two tips. That's it. All right. Well, mine is the Elgato Stream Deck. So we, all three of us, just got these.

We got the 15-button stream decks, which if you've not seen them before, they're really popular with streamers who need to do things like switching layouts or whatever for the streams in order to kind of show or view or do certain kind of common actions. But it's basically like a deck of, in our case, 15 buttons. And each button is like a little monitor that can have a picture. And you can program the buttons to do whatever you want. There's a bunch of ones built in.

There's some that show like stock tickers or CPUpu there's ones you can push and it will open uh there's actually a bunch of vs code extensions so you can like run a build or start a debugger or open up a new window or whatever open up a new terminal and i've just been using it for all sorts of stuff so we talked a little bit before about having like a like a bin directory or a place for scripts on your computer and some people even check them in and uh so since i have got uh the stream deck my my bin directory at work has exploded because it turns out like this is just what i needed to like really kick it up a notch i love not typing the common little things and the problem with that is like uh with typing even bringing up certain websites, like say you want to check your calendar. I have a calendar button now.

The hard part isn't doing the thing or typing the thing or even remembering the command. You know, like that stuff's not too bad. The hard part is finding the right stupid terminal for you to type that stuff or where you typed it last time or where the stupid tab is. So instead of doing all that, why don't not just set a button that will do the thing? So if you want to pull your main branch and get into your directory, instead of trying to find a good terminal window that makes sense to you, just press the button and it'll pull main for you. You want to push it up there?

Hey, why not do that too? Why not do them at the same time? What the heck? That could be one button. You can also switch profiles. So even though it's 15 buttons you can push a button and hey now's a new set of uh you know 15 or 12 or whatever uh so that's been really great and so i've just got a ton of these little and they all just do stupid things but i swear the mental weight of not having to look for these things has made stuff so much better. And I mentioned working with Kubernetes a lot.

I have buttons that will switch my environment. So before it used to be so annoying. Someone would say, I'm having a problem over here. Can you check it out? And I'd be like, I've got to go swap my contacts, go over here. It's just disruptive to my workflow. I don't really want to do that. Now it's like, let me just push the button.

Yeah, I'm there. Already there. So this GitPool, I'm there. Already there. So this Git pull, I'm curious, you have it then launch a new terminal? Yeah. And then closes? Yep. And you can do it in the background too.

I mean, you pretty much do whatever you want. But I like to actually just see it come up just because I don't mind waiting a second for it to kind of flip on. And sometimes it's kind of nice to see. But I have other ones that prompt me too. So it just runs a little PowerShell command a lot and say like, hey, what environment do you want to go to? You can press 1, 2, 3, 4, 5, 6. It feels like the old day of programming. We didn't have sophisticated tools.

So there's nothing here that I couldn't have done with just shell scripts or whatever. But having just a button where you can just kind of reach or even just see the feedback. Like my dream here is to get Jenkins hookup so i can push a button to kick off a build and just like maybe tell me what stage is on four out of 17 green so far great you know how much time is left something like just little stuff yeah it's just so nice and so now i want to get like 30 of these things and like put them all over my house you know that's gonna be a expensive project like i i love the thing it's um like joe said it's basically a picture of 15 button keyboard and every key is a 72 by 72 pixel display that you can you can make the display be whatever you want it to be uh like you want

to update it fine um and now you just made me think of like another use I'm going to use for, uh, one of the, one of the buttons now will be related to like when I am putting the show together, like I already have a script that does the ID three tags for the show. And now I'm just going to like program it to be like button, go do the thing. Oh, that's beautiful. Yeah. Now I have to going to program it to be like, button, go do the thing. Oh, that's beautiful. Yeah. Not have to go open the program, not have to do this, that, this, that.

Yeah. Yeah. Yeah. Yeah. I was going to make one for finding the file, naming appropriately, uploading to the right spot, zipping it up. Like, that's all stuff I could do. And when I described it to my wife, she was like, you need to get this. You're going to love it.

You could turn on the lamp from the other side of the room. She's like, this just sounds like shortcuts. I've been around since, you know, windows 95. Like, no, it's, it's different. Okay. Okay. It's a tactile shortcut. So what about, uh, you know, you're saying like, it's just shortcuts, but like if you,

we, we've talked about my love of the hue lights before you know and like because of where i work in my house like it's if my family needs me for any reason and they they come down to my dungeon uh that's what you know what they call it uh you know to where i'm at, it can be frustrating on their point if they get all the way down there only to find out, like, I'm on a call or something like that. And I can't like, you know, uh, I'm not available at that time. Right. And so, uh, I actually use that stream deck to set up, like I'm on a call button so that it can change hue lights around the house, be like, you know, red so that they can like immediately know like,

Oh, he's on a call. You know, but in fairness, it is just a shortcut. Cause you could have done that by setting it up with Google or, or echo or any of the other. Sure.

But if you're talking about like multiple zones, it's a lot easier when you only have the one button to do the thing, right? Right. And that's the thing, like you could, you know, in the way that Joe described it, right? Like you could script anything you wanted to do. Right. And the thing is like, it has an API that is, you know, you know, straightforward API. That's pretty cool. But really you technically don't even have to, it doesn't even have to be anything to do with your api you can literally like write a batch file like i don't care you write any executable thing and you can assign this you know assign it to a button and it'll go do its thing like really the

api comes in to play more is is more important when you really want to like change the, the display. That's, that's probably more, you know, where, where the API matters the most. And so far, like, you know, the, I have some cool uses that, uh, you know, some cool buttons that, that are updating the displays that like were just actions that were just already available. Uh, you know, like you, you can go, like Joe said, you can go Google around and find stuff. Um, but you know, for me personally, I, I, I like Joe, I'm just like, Oh, Hey, I already have this script, like go execute it.

Well, his Jenkins thing in the, in the back of my head, what I see is, is, all right, let's say that that you got the standard stream deck right and it's five buttons across the top and middle and bottom i mean wouldn't it be cool if it was truly like you you messed with the api and and it could show you like a little thing spinning for it's in the build stage and then and then the next one it's in the deploy stage and these things turn green as they go through right like that would be super cool how many times have you kicked off the building you're like okay i'm gonna come back and check this in uh 11 minutes when it's done and you get distracted

they see though it's like 40 minutes and you go back oh and it failed right okay well now it's another 11 minutes because i fixed the typo but if you could just look at your screen down there and it and it was on your little elgato thing. Yeah. You could be like, ooh, yeah. Now I want to make a little chaos. Yeah. I want to like push a button and kill a random pod or, you know, whatever. The chaos pod killer. Excellent.

All right. So then we'll head into my tip of the week or tips of the week. I'm taking a cue from Alan. So I don't know if you guys have seen this. A lot of people have probably might've already seen this because it was quite popular on Netflix. It was trending across the country in the top. But if you haven't already watched it, it's called The Social Dilemma. And it's a documentary. And Alan, you were talking about how you value your privacy and how that's the whole reason why you picked the iPhone, right? So if you haven't already watched this, it's going to give you a whole new way of thinking

about anytime there's ever a suggestion made to you, if you click it or interact with it, I don't care what kind of suggestion is, it could be a notification, right? Like it was, it was very interesting to watch, uh, the, the, you know, the, their little blurb here is this documentary drama hybrid explores the dangerous human impact of social networking with tech tech experts, uh, sounding the alarm on their own creations. And basically like you'll have people in there, like, you know, the, the eighth employee for Facebook or, you know, or whatever, like, you know, the guy who was in charge of like monetizing Facebook and, and, you know, the guy who invented the, the like button, you know, or whatever, like, you know, the guy who was in charge of like monetizing Facebook and, and, you know, the guy who invented the, the like button, you know, or like all these

different things on, on social media, you know, that we're used to, right. Uh, and that started out like innocently enough, but then became like ways to track you, uh, you know, across the internet or whatever. Right. And, and part of what they were saying too, is like, even the way that the apps would use notifications as like ways to entice you to like, get you back into the application. Like, you know, let's, I mean, it's easy to pick on Facebook for a moment. Cause you think about like, if you're on Facebook, like how many notifications they, they might want to pick, you know, to send you. Right. So how do they decide which ones to

send? Right. And obviously they're going to try to like, be smart about it and pick the one that they think is going to have, or the notifications that are going to have the most likely, uh, you know, um, result in bringing you back into the application. And it was just kind of like talking about how there's also not only, uh, those kinds of strategies that they're employing, but like the way some of these things can like shape thoughts and, and, uh, you know, thoughts and opinions on things, which, you know, you think back in recent years, how some of that has played out, uh, you know, especially in America, which, you know, you think back in recent years, how some of that has played out, uh, you know, especially in America, like, you know, kind of, you can kind of see like, oh, this, it, you, you can kind of relate to it if, if that makes sense. You know, you can kind of see like,

oh, I can see how this can, like, you can easily see a relatable way that like how it could matter and be a thing. And, um, uh, you know, like know, like how opinions can be shaped around the world. And it just takes like, you know, with enough money, you could like, you know, change opinions. And they were even describing like how, depending on where in the world you are, for example, you know, like even something as simple as a Google search, like I made the comment earlier that, you know, you might not have caught at the time where I was like saying like, okay, Hey, this was what came up first for me for my Google search. And, and they were saying that like, you know, uh, you remember like when we used to do the, uh, the, the, the Google feud,

I think is what we called it. Do you remember those, those games? And, and we would purposely use like an incognito mode in order to like, try to like, uh, take away any, um, bias that the search might've had, you know, uh, otherwise. Right. But, uh, you know, in here they were saying like, just your location alone could be part of the influence of like what Google searches might come back first, right? Like what part of the country or what even, what even part of the world you're in could, could influence. So literally you and I could do the same search and get different results. And, and basically, like the takeaway from it was that, you know, or not, I won't say the takeaway, the downside of some of what they were saying was like, there were things that were created with purely innocent intentions, like, you know,

but unfortunately, like what it is, what it can end up happening, if you're not careful about how you use all the technology that's available to us today uh you can end up you know getting completely isolated and like stuck in your own kind of bubble you know so it was it's a very good watch it's a simple watch it's not it's not complicated uh you know you know but but uh you know it's not like a three hour you know avengers movie you know and that you had to watch all the first watch all the first 20 in order to understand where you're at in the current one. But yeah, it was interesting. Really good. Really good.

And I don't recommend movies often on the show, but I thought that one was relevant and interesting. So then... That explains why uh everybody on facebook lost the dang mind i literally see people call each other evil on a daily basis now every time i go to look at like pictures of my niece yeah that's great i mean there was a big there was a lot of uh talk you know on like like social media platforms from it. But I will say this, like, after having watched that, I was like, you know what, I went back through and I, and I curated what apps can even that I even allow to give me a notification. And then,

you know, some on some of my devices, I removed apps like Twitter, for example, because I'm like, you know what, if I'm on this device, I don't need Twitter in my world, right? You know, like, so, so I did that. Because, by the way, like some of this didn't just apply to like social media, like, you're like what you might traditionally think of as social media, like a Twitter or a Facebook, like email counts, by the way, you know, like, so highly, highly recommend giving it the time to watch it. It's, you know, worth it. All right. So in typical Michael fashion here, I'm going to give you a Git tip. So if you ever find yourself in need of migrating a TFS, a team foundation server or team foundation version control repository to Git, there are a couple tools out there for it. But the one that I've had the best luck over the years with is called

Git TFS. And if I remember right, it's a it's written in.net. So you know, from a Windows platform, it works, you know, faster, there were some other ones that I remember, like, several years back, using that were like Java based based but which if you were on a windows environment meant like you had to install another dependency to even run java code assuming you weren't a java developer by trade and but then the downside though like excluding like what language it was written in the execution of those other ones were just horrible in comparison. Like, you know, it would take hours to do the same thing that get TFS could do in minutes. So, you know, the performance wise, it was really good. So I'll give you a link to it. You can just do a Chaco install for this thing,

and it works great. Now that said, the, the original authors for it, you know, they don't exactly maintain it. I mean, they have like a thing where they say like, hey, we need your help because like, you know, we've already done our migration, so we don't need to keep maintaining this thing. But yet they still are like, you know, I'm looking at their repository now and you can see like there was something that was updated nine months ago. So, you know, it doesn't get like daily kind of love or, you know, monthly love even, but, uh, and there, and there are like several outstanding pull requests, but it still works fine. Like I I've, I've had good for that. But there's also this cheater way that I thought that I would like to tell you, and I'll include documentation for this too. But

if what you want to migrate, if let's say you're in Azure DevOps, for example, and what you want to do that the TF VC repository is already in Azure DevOps, you can within Azure DevOps, just migrate your import that other repo as a Git repo and Azure DevOps will do all the work for you. And so I'll include the documentation for it, but basically you would just go in to the project that you want to import the repository into, click on your little dropdown for the projects, click on import repository, and then you would select get and put in the, uh, the, the, the TFVC version

of the project of the other project name and, and do the import. Now, of course there would be a, there's gotta be a caveat to that, right? Because like, otherwise, why would I even talk about get TFS from the beginning? And the reason why is that get TFS, the beauty of get TFS by doing it yourself through the command line is if you want to keep history and all that, like, you know, you sky's the limit, you could do whatever you wanted to do in that, right? Like you want to deal with bringing in branches or whatever, you know, all of that is, is available to you, right? Whereas if you do the, the Azure DevOps cheat method that I'm described, Azure DevOps will limit the history that it will bring in

to the last, uh, I want to say the maximum is the last 180 days. Um, something like that. I believe it's 180 days. Whereas, you know, maybe, maybe that's fine for your needs or maybe you're like, oh man, I don't want to lose the history from like, you know, six months ago, uh, or, you know, no, a year ago. Um, so, you know, your mileage may vary, but pretty cool. Very nice. And, and that also reminded me too, you know, you were talking about, um, there at the tail end, you're talking about the whole, uh, with the game day thing and, and that also reminded me too, you know, you were talking about, um, there at the tail end, you're talking about the whole, uh, with the game day thing and, and, um, you brought up rate limiting, you know, like, and it made me think like, um, have you ever like, you know,

as a user of Azure DevOps, uh, you know, hit their rate limits. I might have, I don't recall it. Actually, yes, I do remember we have. Because I'm hitting them daily, multiple times a day with all of the, you know, with all the migrations that I'm doing now. And yeah, it's fun. It's fun to know that like when you go to, you go to Azure and Azure DevOps and it's like, oh no, you don't get to see the repo. No, sir. You sir have abused it. Yes. So yeah, that's apparently a thing I never knew. All right. Well, with that, we hope you've enjoyed the show. This is, you know, the third way. And we thought we were going direction of this podcast and uh you know or much like that uh the twitter comment that you you shared earlier uh you know a friend pointed him to it you know if you're not already subscribed be sure to subscribe to us you can find uh all the information find us on every platform that you're looking for uh podcasts on

itunes spotify stitcher whatever your favorite place is. And if you have a favorite destination that we're not on, let us know. Otherwise, as Alan mentioned earlier, like we love the reviews, we greatly appreciate them. So if you haven't already left us a review, it's one way that you can definitely put a smile on our face and it doesn't cost either of us any money. Uh, you can head to www.codingblocks.net slash review. Oh, I also see, I have to say that while you're up there, uh, go check out our interwebs at, uh, codingblocks.net to check out our show notes and our examples. And there's discussion and there's, there might even be some rants and miscellaneous stuff. Uh, I think last time Joe put up some kind of random spreadsheet

with randomly changing things in there. You'll see Alan post videos of random things that he decided to buy and review. Yeah, you'll find all kinds of stuff there. Randomness, yes. And if you haven't joined our Slack community, definitely go to codingblocks.net slash Slack where you can go ahead and join over there

and you can send us some feedback, questions, or rants up over there. We've got channels for all of it. And we're on Twitter at Coding Blocks where we're getting into all sorts of trouble. You can head over to Coding Blocks then and you can find your social links at the top of the page and some of ours too. Although maybe

not Outlaws anymore. Wait, what happened to mine? Facebook said they don't like you either. That's right. They don't like you either. Now you know, if Outlaw is ever working for Facebook, they gave him a lot of money. Oh, no doubt.