The Standup with ThePrimeagen - The Secret of the AWS Outage

Episode Date: December 30, 2025

We finally figured out WHY AWS had their outage… Casey reveals the absolutely unhinged and crazy story. 00:00:00 - Intro 00:00:54 - Caseys Intro 00:01:15 - The Setup 00:02:55 - The Story 00:08:44 - The Post Mortem 00:12:11 - The Informants 00:14:10 - The Suspect 00:15:34 - The Code 00:20:33 - The Deal 00:23:05 - The Conclusion 00:24:37 - The Debrief 00:26:40 - Google Testing Utility Failure 00:28:43 - Production Code is Forever 00:40:10 - Outro

Is everybody ready for this bad boy? I'm so ready. I don't even know which topic we picked. Dude, it's going to be so exciting. You don't even know trash because you obviously didn't read the email that Casey sent. Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, sorry. All right, today on the stand-up, let's y'all. Casey, we're doing your topic.

I'm very excited about it. I just cannot wait. All right, today on the stand-up, we are going to be rehashing an older topic. The AWS or the Amazon outage that just recently happened. It was just a couple weeks ago. I forgot the exact date and forgot to look it up. But Casey apparently got the inside scoop. An actual Amazon engineer came up and said,

hold your horses. You need to sit down. No, no, you're jumping ahead. That is not what happened. It's crazy than that. It's crazy than that. All right, hey, Casey, why don't you do the intro? Today on the stand-up, Casey will do the intro.

Oh, start. It's fresh. I love this. Yes. Today on the stand-up, Casey is going to create an urban legend about the Amazon outage because of a ridiculous series of events, which I am going to recount to you now, in the longest, most boring possible way,

because this is an entertainment program where we stretch everything out. So here's the deal. If you are familiar at all with sort of the tech scene here in Washington State, it's all kind of around the Redmond Washington area, where Microsoft originally kind of, well, they didn't originally settle. They moved up there, right? And so there's a bunch of tech around here now, and there's a couple places. It's called the East Side, because it's not in Seattle proper.

In Seattle proper, there actually are Amazon Towers now. There's a Google campus there. There's a Facebook campus there, et cetera, et cetera. So there is stuff in Seattle proper. There's also a lot of tech on the east side. Kirkland, Redmond, Bellevue, etc. I love Kirkland. That is the scene.

I get a lot of groceries from Kirkland. Big fan. Kirkland, Kirkland, that's named after that Kirkland. That's the Kirkland because Costco's from here. Prime, let's take a field trip there. I know. I would actually do it.

I'm totally down. I just got a Costco membership. If Kirkland is out there, we would love to be sponsored by you. I would just throw it out there. There was one brand I would hitch my wagon to. It's Kirkland. I don't have my wallet. Kirkland, I don't have my wallet, but I've been an executive member since 2003.

Yeah, show both sides of the car. Prime. 2003. I get Kirkland stuff at Costco. This story will quickly move its location to very close to the Costco headquarters. So that's how

sprawling and epic this tale is that I'm about to tell. I'm ready. I'm invested. All right. Jeff Roberts is going to think I tell this story horribly. He's a great storyteller and tells stories in a really great way and I suck at it. So he's going to criticize this. I'm sure if you

ever listens to it, hopefully he won't. But so here's what happened. I'm at home. And evidently, I was playing on my piano keyboard too hard. And I broke the little mechanism inside the C4 key. So now when you push it, it kind of goes like clik, clank, clink it, like makes a loud clicking noise. That's an important one, too. You hit that key a lot.

Yeah. A fellow I know calls C the people's key, right? It's also the people's scale, right? The people's scale. So C4 key, you need that key. It's right in the middle. Anyway. So I'm like, well, you know, it occurs to me that I probably wouldn't get this replaced at some point.

And because I have to do music work on this particular project, it's not great if I don't have a keyboard to use for various things. So I'm like, I should have a backup keyboard. You know, I made a lot of mistakes in my life. I had billions of dollars of venture capital. Legions of engineers under my command, tokens on tokens, on tokens. And what did I decide to do with it? Did I fix reconnecting websockets?

No, I shipped an electron app. I failed you. I didn't mean for it to end this way. This prison will hold us back. The love of a founder? You can't stop. My biggest regret of all is that I'll never program with you again. Tupil, we still can.

Tupil, a native app that lets you code together, even from code jail. Free yourself by going to tupil. And use code prime for 90% off your team's first three months. Casey, can I ask? This is not what took down AWS, though, right? Just to be clear. Who knows at this point? Okay.

The smashing of the keys will. So loud, DJ. I didn't know if it was a load-bearing scene around the world. Like I said, I'm trying to fill an hour with a story that's got five minutes of content, all right? So like, just bear with me here.

So, so, what happens is I'm like, okay, we could get a replacement keyboard. The guitar center in Redmond has this kind of keyboard. I will go there and get

Get it. That's it. That's what was supposed to happen. That's all that was supposed to happen in this evening. I'm following. I'm following. Yeah, okay. Got it. Get in the car. Go to Redmond. Actually, ate dinner. Go to the guitar center. Where did you eat dinner? Sorry? Where did you eat dinner? We only have an hour, TJ. We can't go for all the details. Okay, fine. I ate dinner at a place called Musashi, which is this like sushi place that's right in Redmond, Town Center for those who know. The sush. I want that.

Anyway, I had a sake down. It was like a bowl of raw salmon. There you go. Best fish, by the way. Best fish. It is the best fish. Yeah, I was like saying it's the best fish. Everyone knows.

Everyone knows this is the best fish. Casey loves it. I love it. Orcas love it. It's the best fish. There you go. A million orcas can't be wrong. There's probably only 60 left.

But whatever. 60 orcas can't be wrong. So anyway. I go to the guitar center. I'm like, I place this order for keyboard online. They're like, oh, yeah, like our system doesn't really work at all. So we're going to need a while to like figure out how to process. I'm like, okay.

So I walk into the back of the place where the keyboards are, and I'm just like looking around. And this is when a guy comes up to me. It's actually two people, like a guy. I think it was, I think they were like, you know, maybe not husband and wife or partners. I don't know. They were together, right? So this guy and this girl come up And the guy goes

Are you case of Muratory? Or it says something like Are your case of Muratory right? And I'm like Yes I am. He's like I saw you recently talking About the AWS outage on YouTube And I'm like Messing to stand up. Let's go.

This podcast got motion That's right. Am I famous? Am I famous now too? Yes You're famous. We got itchy lips is famous. Let's go. Itchy lips is everywhere, bro.

The whole world is watching your snacks, trash. The whole world. Please, stop for me. Sorry. I'm like, oh, do you watch the stand-up? I'm like, yeah, like, it's really popular. I love that show. I'm like, thanks for watching.

And I thought that it was just somebody who was just like, I saw you stand-up, it was cool, thanks. And I was going to say thanks. And he's like, but no. He goes, on that program, you said if anyone had any more information about what happened in the AWS outage to tell you.

It's like, I have something to tell you. I'm like, oh, really? Do you? Did he open up like a secret door? Yeah, kind of. This story just gets so hilarious. So anyway, so if you remember this outage, it was a DynamoDB outage issue.

And eventually, I think when we covered it, I don't think they had actually published their post-mortem yet. So we were just like, we said, like, we don't know. Like, it's kind of weird. Like, what happened was unclear. There were some vague things about it, but didn't, right? They published a post-mortem later.

And the post-mortem was really kind of confusing. Because what it claimed, and I'll recount this for people now, because we never covered this on the program. So if you haven't read the post-mortem, you wouldn't know, even if you watched our old show. The post-mortem said that what happened, the root cause, not the, like, fall over effects of DynamoDB going down, which was its whole other thing, apparently. That the root cause was there's this thing that's like a DNS planner, they called it. A thing that makes a plan that's what, like, hosts should be contacted for what regarding DynamoDB, I guess, right?

And these plans are updated fairly frequently. They didn't really say how often. But this machine is like cranking along, and it's like coming up with these new plans. and then that machine apparently doesn't do anything. It just comes up with the plan. Like, that's it. And there's these other machines called DNS enacters. And I think they're machines, I don't know if there's machines or processes,

but there's these other independently running things called DNS enactors, of which there are multiple, that take these plans and go and set all the DNSes or something. They were not that clear about it, but that's what they described. Does this sound right to everyone to do? I don't know if you guys ever read this. I didn't realize. It was like a race condition thing, right?

Like, well, we'll get to that in a second. But this is the structure of it, right? Okay. And what they said happened was normally a DNS enactor, right? It goes through and it says like, all right, if I'm going to, like, I was told or I grabbed plan, you know, 12, and I'm going to go around setting plan 12. I'm going to quickly check to make sure that we aren't on plan 13 yet, right, or 14 yet. I'm just to check quick at the beginning.

Once at the beginning, it said. Oh. I'm going to check once at the beginning. And I'm just going to go. And then if I've got the newest plan, I'm just going to go through setting all the plans. And then I'm going to finish, right? And what they claimed it happened was one of these DNS enactors got kind of hung up somewhere. And another DNS enactor that started up, started updating everybody to a newer plan, right?

finished. The old DNS enactor, though, started again and put in kind of the old plan back in. And then they hand-wavingly said something like, and then because it was trying to clean up the plan, it like deleted something. Like deleted the configuration, leaving no way to recover it. And like, when I read this, I was like, that doesn't make any logical set. Like, I don't understand what you're actually saying. So it was an unsatisfying explanation because I'm like, I could imagine something like this, but it didn't all click for me. And I'm like, I would like something more detailed.

So does that all make, you know, you see where I'm confused about the official explanation, right? Yes. Yeah. All right. So that's all I knew. So this guy says, well, look, this is what happened. Or rather, this is what happened to me. We, meaning him and the woman that he was with there, it's like,

We were at this place called the Black Duck Cask and Bottle. Now, this is a place. This is basically like a gastropub. This is what they would call it nowadays. What the kids today would call this thing. It's like a place where you go to drink, but they also serve like what is supposed to be upscale food, right? And this place is in downtown Issaquah.

it is literally like just over like there's a highway highway 90 runs in between literally where this place is just on the other side of the highway and the headquarters of Costco is right on the other side guys like the giant new headquarters

that they built is right there Kirkland's signature the plan for what Kirkland signature items will be coming out next year is happening in that headquarters split keyboard please just throwing that out there Kirkland split keyboard, I would buy. If you ever guys ever want to come out and visit Washington,

we can take it at Kirk. We can knock on the door and say, hey, we're podcasters, can we come in? Can we get Pizog? Yeah. Can we get Pizog? Pizog? Yeah, yeah. You buy a Costco pizza and you buy a Costco hot dog.

No. No. No. Oh, my God, that's the thing. I will totally try that. Do not do relish on top. It is not, it is nothing like pineapple. It is very terrible.

But Pizogs are really like, they're, the word's at. But you did that. You put pizza around a hot dog and then you put relish on it because you were just like, yo-o-swag, I'm doing this. Yeah, yeah. That was terrible. But like pizza dogs are really good, though. So it's like pigs in a blanket, but like a pizza? Yeah, exactly. Pigs in a pizza. Hell yeah. I'm going to try that. Yeah, true. I was like, what? Okay. And so here's how we get into the like, I'm not sure what really happened. No one knows because this is where we get the information. So this guy says, I was at the black dust duck caskin bottle. and we were sitting next to a table where two guys were talking. And one of the guys in the middle of the conversation goes,

so you know that Amazon outage that happened yesterday, apparently that was my fault. That's how he starts the conversation. Is this like a spy movie? Yeah. This is like third. This is why I say, like we have no idea this is true. Because not only am I hearing it from someone,

that person wasn't even supposed to know this. They just overheard someone else talking. And maybe the person they overheard was like a pathological liar who made it all up. Who knows? But here's what happened. Casey, you know right now, though, that guy who is at Scary Name Gastropub, I can't remember what it was, but something about death. He's sweating bullets right now because he's like this happened like, canonical. Because he doesn't work there.

No, no, no. He's dead. better. This gets so much. Just wait. Just wait. I promise you. Let's go. The story is so good that I'm telling it, even though we have no idea if it's true, and it probably isn't, right? Because it's such a great story. Yeah, yeah, I'm ready. Anyway. So, so this guy and this girl are sitting at the table enjoying their meal at the black duck caskin bottle. He, by the way, the guy at least, I don't know about the girl, but the guy at least works on cloud stuff.

So when he hears this, he's immediately like, oh, right? And she also apparently was very interested. So I think they might both work on this kind of thing. I don't know, right? Because they both are like, wait, what is going on? So they completely stop talking and kind of like, you know, it's very loud in here, but they're like,

let's listen as closely as we can. So the dude's like, yeah, apparently that was my fault. And his, the person across him who's eating with is like, how can that be your fault? You haven't worked there in like 10 years. Right? And the guy's like, yeah, exactly.

But apparently, there was this code that I'd written back in 2012. The end of the world. That's what the money calendar was about. This is it. The whole thing didn't go off as planned. but back in 2012, I wrote this piece of code that was just supposed to be an easy way to copy the configuration file from one machine to another because I was trying to be able to set up servers very quickly and not have to actually manually configure them myself. So I wrote this thing that was only for me to use and it was supposed to just do this simple copy operation.

It's not a production thing. It was never released into anything. and it was just some code that I had used. And he's like, well, yesterday, or, you know, whatever it was, yesterday I got a panicked call from Amazon. And they were like, okay, this code that you wrote is about to take down the entire data center, and we need it to be stopped, like, immediately. And he's like, what are you talking about? like I don't work there anymore.

I haven't worked there in an incredibly long time. And none of my code would be running in that thing. And he's like, the person on the other end of the phone is apparently like, no, like this code, it's got your,

like, we looked at like the history of it. It is your code from like your repo. And he's like, yes, okay, I did write that code. And he's like, well, we need to be able to stop it because it's deleting all of these configuration files. And we don't know how to get it to stop. Apparently.

This is literally like what the guy is saying. Again, I have no idea how plausible this could possibly be, but the same idea, I don't really know why these data centers go down for seemingly weird reasons that don't make a lot of sense of the post-mortem either. This is definitely possible, by the way. This sounds like standard operating,

like Fortune 500 company radio. I'm loving this. So the guy says, apparently, they found that configuration copying code in the repo that I wrote, did maybe some light testing on it or something like this, and then must have started using it to actually copy the configuration files from, like, different machines to other machines.

But I guess it had, like, a bug or something where sometimes it would just start deleting the configurations after it did the copy. So when they started actually deploying it to things, it would start, like, it would do copies and then delete things and leave them in, like, dead states that couldn't really be recovered. And so they're like, we need the password to something. And apparently, like, they couldn't tell from the conversation what the password. was actually too. But the Amazon people were like, we need the password to this thing in order to like actually

go like modify whatever it was, the build, the bucket, the permanent storage of the thing, the root password for some container they were running. I have no idea, right? And so he's like, okay, I do remember that password

and this is the part where it ties back very tightly to the stand-up again. I do that, I do know that password because I use the same password thing for basically everything. No! It was me all along, baby.

It's slightly better than Primes, which is that he has a different password for every month and then he appends the year. That's a dumb strategy. I got way better strategy than that. You're trying to tell us your strategy. So the password, because

he remembered that the code was from the fall, the password was Wishbone 12 because it was 2012 and Wishbone for Thanksgiving. So that's the password, I guess, to this thing. Okay. So he remembers the password is Wishbone 12, but he doesn't want to tell them because he's like, how do I know who you guys are? You're just calling me, I don't work there anymore. How do I know you're even from Amazon? Maybe this is a fishing attempt, like, what's going on? And why don't you guys already know this

thing anyway? Like, what do you need this password for? Like, I don't believe you. So he goes, so he doesn't tell him it's Wishbone. He's like, okay, here's what you should do. If you can tell me the interviewers that interviewed me for the job at Amazon when I went there, right, when I was actually working, when I was contracted for them, the people who were on like the phone screen or whatever the hell it was, and you can tell me who those people were, because the HR department will have a record of that, then I'll believe you're from Amazon. And the guy says that then he hears the Amazon go like, get me HR on the phone. right now. This is like the presidents on the phone calling Rambo out of retirement. Please tell me their HR service was down because the data center.

That would have been amazing. It would have been checkmate. That would have been amazing. So is this why it was down for multiple hours? Is that they were just waiting for HR? No. Look at the guy's name. We got to save the internet base on this person.

So they did apparently get the names. They told the guy the names. He's like, okay, here's the password. Wishbone, 12. Wishbone 12. It could have been so much worse, trash. It could have been so much worse.

He goes to the bathroom? Two, three, four. So anyway, so he gives them the password and they're like, okay,

and that's the end of the phone call, right? So, apparently, and now I, like I said, I have no idea. I have no way to verify the,

the, the, uh, veracity of this or not. I can say that the official explanation didn't make sense because it didn't sound logical to me how this would have really taken machines out because couldn't you just run the planner and enactor again?

It'd be really weird because they are reaching in an updating configuration. So it was just, it was weird. Like, what I will say is the idea that somebody who didn't know what they were doing pushed a copy thing to prod that then went rogue and started copying configurations to things completely, erroneously and they didn't know how to stop it actually sounds slightly more plausible to me believe it or not but i have no idea and i also don't know why you would need a password for that that part didn't make a whole lot of sense but i could imagine if it was some kind of containerized thing if the password would be more efficient to like edit something in the

container i have no idea but why that would have been set by a guy a very long time ago that doesn't make a lot of sense either so now i have two conflicting stories both of which don't make perfect sense and i don't want to make of it So the conclusion of this for me is Can I get another random person To tell me a corroborating story For one of these two stories So we can know what happened

Dude I want I want to meet Wishbone Okay that's this code name I want to make a wishbone I mean I'm sorry that we docks his password strategy Because he probably has like what America for July He's just like I have July That's America 12 or America 26 for this year. Oh, crap.

You know what? Wishbone 25 is the password to a bunch of things at Amazon, or not on Amazon, for this guy's poor account. So I do apologize. I probably shouldn't have said what the password actually was. Don't worry. We won't fix it in post.

It's too funny. It's out there now. I apologize Wishbone 12. But if you can anonymously confirm or deny this story, that would be very helpful. because now we just don't know what to believe. Honestly, from all my experience, this sounds very, like, everything but the password part sounds extremely reasonable. Okay.

That's the same with me. That same, same, I had the same opinion, actually. You can't copy my opinion. I said it first, okay. I was going to say the same thing. I was on mute. I was kind of just illustrating this whole thing, like, in my head on how, like, it was raining outside, storming. It's a crazy phone call.

I just love the fact that it all started because you broke your keyboard. Yeah, and none of this would have happened otherwise. We would never have known the secret story, the secret true story. The secret of the Dynamo DVD. See, that's the thing is, I don't understand why they wouldn't publish that, honestly. Just be like, yeah, we had legacy code that was locked behind a password wall in which we had to figure it all out and restore from a very long time ago that was deleting, files in production that should never have been there. I can tell you why.

Why? Sounds terrible. Because the MBAs who read that will think that that makes Amazon the worst software company in the world even though it makes them above average. Oh no. Oh, no.

Oh, my gosh. Well, that was a beautiful story. I'm actually, I'm very happy right now. This is not just made my day. This is made several weeks of my life. I'll be giggling to myself for the next while. I was like, I don't really care if the story's true or not because I loved the whole thing. Like, the whole thing is cracking me up.

And I also, like, it did remind me that I don't, like, I really wish even, let's suppose that the official explanation is the actual accurate one. That is what happened. This story was just a random, like, guy who was talking at a bar and was just making stuff up, right? That someone happened over here. Totally plausible as well. but I don't understand the pathology like the pathology they explained for the dynamo db thing just sounded really weird because I was like

it didn't like it didn't add up to me and I you know it's entirely possible that's like well it's because we've like abstracted away the details that would have all made sense where you're like yeah it's like it's a lot hairier than what we said but what we said is basically true right I'd still love to hear that too so if any if someone at Amazon ever wanted to come on to actually explain like no, here's why the thing could not just run the DNS and actor again and everything was fine.

Like, you know, that would be cool too, honestly. So I'd love to hear that. I'd love to hear the more technical explanation for what it would act. I would really like that. Because there was a, some guy made a testing utility for Google a while back for like launching things in specific regions and then do some sort of auto deleting and all this. And then he accidentally deleted like Australia's pension, if you remember that. Oh, really? Yeah, this happened.

This happened just a few months ago. And it's just like, this just happened. Someone makes a tool that's like, it's great for testing. It's just for me for being able to launch and make my test work really, really fast. Then someone actually runs it in prod and forgets a hidden magic seal I flag. And boom, 90 days later, the entire Australia's pension was deleted. Wow. I can find an exact article.

Yeah, it's basically like, wow, this thing makes it really easy to change environments. That's super helpful. It's like, oh, well, it makes it super easy. to change environments. Casey, how long was this whole encounter? Oh, great question. Not that long.

I mean, it was just, they literally just came up and it's like, I wanted to tell you this story because I like... Did they sit down with you and... No, we were in a guitar center. Oh, wait, guitar center.

Yeah. Casey was playing stairway to heaven. And, you know, the guy came up and just loved his tunes. It was great. Oh, by the way, so Google, it wiped out all the data and all the backups as well.

$135 billion worth of pension accounts. Are you kidding me? Luckily, Australia had a secondary cloud service to store things in, or they would have just been like SOL. Who, who, was it someone in the Austrian? Like who, who, this was a Google person who did this? Yeah, yeah. Google had a testing harness that would just like help launch up environments and all this.

And if you forgot to set a TTR, it set a hidden TTR of 90 days. if I remember the story precisely correct. Yeah, uni super. Yeah. So, 90 days later, I was like, okay,

I know where all the stuff is and where all the backups are. I'll get them all out of here for you. Don't you worry. I can delete those. No problem. I do think, like, this is a scary thing that you learn

when you first start working at, like, a company as opposed to just, like, you know, programming, you know, on your own in school or at home or whatever you were doing,

is like, if you write something there, no matter what, no matter, it does not matter. Like, if you write something, there is a non-zero chance that it will be used for potentially decades. It doesn't matter. It does not matter if it is utter garbage.

It does not matter if you tested it. It does not matter. If you put a thing at the top that's like, do not use this, it will destroy the world. It does not matter. Trash will not read the comments. People will just deploy stuff. Yeah.

Well, it's not even a possibility for me. My code is definitely still running in production 10 years previous. I was at a meet and greet and someone came up and said, hey, I work at Netflix and I was like, uh-oh, and he's like,

yeah, I've seen your code multiple times. I'm like, years later, there it still is. All right, let's go, baby.

And so it does happen. Oh, by the way, I personally think this is why AB testing sucks. I'm going to throw this out there, hot take coming in, is because with AB testing,

you kind of go like a little bit more Rambo style, a little bit cowboy coding, and you go out there and make like a less supreme experience because you just want to be able to do this kind of test to see if it's better for the customers, blah, blah, blah, blah. You get it out there,

and we know every single time what happens. All right, well, we are going to go on to the next A-B test. We're just going to put that in production. No need to productionize it. We'll just let it go as is, and boom, you're off to the next one. Classic. All of your crappy ideas caught 4K cinema out there. And you move on to the next A.B.

It's true. Move right on. When I was at Epic, there was a bunch of like a serious amount of code that was still unchanged from like the 70s or something. Like I don't remember how long ago it was, but it still had Judy Faulkner at the top CEO. She wrote some stuff and you're like, dang, that's crazy. In Cooball. Well, it was in mumps.

Mumps. Right. Oh, yeah. Mumps. Mumps. Let's say the coding style looked a lot different in those ones from like the 70s or 80s. They had developed some conventions that. fixes but it's like whoa that is insane you just stumble on it and like unchanged forever like

i'm closing this file i don't even want to have access to this i don't i don't this is load bearing for real oh man because if there's been a bug there there's countless customers that are relying on that bug to exist like there's there's billions of dollars residing on that bug you do not fix or touch this you close your editor and walk away for a little bit if you accidentally end up in there. That's funny. I've had a very similar thing happen where I was chasing down a bug for computers unlimited back in the day, which then later became known as Total Information Management System,

which is like how a bunch of industrial medical, like planning for equipment and shipment of, you know, auction tanks and all that. And it was just like, oh, yeah, Bisco. I was Biscoe worked there. And one of them is just like, oh, I have to go into Synergy Land. Oh, what's synergy? And it's just like this whole other language. It is old as me.

It's like old as dirt. and I'm just like, I can't touch this language. I refuse it. Like, this is, if I touch this, I'm breaking everybody. I don't know how to test it. I don't know what the outcomes are. I don't know who's relying on this. I'm like, 22 years old.

And they're like, just fix this bug from the 80s. And I'm like, no, I'm not touching it. I did a really bad one at Rad Game Tools. Like, definitely the worst thing I've ever done in my career by far is, uh, so we, we didn't have like a build system of any kind. because back then there was only like a new make or visual studios like built in one.

There was not much for cross platform build things at all. So I'm like, okay cool, we got to get this stuff running on like Mac as well and things like that. So I'm like, I'm going to make a build utility like this famous worst decisions of all time. So I make this build utility called C-DEP

and it is like this it is absolute garbage. Like it's basically like it has one really kind of cool idea like I liked the idea of it which is that you don't write separate build things

just the it reads the comments of your source code and that is how it does the build so sweet sweet he doesn't know Tom's a genius okay he doesn't know Tom's a genius sorry we'll go over we should go over Tom's a genius in a separate

stand-up episode we should okay I was about to ask who Tom's the Yeah, it's too long, but we will do a full episode on Tom's a genius next week. Yeah, as slides. Yes. So I'm just like, this is great. So like, here's how it's going to work. And again, all of this stuff is like fairly new at the time.

So the things I'm saying are not derogar at this time. I'm thinking these are like really cool, like innovative things. It's going to, you're just going to point it at one C file. It's going to spider out from there. It's just going to like parse the includes. It's going to figure out all the things that it should depend on. It's just going to do it. So you don't have to make, make files.

You don't have to list the dependencies. It's just going to point at the thing and go. And if you need any kind of build stuff, you just put it in the comments. Right? In the comment, you just say, like, here's the build parameters I want. It sounds nice. Like, I mean, that is like, you know, theoretically. Right.

And if you don't know, yeah. If you don't know what the hell you're doing, you're like, this is going to be great, right? Yeah. I still think it could be great, Casey. Yeah, well, you and me when I was 22 or whatever it was. Anyway, so I write this thing, and I don't know anything about how to do the, like, Like, it, I don't really think it through. I'm not good enough to, like, figure out how to do it well.

So really what it ends up being is sort of, like, just a string replacement kind of language. So you, like, you can define stuff in this, like, in your comments, you can define something. And really all that's going to do is just replace that thing with whatever you defined it to be. That's all it could do. And it does this by like literally string rewriting. So all it is is like if you said like foo is now this crap, then any time it sees foo in the things that's doing, it just replaces it with that.

And that's it. There's no, there's nothing else. It doesn't have anything else. And to the extent that it has conditionals at all, they're just like it won't rewrite the thing. That's it. Right. So you reinvented pre-processer macros?

Yeah. But it's pre-proster macros at build time. That's what it was. It's preposter macros at build time. That's what it was. Nice. So this thing is a tremendous pile of garbage. It's horrid, right?

And the worst part is it's the worst kind of horrid. It's the kind of horrid that seems good at first, right? Because you're like, the first time you point this thing at like a C program and it builds the whole thing and the dependencies all work. And so you're like, oh, my God, this is amazing. I just like revolutionized the build process for the world. I'm amazing, right? and you don't realize that actually what you did is just fucking destroy. Like everyone's attempt to build things at this company for the next like 30 years.

Okay. So that, I do that. And Jeff Roberts, who's the guy who runs rad game tools, excellent programmer. Excellent, excellent programmer. He goes like, oh, cool, okay, so someone made this like little build utility. That's great. I can now start modifying it to do what I want. and what Jeff wants is always for things to run very efficiently.

That's what he wants. So he adds like multi-threading, like running, distributing it on cores, all this stuff. He optimizes like the thing so it runs faster. All the while, no core improvements to this thing because it's still just a string replacement thing. Rad ships on dozens of platforms. More platforms than any company I've ever seen. It ships on everything. Like, you know, it's like, from like Game Boy Color to like, you know, Amazon's special arm server deployment, they run on everything.

So he builds this giant, like all the builds scripts and stuff. He builds all of like the compile time settings, all that stuff. There's like tens of thousands of lines of C-Dep code by the time he's done with it after several years. It builds all the stuff for like, for the video decressor, all that stuff. now to this day I am told the rad builds still run on C-DEP

it is undemuggable it is completely unreadable it's huge amounts of institutional knowledge that Jeff has built up to build on every platform known to mankind and to distribute the builds

it is so insane so like literally I don't know what to say other than like I have created so much pain and suffering for that company just by creating this this freaking terrible utility. If I had just done a better job on that, it would have saved, I don't even want to know how many hours people have lost spending trying to figure out why their builds don't work because of this piece of junk. Is the Depp and C Depp short for depressed?

It should be. It should be. Casey, you truly sound like a 10x engineer. The truest 10x engineer makes 10X more work for everybody else. And that is what I did. Yeah. Because literally the not having this tool would have been much better. Yeah. Right? Because then Jeff would have like made when if he ever did all of that work, he would have made it in something much less crappy.

Like he would have just opened up a C file and made his own little build utility that wouldn't have sucked. Right. So it's like it's like I basically took all of this other work that was actually useful and I made it suck. by having it start with something awful that it never got rid of, right? And multiple people have had to come on and like fix crap, like, or try desperately. But there's only so much they can do because now there's this huge, unparsable, gigantic, like, you know, tens of thousands of lines of build knowledge about, like, how do you properly do a annotated debug build of a GameCube binary from like, you know, the year 2001. or something like that, right? And so, and no one wants to fix that.

No one's going to rewrite that. Yeah. So I basically ruined, I ruined that entire. It was awful. It was really bad. Casey, I just wanted to say this will make a lot more sense after Prime does his Tom talk, but we just got to say, Casey is a genius. Casey is a genius.

Yeah, I agree. That was a real genius move. It was a real genius move. Okay, hey, TJ, or hey Josh, we need to put this in the beginning. TJ, where do we find? Because obviously, this is the extended. By the way, it's not, it's not Jay Diesel anymore. It's obviously Jay Muratory. Uh, so trust me, it'll all make sense here in a moment, Casey, when you learn about Jay Diesel. It's Medesel, Muratorie, domain specific language. I love it. Oh, God, yeah, yeah. Medesel. But the, Medesel. So, TJ, can we call it a Dissal? Can we call it a domain inappropriate language or Dill? It's a Dill. A domain. A domain. inappropriate language. If you enjoy

this podcast, check us out on Spotify, The Stand Up, also on Apple Podcast. Personally, I can't find it on Apple Podcasts, but I've been told we have one there. It will be linked in the description. The name is Casey, Computer Enhanced.com. Trash. He works at Netflix, by the way. Teage. T-streams, by the way, the Primogen.

Thank you for watching the stand-up. coffee in here.