The Standup with ThePrimeagen - Casey breaksdown AWS outage (Whiteboard Edition)

Episode Date: January 13, 2026

SQUASH THOSE BUGS with Sentry - the best way to monitor your bugs in production (we know you have a lot of them) and fix them with ease. Check out https://sentry.io and get started today! Casey breaks down the AWS outage in great detail with literally the coolest whiteboard demonstration you have ever seen.

This episode of the stand-up is going to be extra special because Casey is going to do the intro. Casey, what are we talking about today? Hello, everyone, and welcome to the stand-up. The number 45-6 best tech podcast on Spotify, according to the most recent, uh, something. True. Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, uh, anyway, sorry. Today on the stand-up, I wanted to cover something. I'm going to talk about the AWS outage that happened in October,

but I'm doing so because I kind of wanted to talk about a bigger thing, which is the idea of actually understanding something versus saying you understand something. So, like, one of the things that happens a lot, especially, I think, to people who are earlier in their programming career, like if you're a junior program or something, you're coming in, and I know this was certainly true of me, is you want to seem like you know stuff, right? Like you don't want to seem like you don't understand what's going on. So there's a lot of like external pressure,

whether it's really there or not, you feel like you should kind of say that you understood something or pretend to understand something, even if it's like a little bit hazy or you didn't quite get it. And even if it wasn't your fault, like even if the thing wasn't explained properly or didn't include, like, important information, you're still incentivized to basically act like you knew what it was, right? Because it just makes you seem smarter or something or at least doesn't make you seem junior, right? And so one of the things that at least I've found as I got older and programmed had more programming experience and things like that is nowadays, I like almost over-ask for things to be explained.

Like I don't care about looking dumb at all. I'm like, wait a minute, go back. Like, I didn't understand that part. Like, what do you mean by this? Or like, what's that term mean or whatever? Because now I just don't really care about that. Like, I'm not as worried. And I want to actually know because I've had so much experience programming

where I thought I knew something or I pretended I knew something and it came back to bite me. I'm like, I want to actually know. Like, I want to be sure that. that when I have an explanation of a bug or I think I know the reason of a performance slowdown, I always in the back of my head, I'm like, if I haven't really gotten to the bottom of this, it could be something else. It could be that the real thing is still hiding in there,

and I just don't know because I haven't really looked at it all the way. I'm just moving on because it's convenient or whatever. And so the reason that I wanted to talk about the Dynamo DB outage is because recently there's been kind of a string of high profile outages. So there was like a big one that took down Google, and it turned out it was a thing where, like, they didn't handle a field being empty, right? So they're programming, the way they were programming,

they were like, okay, we have this thing, we load some JSON, and if there's nothing in the JSON, it's just we like, we de-ref a null pointer or something, right? It was like literally that, right? And then there was one with CrowdStrike, where they were like they took down the entire world with blue screens and that was they gave a very good it was like a really good explanation of it they were like we have here's we do this certain array sizing thing and we had too many rules so it like overflowed the array right and so these were

like pretty good when they when they gave what they call rca's or root cause analysis right when they said like here's why we went down when i read them i didn't feel like there were a lot unanswered questions in my mind. Like maybe I didn't know like literally the line of code that because they maybe didn't publish literally the piece of code. But they gave me enough that I was like, okay, I understand how someone wrote this code and I understand the stupid thing that they did, right? That like, okay, don't do that thing. I understand. And I'm totally like, okay. With the Dynamo DB one, because it came up on this podcast, right, we talked about it when that dude at the guitar center, right, was like, I overheard someone talking at the pub, right?

Yes. Incredible. Here we see the elusive programmer, a simple creature that spends most of its time working alone, often in darkness. Well, what's this? Someone being wrong in the internet. Our coder springs into action, reaching top speeds of 120 words per minute before flash. A light mode website. The natural enemy of these code lovers stuns our friend.

The chase is called off. We'll have to get them next time. When not on their computers, they can spend hours drawing crude symbols than something they call whiteboards. Researchers have discovered thousands of dialects, often with more than a dozen used in a single office. However, no linguists has yet deciphered what their purpose is. Vane creatures, their bodies have evolved over a millennia

to be able to sit in unusual postures while looking at themselves online. This will often last for many hours. Using the excuse, they're waiting for code review, repressed to why they're so inactive. And finally, after a long day of accomplishing very little, our keyboard warriors ready for bed. Quick read and its lights out. Good night, little coder.

So how do I sleep so well at night? Well, I have Century to help me crush those bugs. And I'm not talking about Little Teen Tiny South Dakota bugs that die in the winter. I'm talking about big, mean jungle bugs. And I'm not scared of any of them, by the way. But I can squash those bugs with Sear by century. So I was kind of a little more motivated about that one to go like, okay, let me go see like what, how much information they've posted.

And I had read, I had already kind of read afterward they had a summary where they posted an RCA and it was very vague. Like the RCA just did not really explain very much. I then noticed that they posted a full presentation. Like at ReInvent in December, they, or I guess I don't know if ReInvent was in December, but the video went up in December of the Reinvent presentation where they covered this outage

so I went and watched all of that and after having read the entire RCA and watched the entire presentation I still was left going I don't see an actual explanation of the bug here right like I'm trying to figure out what the actual bug was and it just wasn't ever explained

and so what I kind of wanted to do was just talk about that go through why I don't think they explained what the bug was and just use that as an example of like, I don't think people should just go oh, okay, I get what the bug was. Because people have like replied to me and gone,

oh, here's, let me explain to you what the bug was. And then they just explain the same things to the person. I'm like, that's not the bug. Right. So everyone is like incentivized to go like, I understand it because I read it's like, no, if you can't tell me what the actual bug was, then we're not done here, right?

Like, we should have that fuller explanation. So does that all make reasonable sense? Like what I'm saying? Yeah, first off, I just want to say, I knew exactly what you were saying, Casey. Like, wait from the start, right, T. It's like, right away, you were like, okay, I know, I know what that. I know exactly what you're saying. No questions on my end.

No blockers. Thanks, everybody. I'm great. I'll see you guys tomorrow. You know, no problem. I just want to say, I really like listening to Casey talk on the podcast when I listen on Spotify, but also just right now, like I could listen to you talk for an hour. Great shout out, too, for the Spotify.

podcast. I was going to say like, especially when you listen on Spotify. Your choice. The quality is incredible. You also get the bonus extras, right? You get all the banter before and after the actual main. Extra yapp. Oh, really? I didn't know that. We started posting longer, longer versions on Spotify that are like more of the extra yap time. Less of the on top. It's not on topic stuff, but a little more yappanings on Spotify. Because the live audience gets the yapping. They get to come in here. here they could hear about trash and his Pokemon addiction, which you probably don't even

know about because you were listening to this on YouTube, right? You don't get to hear all the fun stuff. But that's kind of a hard sell for the first 10 minutes of a YouTube video. It's a very hard sell for a YouTube video. Be like, I'm going to watch four guys talk about something I don't even understand. And it's called Dynamo DVD. Since we're starting the podcast, maybe we should introduce Adam. Oh, yeah. That's a very good point. We haven't done any discussions at all. Hello. Tell us a little bit about you, why we brought you on to the podcast today. because I am at Teague's house.

And that's number one. There is. Number one reason. DJ requires all people who visit his house to be on the podcast. It's been awkward at a couple times. Yeah. Yeah. Oh, but Adam, who are you really other than an AWS hero?

I'm not even that. I was an AWS hero. R-I-B. Too soon. You can kick that out of the superhero group? Like, how's that work? You just don't get renewed. I was a one-term hero.

and they decided. Oh, is it like a paid up thing? You pay to be a hero? No. No, I just, I didn't really care about it anymore or talk about it ever. So they were like, maybe he's not a hero anymore. Now he's a villain. Casey looks like he's part of like some murder mystery.

He's standing now. Oh, dude, we're about to get like the, what is it, Nick Hill? What's the person that does all the like drawing on the board and then it shows up? Casey Moratory. That's what I'm thinking of. Muratory. Is it a miratory or is it? moratory.

Oh my God, you're about to do visuals, aren't you? Yes, I know. This is the best podcast. It's literally, this is the best one to be a part of.

It's pronounced muratory by my family, almost like there was a why there, like muratory. That's correct. It doesn't really make any sense because in Italian, it's an Italian name, and in Italian, it'd be muratory.

Mutory. No, no, no. It's not a murator in Italian. It's miratory. Yeah, it doesn't make it. So how it got Mure, I have no idea. That was some Italian-American, like, immigrant thing that happened, I guess. I don't know.

So here's effectively what they said. They have these things called API Endpoints. They call them, right? And these are the domain address, like, if you look up in DNS, it's the name that you're going to look for to know who you're supposed to send, like your DynamoDB requests to. And these things, I guess, look like this. and Adam can probably confirm this because he is or was a hero.

They look like, oh, it's behind. Yeah, we're a few seconds behind because our video disappeared on Rivers. Yeah, yeah, yeah. Okay. Oh, there we go. So they look like dynamoddb.us-Uest-East-1.API.ws or something like this. And I guess it depends whether you're using IPV6 or IPB4. Like, they have different names depending on things or whether you're using like a specific, like,

they talked about governments used like a different one or whatever. So these names are like names that you effectively hard code I guess into your application where you're like when I need to do something with DynamoDB I'm going to like ask for this. Does this make sense? And does that

sound right Adam to like because I don't use AWS stuff? Yeah yeah that's all right. So you know you you ask for something like this and you're going to send you understand perfectly. What? I do. I mean I know what he's saying. Yeah. Yeah. So that then is going to redirect you somewhere

because obviously there isn't like one machine that's going to handle all the DynamoDB traffic in the entire universe. Even if you subdivide it by region, which you can see here you're kind of supposed to pick a region. I guess you don't send it to some main address. You send it to a regional address. Or maybe there is a main address you can use that we'll figure it out. I don't know.

But anyway, at some point you're talking to this. And this needs to point to effectively, like a load balancing scheme. So this thing is supposed to point to effectively what they called a DNS tree, although they never really explained the tree nature of it at all. It sounded more just like a weighted array, if you will, where you just said, here's a bunch of machines, and you're going to pick those machines based on weights that we set

so that we can load balance, right? So if a machine gets behind, maybe we set its weight to lower, and if a machine seems kind of empty, we set its way to hire. And so they called it a tree, so I'm assuming it's a tree. They never explained what the tree part of it was. But this name is supposed to point... Can I interrupt for one quick second? By the way, someone did get their L6 promotion based on that tree.

So I do think next time you should find out what that tree is because that meant a lot to somebody. There was a packet and engineers happened. I do agree the tree is probably important. It's just not important for the bug. And even this, okay. So that, I will say there was no need for them to explain the tree. So I'm okay that they skipped out on what the tree is doing. But I got a quick question as well.

Yes. Is it called a tree because it's a root cause analysis or no? Yeah. Yeah. No more jokes. We're too off topic. I'm sorry. I'm sorry.

I'm sorry. So anyway, this is supposed to point to that. And that sort of, this load balancing scheme basically of DNS entries. And the way that they described this in their presentation is they would use a thing like, I'll say, plan 145. Dot DynamoDB, like DDB dot AWS, right? Now, this is the root of that tree, I guess. Not root cross analysis, but like this tree, this would contain, like, this is the top level record

of a bunch of records that allow it to do its load balancing. And I assume Route 53 kind of has this load balancing capability. I'm reading between the lines of their presentation. They didn't say that outright. But I'm assuming Route 53, which they're doing all this through, you know, which is their own DNS thing, is allows that load balancing to happen by you just set stuff up in here that says how the load balancing should sort of be working right now. And then it will pick the correct machine based on like some kind of randomization in the weights or whatever. Now what they said was this name, which really does exist.

And apparently there's a tree or something like this. This name is one that they just kind of used for the presentation. They never actually used a human readable name for this plan like 145 that I've written here or whatever. It was really a hash of something. So it would really be like, you know, 0A, F, E, 1,2, 9A or something like that, right? Is actually what would be there. So if you went and looked, you would not see a human readable name, or at least at that time you wouldn't, I guess. You wouldn't see like plan 145.

You just see that. And so the idea was, okay, a user goes to use it. They query this name. Route 53 will direct them like to hear. And this thing is some kind of a load balancing tree that Route 53 can use that will allow you to get where you need to go, right? They will give you an actual machine you can send traffic to eventually. Again, they did not describe any of that, so I have no idea how any of that works.

I've never touched or used Route 53, so I have no idea. But we'll just assume that that happens because it doesn't matter for this bug. We do have an AWS hero. So if you do, if you are confused, you can always ask Adam and he may have further insights. I mean, yeah, go for it. Well, Route 53 does have a lot of different ways you can like split the traffic. So yes, weighted is one of them. And that sounds like what they described.

So somehow they've set up these records with that. And they just didn't say how, but something. Something in a tree for him. You did that. My guess is there's like a weighted, like the tree has like weighted, like there's a couple of weights at the top that branch out to more weights or something like that. because that's easier for it to deal with because there's a lot of them or something, who knows? Anyway, I have no idea. Point being, this is what's supposed to be happening normally.

Now, the reason that this is called Plan 145 here, even though it actually would have been some hash code, but they refer to it as like Plan 145, is the load balancing, as you might imagine, has to be kind of continuous because the DynamoDB machines are like doing stuff all the time. They're becoming more overloaded. There's machines are going down or crashing or who knows what, right, could be happening being taken offline. New capacity can be added. And so this stuff has to be updated constantly, like all the time. So this main API endpoint that you connect to, it constantly has to have that tree that's pointing to be adjusted.

And so the way that they do that is they create another tree, the tree that they're going to move to, right? They create like, you know, Plan 146 or something, and they make the whole tree here. And then when they're ready, like when this tree is done, they take this, you know, this record here. And instead of it pointing to that one, they point to this one, right? So they make the new one and they move over to it by just changing that name. Now, for some reason, and this reason is not really explained. The way that they've set up that process is they split it into two pieces. There's something called a planner which figures out what the new tree should look like, basically.

So you can imagine there's some machine called a planner, and I don't know if it's an actual machine or if it's just a process running on some machine that's running other things. Who knows? But there's something called a planner, and as far as I could tell, there's only one, meaning there's just a planner that sits there and figures out what should the new plan look like that we're going to switch to. And it's constantly doing this. So it generates plan 145, then it generates plan 146, that it generates 147, 48, 9, you know, blah, blah, blah, blah, right? And it just keeps putting out plans for all of eternity because that's its job.

Now, it never actually creates them, apparently. Its job is not to ever make them in Route 53. It's just to figure out what they would be if someone were to put it into Route 53. Then they have three enactors. These enactors get the plan from the planner and they put it into Route 53. This just makes sense. Now, one planner, as far as I am to understand for the station, three enactors. There was no explanation for why this would be the case.

They said the reason there are three enactors is because it's supposed to be fault-tolerant, like if one of them goes down or something. But they never explained why you wouldn't then need three planners, because if the planner went down, then the enactors have nothing to enact, so it didn't really make any sense. So there wasn't an explanation in the thing about why this structure looks the way it does. It's not really that important to the bug that it looks this way, although it kind of is, as we'll see later. So I was a little weirded out by the fact that they didn't justify this, but that's fine.

So hopefully that makes sense. We have a planner. We have three enactors. The enactors are all trying to enact this plan, right? Now, what happens here is that for, again, reasons that all the only thing they said in the presentation was it makes it easier to, to reason about. This is the only information you have. They said it makes it easier to reason about.

Because it makes it easier to reason about, these enactors use serialization. So instead of them just trying to create records, and if the records are already there, just not creating them or something, in other words, I have three people running. We all want to create, you know, let's say this top level record,

plan 146.db.d.b. You know, AWS, right? We all are trying to do that. One of us does it first. The next person tries to do it and it's already there or something, right? We're all trying to create the same record. So in theory, we could just have three people randomly hammering on whatever part of the plan they're trying to hammer on. And in theory, it should kind of all work, right?

And I sort of got the sense, although you didn't come out and say it, I sort of got the sense from the presenter that he would agree with what I just said, meaning that they could have just had them run arbitrarily, and it would or should be okay. But, he said, they use serialization to make it easier to reason about. What that means is instead of these enactors just hammering on it like that, what they do instead is they attempt to acquire a lock for whatever the endpoint is that they're trying to update. So in other words, if this person is trying to update one of these things,

And I got the sense that it was if you're trying to update this one, but it could have been if you're trying to update this one, or it could have been on both. They never really 100% said, if I remember correctly, exactly where the locking was occurring. But the locking occurs by them going, okay, I'm going to create a lock that is a DNS record. And by using the fact that Route 53 has the idea of an atomic

which is, you know, I can do two things, and if they both wouldn't succeed, then it won't do either of them. They basically made a locking system that locks via Route 53. So Route 53's DNS records are actually the lock record, if that makes sense. Can I ask a quick question? Yes. You said it does this through serialization? I don't quite understand what that means, because I thought serialization is just converting from one memory to a different memory representation of something. I'm struggling in that part.

Different serialization. So, yes, that is serialization. In this case, in this case, we literally temporal serialization, meaning they wanted these enactors to have some kind of a way in which they would organize their behavior into an order rather than just being arbitrary. And the way that they did that was locking. Okay. So what will happen is, instead of this person. just doing whatever it is they're going to do. Like, okay, I'm going to like, I finish this, I'm going to point this guy at Plan 146 now.

Instead of doing that, it attempts to acquire a lock on like this, right? And if it doesn't get the lock, it won't make the change. So only one of these enactors can be in the process of updating this at any given time. Does that make sense? Mm-hmm. Now, again, exactly what they were trying to do with that was never explained. They just said makes it easier to reason about and left it there. So I don't know why they thought this was an improvement.

And amusingly, it's what ends up uncovering the bug. So it wasn't an improvement. If anything, it was probably bad. But. So, Casey, are you saying they don't have, like, they don't have a good reason for, they're saying we're going to make the enactors run almost like one at a time. Why do they have a, why do they have three enactors? I don't understand.

Like, why do they not just have one? They just don't say that. We don't know why. And they didn't quite explain, like, I didn't really. hear an explanation for how you have three concurrent enactors, you expect them to be able to go down, which is why you have three, but they're taking a lock? So what happens if this guy takes the lock and then goes down? Like, I didn't hear an explanation for that either. So this was all very confusing to me. Like, I, I'm not complaining about it as part of what we're talking about here,

because it's not important for the cause to me. But as a presentation, I had so many questions. Like I was like, I don't understand why you did any of this, to be completely honest, right? And maybe that's, again, part of it could just be that I don't use ADWF services. It might be that some of these things would be obvious if you are someone who regularly uses Route 53 or something. You'd be like, oh, it's because locks can be set to a timeout. I mean, I don't know, right? But anyway. So, yeah, so they're doing that.

And what ends up happening for this, the thing that uncovers the bug, is that what ends up happening is these enactors, when they don't get the lock, they just do like a back off, right? They'll basically just do like, okay, let me wait and I'll try again. So this enactor tries to get the lock, but somebody else already has the lock, so he just waits a little while, he tries to get the lock again. That's what will happen, right?

and what they said happened was they hit a pathological case quote unquote where one of the enactors is you know has enacted some plan and that plan let's say was pretty old I think they used 110 was an example that they used so it enacted plan 110 and it wants to point you know it's like I got to set the API to point to my 110 tries to get the lock to update dynamoDB.us east.1 or whatever and fails because someone else is enacting plan 111 or something like that, right?

Or plan 109. It could have been a previous plan. So the other enactors are doing it. It can't do it. It backs off, right? And remember, this enactor here, we're on 110. It's trying, it really wants to enact it. It tries again, someone else has the lock now.

It tries again, still locked. This person sitting on 110, desperately trying to enact it, can't do it. Apparently, this just happens so many times, that the other enactors and the planner is just turning out new plans this whole time, right? The other enactors, they get up to like 1.45 or something and 146. They're enacting plans that are like way ahead of 110, right?

And this guy still stalled because he just unluckily never gets the lock. Right? Finally, at some point, after like Plan 145 has already been enacted and pointed to by some other enactor and all that stuff, Plan 110 this enactor that's still trying to do it, finally

gets the lock. I mean, it's like, yeah. And so then he says, okay, we're pointing to 110 now. Yes, right? So now it's on a super old, stale plan. But this really shouldn't be a problem, right? Because eventually the next time some enact or has something, it's going to be a much later plan. They'll just enact plan, you know, 146 or 7 or 8 or whatever. And we'll re-point it back to this and we're back to a fresh plan.

So everyone will just have bad load balancing for like a few minutes, but then it'll be fine, right? they did have bad low balancing for at least a few minutes right yes true uh well it's a lot worse than that that's what was supposed to happen right meaning that's how they would expect this to work too okay the problem is these they also didn't want route 53 to become clogged with all of these records because if they just left them around eventually after you know three months you have like 8 billion records that you stuffed into Route 53 for every, you know, couple minutes you're putting in this big tree of weights and stuff. They were like, okay, at some point, we should just clean up these plans.

So enactors also look for plans that are older than a certain amount. And if they are older than a certain amount, they'll delete them. So what happened was they pointed to plan 110. This enactor finally gets the lock. It points to 110. another enactor's like oh wow, 110, man, that is old. We should get rid of that and deletes it.

So now, DynamoDB, U.S.East1. API-D-B-U-S. is pointing at a record that can't be resolved, right? It's just something, it would actually, again, it wouldn't look like Plan 110, it would look like OAF-E-129A, some hash, dot, right, D-D-B-B-A-WS. But it's pointing at that name, and if you ask that name, you get nothing. So what would happen at that point

is everyone who was trying to get a endpoint to send stuff to would get back an unresolvable name, basically, right? And I don't really know what happens in Route 53 when that occurs, but you would basically be getting back something that you either couldn't use or just gobbledygook for an IP, who knows.

But whatever it was, if you attempted to actually use it, you weren't going to get a response, right? Interesting. Is this because AWS doesn't use enough rust because that's obviously a use after free bug. And so I think Russ would have solved that, right? If you rewrote Route 53 entirely in Rust, obviously, all of these problems are not there. No, to be specific, I do think in the presentation they did say, not about Rust, but they did say what would happen specifically,

which is I think when you asked for this thing or either this thing or this thing, I don't know which one they were referring to, because I can't quite remember, you would just get back, I think, that says no records found. So that's the endgame of what would happen. Whether it was from asking for this or asking for that, I'm not sure, but to just get back, no records found. That's what you would have received when you were trying to call that API. So whatever, like, whatever, like, library you were using to, like, use this,

to use DynamoDB, it would just be like, hey, no records found, bro. Sorry, right? So, this, if you ask anyone on the internet, right, they're all like, yes, they explained the bug. That's the bug. Like the bug is that there was this race condition, right? Everyone, because everyone, as soon as you say race condition, everyone's brain shuts off. They're like, oh, okay, well, it was a race condition. Done, like, nothing to see here, right?

So they're like, it's a race condition. They explained it. It's like, no, they didn't explain it. Because if you think about what would happen here, immediately after this, everyone's getting this. A new an actor. A new one actor will just enact a new one, right? And so the bug. right is why didn't that occur that's the actual the actual RCA that I wanted to see is

why didn't the next inactor come and fix it can I can I throw out something else wouldn't it wouldn't also be a bug like why write a record so old that it should be deleted immediately well it wasn't it was it was this guy had written it quite a long time ago and it was it the way well I mean if you're asking why didn't they write in actors with better code yeah that's a break okay okay okay fair. It seems like if you are updating to something that should be deleted immediately, isn't like that feels like the problem right there.

You've done something wrong long before. Yeah, even though it doesn't really fix the theoretical structure of this thing, a simple check in this guy when after he finished backing off on the lock, he should maybe check to see whether he's about to set this to something that he would delete if he was running his deletion code is probably a good safety measure. But yeah, so 100% agree with that. Okay, okay. That an actor worked really, really hard to get that.

record, great. It's been waiting a long time. And it's going to have its Pokemon cards. Anyone ever waited. So just let him write the record. Okay.

So I wanted to hear about that. Unfortunately, if you look at the presentation and you look at the RCA, it's nowhere to be found. The presentation at least has one 12 second little tiny

chunk where it does say where the bug roughly would. be. And so let me explain what that is. So what apparently occurs alongside this. So when you do DynamoDB U.S. East 1, when you point that at your plan, you also do another operation at the same time. And that operation is to set rollback. I think it's DD. Is it DD? Is it DD? B.rollback.AWS, I don't remember exactly what it is here. There is a rollback record.

It sets that record to whatever the old plan was. So if we were here pointing at 145 and we're now going to point at 110, right? This old enactor is like, I'm moving to the 110. It attempts to set, take whatever this name was, right, currently, and move that name, which would have been plan 145, move that. so that the rollback address points at the old plan. Right? And this is just for debugging or, you know,

it's basically just for operator ease, right? If they want to roll back to the previous plan or something like that or if you just want to know what the previous plan was, you can see it here, right? That's part one of what they said about failure. I would want to point out one thing here was this also didn't make any sense to me because I was like, okay, you're telling me that these things update every like, minute or something, what good is it to have one of those? Like, by the time you even logged in, it's been updated from the one that you wanted to

roll back to to some new thing that's actually the plan you don't want because everything went down, right? Like, it's, right, you, you don't want this, you just want these names in a list so you can be like, what was it at at 1230? Like, that one, right? So this made no sense to me. I have literally no idea why this would ever be good, right? It did not sound like it would do the thing you actually want,

which is to be able to mark a point in time and go, we need to go back to 1 p.m. because everything went to crap after that, right? Anyway, so that didn't make it sense to me, but again, not exactly specific to the bug, so I didn't ask why. I'm just saying, okay, that's what thing it had to do. So it can only roll back one version is what you're saying.

Yeah, even though the other trees do exist. So you easily could by just knowing what the name, was. So all this is is putting a human readable name on something you almost certainly don't care about, right? But they don't really, they can't really store that much stuff, Casey. I don't think they can really put, like, I don't know, Adam. Like, they don't have a lot of scale there, right? You like, like, eight bites. Dog, that's a lot of fines. If it were me, I would have just made this a timestamp, if that's what you wanted, right? I would have said, when did the planner, or when did this person point to this thing? Like, when you got the lock, you'd, you'd

change this name to the timestamp and update this in one atomic. So then you just know if I want to roll back to 1pm, I just look for like whichever had the timestamp just, you know, the earliest timestamp, not after that time. And that's what we were running at that time. That's what I would have done, right? But I don't know. So I have no idea why they did this.

They did what they did. You know, it might make perfect sense. Again, I have no knowledge of their system. All these things may make perfect sense. So I'm not really, I'm just saying I don't understand them. They might not be bad ideas, right? There might be good ideas if you understood the rest of the system. So anyway, so what they say, and this is all we get, is this operation,

meaning setting the rollback to point to the old plan that was being, you know, which in this case would have actually been newer in some cases, right? So it's not really the previously pointed to plan, which may be older or maybe newer, doing that activity if that plan no longer existed meaning like it had been deleted like this then the enactor stops permanently so every time

like once you get into a state where dynamoDB us east that one right so we do the whole sequence of steps that we said here this plan gets deleted so now this is pointing at an invalid like unresolvable name we cannot resolve plan dash 110 which is actually some hex code but whatever that was, we can't resolve that anymore. Once that state is true, then the next time an enactor comes and tries to make it point to a new plan,

whatever that new plan is, it cannot, like when it actually gets this far and tries to set the rollback, that will crash it permanently. Therefore, all three enactors will now stop, because eventually all three will try to enact a new plan, they will try to set the rollback first to point to whatever the old plan was find that there's no plan

there and that apparently is just a hard crash. Now I thought the three in actors was supposed to make it so that it had redundancy. Now again this is why I get grumpy with people online who are like replying they're like it was a race condition. It wasn't a race

condition. The race condition is not necessary for this. The race condition is just why you ended up with this name being unresolvable. But if you didn't have whatever code did this badly, it would have just worked. You never would have known. You would have had a momentary, like, you know, minute outage of DynamoDB or something. But I'm guessing there are minute outages of DynamoDB from time to time, right? Like, that's not global news.

What's global news is taking it down permanently, which is what happened here. And until an actual human goes and figures this out, resets it, gets these enactors going again, it's just gone, right? It's just out permanently. So hours potentially, right? And it was long enough, I guess, in this case, to then have cascading failures. You would never have had that. It's just a momentary out. Like, if some people momentarily got an unresolvable name or no records, right?

Then they would just try again. That's usually what happened. Like, with DNS, like that's like your phone, you went through a tunnel, right? That's all that would have been. So I want to know what did the code look like here? How did you write something that if this wasn't a valid name, which it wouldn't even be on stand-up, meaning if you were starting this system

and the operator hadn't pre-configured it, it wouldn't be pointing to anything, right? That's the default case that you would think you'd start with. So if you're going to do this, you would think you would just handle that case because the rollback address could just not point to anything, right? Right? Just take whatever this is. If it's nothing, set the rollback address to nothing.

Done, right? So there's something really weird about the way they wrote this code. And that is what should have been in the RCA. That's the whole bug to me. This is just set dressing for how we ended up having this thing point to nothing. But the same bug would have occurred if someone had accidentally deleted this record. Like some operator was just like, oops, crap, I said it to nothing. This is the same bug would have happened, according to the presentation, right?

Right. So the root cause is not the race condition. The race condition is an aside. Does that make sense? Quick question. Yeah. So I'm legitimately thinking through this. And so that means the thing that sets the rollback probably assumes some sort of struck with a bunch of memory or something has been passed in, does some sort of, like some sort of access, it explodes.

Or do you think this is the same style of bug, which is the one line that took down Cloudflare, which is they just assume it's there and unwrap it? It's in rust. It is memory safe rust. Unwraps it, explodes it. I really don't know. My guess, like in my head, I was like, what is the thing that I see people do a lot of times where I'm always like, why would you ever do this? But it's just because that's the way they learn to program. And I was thinking, like, if you were writing in one of these languages that likes to throw

exceptions for error conditions, this would be a great example of that. So if you had a thing where you were like, oh, I went to go get the DNS record that this thing points to. And normally in a sane programming environment, no one is storing an exception there. If they get back nothing, they just return nothing. Right? And then when the person goes to set DDB.D.D.B.Robot database, they just

set it to nothing, which is the correct behavior. Like, nothing flows, literally the value nothing, flows correctly through this flow. So if you were writing it to be, since it is a core foundation service, assuming you were trying to write something that was fault-tolerant,

you would never do something like throwing So in my brain, I'm thinking, I bet what happens in here is when you ask for this record, they just use some library call or something that throws an exception when the record doesn't exist. And it just threw an exception and the enactor was down. That's my guess, right? And I could be very wrong about that because I'm just wild guess, right? But this is why I want to see the RCA. What was it? It could be exactly the stuff that Trash was talking about.

I mean, it could be the stuff that Prime was talking about. It could be the stuff that I just said. Could be anything. And I want to know. because that's where the actual education would be here. Avoiding this race condition is completely unimportant. This race condition could have lived there. And while it was important eventually to fix it to avoid those once a year weird outages for five seconds or something,

it is not actually the thing that we most want to learn. What we most want to learn is don't write this thing. And we don't know what this thing even was. So how do we not write it? This is why I think it was about RCA. Does that make sense? Yes. Yes.

Mm-hmm. All right. What is most of AWS written in, Adam? It was Java when I knew last. I was about to say someone from the chat said Scala. They said they worked at AWS for seven years, and they said most of it's written in Scala. Well, that's technically Java with extra steps. So.

And that will anger all of them endlessly. So that's really it for me. Like that, this was the thing where I was like, I don't feel like I saw the explanation. and I actually feel like it's important to hear because there was a bad programming practice at the bottom of this somewhere, and I want to know what it was, especially because it helps people like me

when I, you know, I don't really do a lot of architecture education right now, but at some point I probably would like to do some of that because I think there's a lot of bad architecture out there. And so I kind of try to pay attention to these things, like what are the kinds of architectural mistakes that people are making? And I bet this was one of them, right? And so I'd like to know, I'd like to know.

Yeah, I think like what I would have. expect is like at least like one simple reproducible example of like why it blew up like a whole like little code snippet so like and that this is something you brought up earlier is like kind of like how we approach this type of things like if I'm like reviewing someone's code and I see something that looks weird I will always do my best to make my own little sandbox and like prove my theory out and then like actually show them the code like this is why this is probably wrong here's like a small simple reproducible step so I would expect something like that and that also helps me like truly understand because a lot of people like you said they'll

see something like that looks funny but I don't know why it looks funny but I can't stop there I gotta like actually like build it out and then like understand so that's what I would expect and you know like I said the crowd strike and the Google outages I thought were better

at like just telling you that they were like look it was a null pointer DREF in here or it was an out of bounds array because we thought there was only going to be 20 and we put 21 in the config file right and like okay I know exactly what kind of code that is causing that kind of problem, right? And furthermore, furthermore, to like an earlier comment,

literally, as far as I know, everyone who programs in a rust only does it so that occasionally, when they see something like this, they can say, well, if they had written it in a rust, it wouldn't have happened. They were not given enough information to even make that comment. They probably made it anyway, to be fair, but they were not given it. So you have to give one rule that should be followed. in RCA's is you have to give rustations enough information to, if they so chose, correctly say that it would have been prevented in rust.

True. And this, we do not have that. We do not know whether this would have been prevented in rust. We have no idea. It probably wouldn't have, but we don't know. Well, Casey, we do have a pretty good chance because it's like probably would have never shipped. So we would have prevented it. True.

We would have zero enacters because we would. Yeah, no an actor. Set an actors. Yeah. I will say that there's something, Cloudflare does really good job at this as well. They like go in and show like a lot of lines of code and say like this is exactly what's going on. This is, you know, even though the problem's up here, this is the line that exploded due to all these previous conditions. That was me making fun of Russ with the unwrap, which actually wasn't truly the problem.

But, you know, it's just like all these things kind of happen. So they do a really good job. I'm surprised at how poor of a job AWS has done for this one. Well, and the other thing, too, is. is it was one of those things where now it makes me, so it makes me unnecessarily suspicious of you, right? When I read this, I'm like, are you hiding something? Did you not really figure out what the bug was?

Like, you talked all about this race condition, but even from your own presentation, I can tell the race condition really wasn't important. That was just what led to the record having been set to nothing, but who cares, right? Like, that's, like, something that's nice to put in the RCA as like an explanation of why this bug occurred now as opposed to some other time,

But it's not the bug. So it's weird to me, like when I see an RCA that doesn't talk about the bug, now I'm suspicious, right? And unnecessarily so, because if you actually did find it, then just tell me, and now I know you found it, right? So it's like, I think it also is a confidence boost for the people who are looking from the outside who want to know, can they trust this Dynamo DP thing. If it looks like you actually found the bug, I have a little more confidence in you. If it looks like you have no idea what the bug was or don't seem to understand what the bug was, then I'm more concerned. And so I think that's also another reason to do this in your rest of eight. It provides confidence to your customers.

Maybe that's where they fired Adam as an AWS hero too. Maybe it's all connected. Could be. They didn't want him exposing these dirty secrets. Yeah. He was too much. He knew too much. Could you give a quick like three minutes summary of the guitar shop like what that, what that was revealing?

Because I'm trying to remember what it was because it involved like a single point of failure guy who was out. here for this failure as well. So I don't know how to reconcile the two things. And of course, we have no idea. We have no idea if either are telling us the truth now, right? Because this was such a bad RCA, I have no idea if it's correct or not. But what was the password? Yes, the password was Wishbone 12, I think.

There you go. I always tried to kill me. That's my recollection anyway. So, yeah, that story was that there was a thing that was designed to copy. be configurations and that thing had kind of gone rogue and could not be stopped.

Like it was just copying configurations totally incorrectly and it needed to be like fixed or repaired or something and we don't have any more information because it was an overheard conversation. Right. And so does that comport with this? Well, a little bit

because those enactors do sound like the kind of thing that would be running a configuration copy. But on the other hand, it's not really a configuration for machines. Like a DNS entry is a DNS entry. It's not really a configuration. So I would say the two stories don't line up that well. And so that's another reason why I was kind of hoping

that this RCA was a little bit more believable because I wanted to know for sure that the story was false. And I still don't really know based on how bad this RCA. What if the tool that the guy wrote to copy the configs is just literally the enactor? Like they just productionized it. and they haven't changed it in seven years. That was kind of my, like, connecting the dots there,

was he's like, guys, I wrote that as a way for me to test stuff in my local environment. And you just decided to make three enactors and put them next to each other in prod. I don't, how did this happen? I do. I have a ton of questions. Alternatively, is it the rollback? Because that's the one that did the copying of like, hey, here's the previous one. Right?

And so I'm going to copy the previous one. And then it gets like this null issue going on. And it just like the script never encountered a null. It just goes rogue and starts writing over and over and over and over again to where you can't. You can't do anything. I don't know. All I know is that like as far as I can tell from their explanation going only on what they were providing, I still just don't think the race condition is even relevant.

Because again, a literally an accidental update to the Route 53 endpoint would have taken down all three enactors immediately. Because according to them, all that's required to stop them is if the end point points at an unresolvable name. That's all you need. And so if that's really true, literally an operator typo could have taken all this down. No race condition necessary, right? And so, again, the RCA just does not do a good job convincing me that you've talked about

what the real bug was, because I can think of so many ways that you could have triggered this exact same thing that don't involve this race condition that you spent the entire RCA telling me was the bug, but it don't think it is. Well, we'd like to extend a formal invitation to Jeff Bezos. If you want to come in here and explain yourself? I believe it's Andy Jassy. Andy Jassy is the man you're looking for him. Or wait.

I'm going straight to the top. Jassy is a previous as well. He's not AIDAX anymore. He's head of Amazon proper. Yeah, that's what we want. I mean, we had head of, we want the president. Bezos was previous head. Now he's just chairman.

So he's no longer day to day. You know, he's gallivanting. So he wants someone more. Let's go both. I want a real chairman analysis, though. I want the. The Sunday chairman analysis. The real chairman analysis.

The armchair. The arm chairman. That's the one I'm looking for. The arm chair. Dude, why is an arm chairman a phrase? Like, I'm arm chairman of the board. I'm pretty sure that's the guy that does the chips, right? The arm chairman.

The armchairman. I don't get the joke. I'm sure it's true. Armed. Armed risk machine. Yeah, ARM. Oh, my God. Not potato chips.

I know trash is on the pod, but. Yeah, yeah. Oh, my God. Dude, wait a minute, wait a minute, wait a minute, wait a minute, wait a minute. This is our sponsor opportunity. Potato chip companies make a line of potato chips chips where the bags are labeled by, actual chip. So it's like this is a 9950 X3D chip.

Like the bag just has that on it. And it's specifically patina chips for developers. I have been trying to convince Sun chips to sponsor us forever. Because think about it. Sun. Sun microchips. Yes. That's what I've been saying.

Oh, I was not going on that path of mine. It's a match. What we're in trash. She's like, garden salsa. Yeah. The garden. The programmer's garden. Yes, microchips.

Some microchips. Yeah, yes, yes, yes, yes, yes, yes. That's the one that we need. If we can get a Sunchips sponsorship, that's the perfect one. It's a match made in heaven. It really is. Okay. But Casey, I've been tweeting at them for like five years.

They don't care. They're just like, who is this guy? And why does he keep tweeting at us? I just see if I can find the sun chips. My oldest sun chips tweet here. This is May 28, 2001. Okay, we're really digging back. 2020.

Dude, you are really working hard to bab this chips monster. Well, I was writing a lot of Lua at the time, so I thought that it would go nicely together. Lua, like, moon, sun, right? It's like moon does the sun. We're echoing back to each other here. So that was a classic hashtag computer chips. There you go. That's a new terminal line.

They were not. Oh, they were. That was part of the meme. They did not reply. They've never replied to me. They've never interacted with me. So disappointing. I know.

So we're, I'm still waiting. Still waiting. So disappointing. I can't believe it. You like sun chips though, right? A harvest cheddar specifically. Harvest cheddar?

What you mean a harvest chet? Are you kidding me? Oh my gosh. Hold on. Hold on. on. I am. I am a fan. When we're at the tower, all those garden sauces, threw them to the sign.

Disgusting. Oh, wow. Oh, wow. There's shaking everything. Nope. It does not. You can't be a part of this sponsorship. I thought we had the same taste in snacks.

Well, I ate all the harvest cheddars over in that tower. That's actually great because then we make a good team. See, that's what I'm saying. You like, yin and yank. I actually enjoy French onion also. I don't think what is there a French I don't think green pack green packs French on blue packs a ridge which I'm not a It is original was not that good Adam our sunship's vegan I have never had a sun chip in my life, but I don't think Are they even my house right now?

Leave his house now They may not have many Ozarks it may not be a vegan They don't have them in the Ozarks Oh my goodness. Okay, well I think this is probably actually a reasonable time to end at this point I can't believe if Casey wrote backwards that whole time like that. That's what I was going to ask. Yeah. You're not riding backwards.

Yeah. But you get used to it. Adam, it goes like this. His right hand, he writes forwards. He trained his left hand right backwards. That is not how you're not doing it. Yes.

So it's like right hand he does it the right way. He cannot write his right hand backwards. But you can watch. He does his left hand. He does it backwards. Are you guys trolling? Now. You just saw it happen.

Casey's nodding yes. I spent most of the time you're talking. You'd use the left hand. That's how you do it. You just. Because if you aren't, so if you're ambidextrous, this would be hard. But if you're used to writing with your right hand, then when you train your left hand to write backwards, that just seems normal. Right?

Because you don't have, there's not another thing. Yeah. But you are trolling. You don't actually write backwards? Yeah, we're totally. Do you think he would actually write back? Actually, I can keep with those things. So many people ask me that.

It's weird. I was watching so closely your hand in it. I couldn't figure out if it could be. But it was like, no, I think that's some right way. Like, I'm trying to, like, stand where you would. standing and like I could not get that question out of it. Adam starts writing on his fake. On his fake way more.

For writing backwards for six months and come out of our I don't know how you do it. It just seems magical to me. It's just a mirror camera, right? Thank you, trash. Yes. Well, no, no, but how do I see through it? Yeah.

It's glass. What do you mean? It's glass. through the glass, right? Because if I'm writing on one side, you see it backwards. So if you just mirror it, does it actually come out correctly? I haven't done the math to know. Ron, glad to podcast. This is a wrap. Wait, no, wrong, false. What did we say at the start of this thing? If you're actually

want to know, you ask questions. You don't just freaking pretend. And Prime is doing the exact thing he should be doing, which is saying, I'm not sure, does that actually work? Yeah, because my only question is, how many times do you reverse it, right? Because you're going to, if I draw like this. Right? If I draw like this, but then it goes through, if you're looking at it from the other perspective, for you, yours, it's going to be flipped, right? Uh, there you go, flip this thing.

It's going to look like, it's going to, oh, that's vertical. The glass was clearly between the camera and him. Yeah, the glass is clear. So that's backwards looking through it. and so then you re-flip it again does that work out? Okay, maybe. Yeah, sure. All right, I buy it. Yeah.

But if you were, if you were, like, you got that right now, you wrote the letter D, right? Of course, unfortunately, that could be letter B. So you probably want to write a letter that's only those one way around. Because it's different letters. Right? Right. But like, you know, there we go.

So you write the letter Z, right? You're writing it this way. You're looking at it. So think of yourself as standing and you're looking at the board. You write the letter Z, right? Now, if you were to walk around, behind your screen and look back at it, you'd see Z flipped.

Right? So all you have to do is just flip it one more time and it's correct. Can I show you guys one thing that gets people always really riled up? Sure, you'll see how I draw my S's? Yeah. What?

Bottom up S? Are you kidding me right now? But look how beautiful they are. I mean, that's not a thing. No. It's like a serpent. No. You can tell when it's done that it's wrong. Your Z?

Do you actually write your Z like that, too? I write my Z that way. Yeah, sorry, Matt. Sorry, Trash. We did math. So we cross our sevens. Yeah. Okay, I do my, I do my sevens like that.

I don't do my... It's called Zed, by the way, but sure. Whatever. Oh, dang. A editor? Oh, got it, got it. That's fine. Also, can I show you guys something?

Mm-hmm. I have a very magical ability. Are you ready for this one? Yes. I drew an ampersand first try without fumbling it it's considered pretty impossible by many people's standards when they first draw an appra sand it's very difficult

also can you zoom in on the line looks pretty good from where I'm at almost a perfect line right there too that's computer enhanced no you can see that I goofed it right here and I goofed it right here you can see the two spots that I go Enhanced that's why you're gonna go to computer

Enhance the way you write your ass that has to slow you down so much I just decide to write really nice instead. I don't care I write. Why do you explain that to me? I don't know. It's just to me. This is not the same amount of writing?

Is that how you hold your pencil? Like this? Can you do that? No, no, no. How do you write? No, that's like this. Okay. Well, that's, there's your problem.

You hold it like a weirdo. Dude, we were just talking about drumline. What, do you hold it like this? Yeah, like a proper, yeah. Neither of those. That's not how I hold it either. There's too many fingers involved. Right?

Right on top. Boom. That's how you hold it. Can you guys do this? Can you guys do this? I can do that. Okay. Oh, we see that?

Please make that be the opening. Can you guys do this? And it goes down. And then you cut to the start of the podcast. Dresgey. That's pretty cool, actually. I was at my friend's house. We were all sledding together with us and our

kids. His, uh, his son goes down the hill and he goes, and this is how a 13 year old sleds. He jumps down the hill, immediately gets the sled caught down, fall face first into the snow and just biffs into the snow. He's not crying. So now every time I see him at church, I'm like, and this is how a 31 year old runs. That is so mean. He thinks he's hilarious. He cracks up every time. All right, I want to do one more little trick for you. Okay. Are you ready? Okay. All right.

You got to hold a pencil like this. All right. And then you without taking your hands off the pencil, okay? We did this one. Yeah, you got to end like that. Oh, I knew they answered at one point. Okay. You just don't take your hands off of it.

I can't. Dude, that. Do it again. Do it again. That's the one that I'll do. Okay, it's very simple. Okay. So you put it right here, right?

And then you got to turn it. And then you just put your hands the other way. It's like three pixels on my side. Yeah, it's great bad. The video quality. It just doesn't even look like it is a pencil. Whatever. Can you do this?

Boom. But you can't do that. Can you spell blood with your hand? Okay. Oh, trash. Bro, you're going to get this podcast. All right, trash. All right, trash.

Can you Fortnite dance? Probably not. Yeah, okay. No, please don't dress. Trust. You don't want to do this. You don't want to do this, Traster. I don't want to do it.

I don't want to do it. These are just baiting me. No, no, no, no, no, no. I'm good. If you hit an orange justice on this podcast, you will literally never stop orange justicing from here until the universe ends. No, we're good. End it.

All right. Thank you, everybody for watching. I'm terribly sorry about this. Not 50 Tech Podcast. Yeah, I showed it on. We got out of this one. We didn't even have to show a nipples. That was a good episode.

Can you believe I told him to go to Spotify? for this.