Coding Blocks - Designing Data-Intensive Applications – Transactions

Episode Date: January 23, 2023

We decided to knock the dust off our copies of Designing Data-Intensive Applications to learn about transactions while Michael is full of solutions, Allen isn't deterred by Cheater McCheaterton, and Joe realizes wurds iz hard.

You're listening to Coding Blocks. I said a er in the beginning of that. Er, you're listening. That's how it came out. I don't know. Whatever. I'll figure it out as we go along. Well, can I get a, is it too late? Can I get a pull request to fix this? All right. So episode 202, that's what you're listening to. Well, that's the episode. The show is Coding Blocks. You probably figured that out out i might have said that already we'll get there don't give me that look i see that just wait for it wait for it patience no this is this is the success this is what success looks like 10 10 years of this you know at this point i'm not going to apologize for it like yeah you know what you're in for i'm with you there.

All right. So, uh, subscribe to us on Spotify, iTunes, Stitcher, wherever you like to find your podcasts. Uh,

really hope we're there by now. Uh, Hey, and we fixed some Stitcher issues. So, you know, Hey, uh,

we're probably there. Probably again. It's so, it's so funny. You say that. Cause I was actually going to mention that. Hey, we have one feed on Stitcher now and it has all the episodes.

We're no longer in triplicate. So, you know, it took us 10 years to fix that. Everything from now on is going to be like it took us 10 years. All right. So visit us at codingblocks.net where you can find our show notes, examples, discussion and more. Yep. Send your feedback, questions, and rants to comments at codingblocks.net or follow us on

Twitter at CodingBlocks. And if you notice any other mistakes, you can hit up our JIRA at JIRA.CB. I'm kidding. That was CB number one that we closed, actually. But anyway, codingblocks.net. You can find all our actual links to socials and whatever else at the top of the

page. And with that, I'm Joe Zach. I'm Michael Outlaw. And I'm Alan Underwood. All right. And so today, as was one of our New Year's resolutions to finish a book, we're continuing down that path with designing data-intensive applications. And we are talking about transactions. But first. Yes, first. Here we go. Here we were talking about transactions. But first,

yes, first, here we go. Here we go. Yes. There you go. Yeah. I was going to say,

I'm not even gonna wait for you guys to be like, you do it outlaw. Cause I was like, I know what's coming. All right. So, uh, from iTunes,

uh, obviously we're, we're, you know, given thanks to everybody that left us a review. So from iTunes, we have, uh,

J law one 15 cutting corner barbershop, uh, um, Mergy maybe, uh, and Jack, you, Jack,

you never, ever, ever. Okay. I didn't think you were going to struggle with these, man. I'm sorry. Well, that one, that, well, the Mergy one threw me because I wasn't sure, like, you know, how the E's would be pronounced, you know. But I definitely didn't think about pronouncing Unver as, you know, as Unver.

Like that. I don't know. Whatever. You know what? I'm renaming him. He's now Jack Unver would. So welcome to the club. And,

uh, from audible, Mr. This is, I gotta be a formal here, Mr. William M Davies. So thank you to all of you.

Uh, and so, you know, speaking of, uh, mergies, did, did you happen to catch his, Mr. William M. Davies. So thank you to all of you. And so, you know, speaking of Mergees, did you happen to catch his? His was like a super long review, but it was also a super good one too, where he was talking about like the, you know, he made a complete 180 in his career choice. It was a really good review if you haven't read it. I'm going to read it right now.

So at any rate, just wanted to say thank you to everyone. In his case specifically, if I remember it correctly off the top of my head, he had studied mathematics. He'd gotten a mathematics degree and was having trouble in the real job world and ended up pivoting to, uh, software development and, uh, listening to us in the, in free time and, you know, recommendations on books and whatnot helped him actually land his, uh, first job in the software development world. So, you know, it's great stories like that, that, uh, you know, are always like very nice to read. You know what I mean? Yeah. Yeah, totally. That's great. You know what I mean? Yeah.

Yeah, totally. That's great. Hey, I got a tip for you, Mr. William M. Davies. You left us a review on Audible, which is anyone can do who uses Audible. I just noticed the Audible released a program called Audible Rewards where you can earn rewards for doing things,

mostly spending money with them, of course. But also it's got like little challenges. Like if you, uh, listen 20 minutes a day for the next five days between January 7th and 31st, then you get to, uh, pick one of these titles or you get a $5 reward or just like little stuff like

that. So it's kind of cool if you haven't been to the actual, uh, audible website for a while and you're a member, uh, and you like audio content, I hope you do. Then,

uh, that's something to check out. Pretty cool. I just saw it today. That's excellent. Also, Game Jam is probably over by the time you're listening to this, but you should check out all the

games. You can go there, play them, see what other people do. And if you didn't do it this year because you were afraid you didn't have time or you're not really experienced with the kind of tools and you were worried about something, then you should not worry about that. And, uh,

next year you should just sign up. Cause even if you just got a couple hours, start with a tutorial, you know, and then jump off, personalize it, stick your face on, you know,

instead of the character or whatever that you throw darts at, um, or, you know, somebody else's face. I don't know. Uh, it's your game.

Do what you want, but you should try it and don't worry about not having enough time or, you know, don't worry about anything. Just go have fun. because it's fun i recommend using joe's face like set up your dartboard get a good printout get a picture of his shins yeah there's a bunch of them on myspace still actually if you go awesome oh man i'm still on friendster okay hey and lastly um we actually have had a couple people visit this link recently and send us some self-addressed stamp envelopes. So if you'd like some swag, some stickers and whatnot, make sure you head over to codingblocks.net slash swag.

And you can see where to mail us a self-addressed stamp envelope, and we will get some stuff into the mail for you. So check that out. That's how the Internet gets into the real world. Right? Yes, the virtual into the reality. Yeah. Alright. OneSpirit doesn't exist anymore.

Oh, really? Yeah, it's just down. They didn't even forward it anywhere. Somebody had to buy it. Come on. You're born or it come on you're sitting on it waiting for the renewal revolution all right transactions is what we're talking about so um so we all you know read the chapter of course um were there any like big kind of change in minds or like any perspective changes you got from this first kind of chapter or did it line up with your preconceptions about what transactions were in databases and systems in general it aligned yeah i would say that with me too but i like the clarity and articulation i i liked some of their takeaways that we'll get into with like acid and that kind of stuff. I really enjoyed their take on, on some of it.

But yeah, I think for the most part, it all, it all lined up pretty good for me. Yeah. Same with you, Allah. Yeah. Yeah. I mean, just to make sure though that I didn't mess up cause we're focusing on the acid portion. I thought, yeah, we are. Okay. It sounded like you guys were talking about like the rest of it, you know, we're focusing on the acid portion i thought yeah we are okay yep it's not mainly it's not like you guys were talking about like the rest of it you know like later and i'm like um but

this part was just kind of like uh rehash or not rehash but uh solidifying stuff we've talked about before just going into more detail yeah i think it definitely fills in some, the details on some of this stuff, but also for people that aren't familiar with transactions, especially, I mean, this was mainly focused on like, um, relational databases and, and how those have been doing these things for years. It's, this is a really good episode to listen to if you've never actually done a transaction or don't even know why you would do one. so yeah so that was my reason for asking is like coming up from like a relational world it's kind of like yeah transactions like it's obvious to me why you need them and want them and you know that's great and i thought there's like a whole generation of like web

developers or whoever that have like grown up with firebase or uh you know document dbs or not what's all what's S3? No SQL. Yeah, what's Dynamo? That's what I was looking for. Oh, Dynamo, yeah. We've grown up on Dynamo primarily and maybe transactions and those, we'll get into it, but they're not as prominent

or they're not as powerful or there's some kind of trade-offs there. And so it was just kind of interesting to me to think that there's people that this wouldn't be reviewed for. So I don't know. Hopefully, we'll get something out of the episode though. Uh,

dear listener. Yeah. And so with that, they opened up this chapter, um, which does anybody remember which chapter it was? I think chapter seven. All right.

So Joe recurs and Joe a long time ago, um, mentioned that we should at least like say that stuff. So if you're trying to follow along, you know, um, this, this would be a good place to pick up here. So to start this off, one of the creators of Google Spanner, which is one of their databases that can scale like crazily, when they were talking about creating this thing, they said that the general idea is it's better to

have transactions as a feature that's available, even if it has performance issues and let the developers decide if the performance trade-off is worth it instead of not having transactions at all and putting all that complexity on the developer. And I thought that was really interesting because like we said just a second ago, we're mainly talking about relational databases here. I think Google Cloud Spanner is sort of an evolution or an extension of sort of, I don't know, a scalable database to where transactions are more difficult, right? Like it's a distributed type database system. And relational too.

And relational. Yeah. And so doing a transaction, as we'll talk about as we get into this a little bit further, is way more complicated in that world than it is when you're dealing with a single system, right? And so it was interesting that they added that there saying, hey, we know there's going to be performance hits,

but it makes sense to add this so that people could take advantage of it if they need to. So. There's, there was also like, um, later parts in this chapter two where the author calls out like not necessarily related to this quote, but kind of stating this quote in another way where, you know, you might decide that, um, depending on your needs that, you know, you might pick another system because of that transaction cost, the performance cost, right?

Like, you know, maybe, maybe, uh, you know, parallelization matters more to you, for example, or, you know, whatever the case might be, you know, um, right so one other quick tidbit i just can't resist uh so spanner when it first came out i remember kind of thinking like this was like uh when i was studying for the gcp um certification uh for google cloud and i remember kind of reading about it and thinking like why why did we need another database and i ended up reading about it and like realizing that it was all about scale plus relational and it offered transactions but at the time i i didn't know any other databases that were like distributed to that kind of level and relational and offered transactions but i was just looking uh i've heard the term new sequel a few times and i still can't really you know

describe like i don't know enough about it but um i was just on wikipedia and noticed that it's listed as one of the few kind of, this is like new generation of what they're called new SQL, uh, terms and, you know, uh, databases. And who knows if that term is going to stick around, you know, it's a, it's awfully cute. So I have a feeling it will, but, uh, other databases here, you might've heard of that are kind of like considered in this kind of new generation or like cockroach DB. Um, they've got couch page based listed here, which I'm kind of surprised about. A bunch of other ones I've not heard of.

YugabyteDB, which I've seen on Hacker News a few times. So it's just kind of interesting. I just wanted to call out that Google Spanner in particular is a very interesting database. And I think it was probably like brand spanking new when this version of the book came out. And I think I heard there's a new version of this book coming out sometime, I think this year, and I'd be interested to see what they've added. I'm going to look that up right now. We're going to have to hurry up and finish it then. Hey, so seeing as how I'd never heard of this and you threw it out there,

this Wikipedia page, NewSQL is a class of relational database management systems that seek to provide the scalability of NoSQL systems, but for online transaction processing. So OLTP workloads while maintaining asset guarantees of a traditional database system. So that's pretty crazy. There's like you said, there's,

you know, a dozen or more that are listed here that are all looking to fill that niche of scalable, but yet transactional. Yeah. Pretty cool. Pretty cool. Should we mention too that this is my last version for at least five minutes.

Should we mention that if you leave a comment on this, we will send you a book? Yeah, we should do that. Well, we might send you a book. You'll be entered for a chance to get a book. You'll get a thousand comments on this. Let me check with the lawyers real quick. Hold on.

Does that take? Does that work? Can we roll with that? Okay. They gave a thumbs up. All right, cool. Yeah, we got plywood on the windows next. So we're shattering up.

Oh, man. All right. So there are a number of things that can go wrong during a database interaction, right? Never. No. I've never seen it. Have you seen my code? It doesn't.

No, it's perfect. That's not going to happen. That's right. So the database software or the underlying hardware could fail during a write. We've actually seen hardware failures on database systems, and that's never fun. I mean, you really hope it's not the database software? Like, you really, you really, I'm not saying that it can't happen.

I'm not saying that they don't have their bugs, but, like, of those two, I'm going to venture to say, like, more often than not, the average person, we're probably going to see the hardware failure first. Would you agree with that yeah i i gotta say i think four databases i think you're right they've mostly been pretty rock solid over the years right yeah they're hardened pretty good unless you run out of disk space and then all kinds of weird things happen right that's but is that but that's hardware though would you count that as a software thing you ran out of space your hardware ran out of space yeah it should have it should have done something because it was approaching the limit

right i don't know i don't know who to blame that on but if your database runs out of disk space i kind of blame your hard drive for not growing. Time to be a big boy and grow up. That's right. So the other thing, the application that uses the database, it may crash in the middle of a series of operations, right? So your sweet little web endpoint that you created that's doing 20 different things, like it might fail at some point. Looking at you, Jay-Z.

Yeah. I can fail. Network problems. Those arise. I think in the cloud world, we see that more than we used to with on-prem things. But definitely a hard- You're a database local host, and then you don't have that problem. I'm full of all kinds of solutions tonight by the

way oh man somebody's gonna go write the next version of something awesome out there and everything's be running on a single computer um multiple writes to the same records from multiple places aka erase condition um reads could happen to partially updated data which may not make sense, right? Like if something's being updated somewhere and then you read part of another record somewhere and they don't match up anymore. And then race conditions between clients can cause odd problems. Yeah. And so reliable systems, you know, the word reliable, you know, it's kind of got a lot of different definitions. But like your general conception of the word reliable being a system that you can trust.

They handle those situations and ensure that they don't cause catastrophic. So I'm so excited about this book. We're getting back to it, y'all. Catastrophe. Catatrophy? Catastrophe failures. Butrophy? Catastrophe failures. But it's a lot of work. And we've talked about that a lot of times.

Like, you know, getting that extra nine in your nines is a ton of work. Like exponential amount of work. And transactions are basically our tool that we've been using for decades. Do you remember what year you said that the white paper kind of describing transactions came out? It was like 40 years ago. I think it was late 70s. I thought it was like 78 or 9, something like that. Am I wrong? No, that's what I recall. It was somewhere in the 70s. Yeah, I'm going to look up a link to that and we'll have that in the show notes. Because the term for acid

came out in the early 80s if i recall but okay uh yeah the term for acid was coined in 83 we talked about this too like uh the bee trees like a lot of the technologies even though like lsm kind of different databases we've talked about in the past a lot of them were uh written based on research done way way way back and really haven't evolved a whole lot since it's really just been a matter of like getting the tools in place and getting the experience kind of building these systems uh that to finally kind of match up with the theory behind it which is crazy to me well some of that i wonder a question i've always questioned like is that a matter of you know hardware wasn't where we needed to be?

Like, um, global networking definitely wasn't, you know, so that, so there were things that you could theorize about, but you didn't, you know, you were maybe ahead of your time. So you, you couldn't like put it into use practically. Yeah. I mean, it's the tools. I think about like, uh, what,

uh, you know, visual studio was like 40 years ago. It probably didn't exist 40 years ago. Like who knows what it was? Um, I think it was a lot more visual back then. When did visual studio was,

okay, now we got to remember back when it first came out, the big appeal was like, you could like drag and drop buttons and stuff. Well, I'm going going guess like i'm thinking maybe it would predate it wasn't like necessary studio but i'm thinking more basic you know like a visual visual basic or something yeah

so let's see 97 is the first version of visual studio. Before that, they were all separate products. So now I got to know when Vim was released. 91. So this stuff predates Vim. I mean, that tells you what tooling was like at the time, right? Yeah, I mean, but to your point, though, it is insane that this stuff has basically been around unchanged for several decades now like it works the same way now on these single system pieces of software these databases as it has been for years and years and i should say too i looked

up vim which is i'm so used to seeing it that way but vi was released in 1976 so vi was in fact around and uh that's probably the tool they were using at the time they were researching this stuff it's crazy we the the m in vim is for improved for improved yep well it's it's i think it's supposed to be like a concatenation of the two words where they share the i or at least that was what i thought yeah that's my understanding as well all right so what is a transaction give it to us okay so it's a way to group uh related reads and or writes into a single operation. So as an example, if let's go back to our canonical e-commerce example, if you were to place an order and you hit that, you know, submit button on that order.

Depending on the system, you might want to log what the order was and including payment information, but you also might want to decrement current inventory as part of that process too. And so the table that has the order and the table that has inventory, those are going to be different things. So you're going to need, we're dealing in a SQL world here. So two different SQL statements that are going to do that thing. But should you be able to successfully write the order, but you are not able to decrement the inventory, you don't want to leave that order in place in this theoretical example. So you would roll back both of them and let the customer redo it. Now, the reality is that's a horrible example because you'd just go ahead and take the order.

Well, bank accounts is a good one, right? It's like, I'm moving money from this bank account and moving it over there. Which of those should happen first? Well, no. They need to both happen or they need to both not happen. We can't have money disappearing over here and then not making it over there or vice versa. Money just appearing over here without coming from anywhere. So that's a great example of somewhere where like it absolutely cannot ever go wrong. You imagine like even say like a Bank of America or something like how many transactions do they process in a single day? All right.

Even in an hour. It's insane. So you can't have even a 0.1 failure you need to have like zero failures oh well as soon as you brought up the banking example i immediately i mean i think it was bank of america like they they there was issues where they had to um there was regulations put in place related to, uh, overdraft charges. You know where I'm going with this? Oh, they're arranging, right?

Yeah. They, they had figured out that if they rearranged the order of the transactions, they could maximize the number of overdraft fees to charge more. So like, you know, in reality, in, in order of events, it might've only been once, but they were like, Hey, we reorder these things we get you like three times yeah order my biggest first there we go yeah isn't that ridiculous yeah so you know transactions matter no that had nothing to do with transactions but whatever it was it came to mind that was that was my tangent so i according i i get at least like what, eight more of those? Because Jay-Z had several.

Something, yeah. Yeah. Yeah. So yeah, the idea here is that a transaction as a whole either completes, and we call that a commit. And it says done, it's committed, everyone's happy. Or it fails, and you can get like an onboard or rollback. And there's a couple different options there. But if the application

fails, the application can decide what to do. Wait, you said that wrong. If the transaction fails, the application can decide. You said if the application fails. Oh, no. I mean, that doesn't happen. Yeah. Thank you. Thank you for recognizing.

Finally, somebody's on board. Yeah, if the transaction fails, then the application can choose what to do. Now, that's interesting that it says it that way, because that that to me speaks of something different. Because typically, if you're doing something in a database, right, like let's say it's a store procedure or something like that. Right. You wrap you wrap the entire thing in a transaction and you don't necessarily tell it what to do right you basically hit abort and it rewinds everything for you yeah to a certain degree but then then then the calling api layer would get some kind of error and that api layer could decide oh okay i'm gonna i'm gonna retry it again or i'm gonna like bubble up an error message to

the user and let the user decide you know hey look, I do want to place that order and I don't know why you won't let me. Yeah. All right. Yeah. Good point. So so to separate that out again, say it, say it one more time. You have a transaction. It's going to rewind everything that was done in those processes. Right. And then your application is going to choose, hey, what do I want to do? Now that I know that this thing didn't succeed, what do I want to do from here? I can either go into a retry loop,

send some feedback, whatever, but they're separate, right? And that's the important thing. The transaction is doing something and then your application is responding to the failure. Now, let's think about this though, because this being the book, Designing Data-Intensive Applications, we can't have these types of conversations without actually thinking about like how we might implement this in a file. Right.

And so immediately, like as I was going through this book, I'm like, you know. And how how how how would you implement that? And so I was kind of thinking like, you know, behind the scenes, there are these, you know, these are all files where this is all being written to, right? But it's being stored in some kind of a fashion and we've covered the different data structures

that it might be using and whatnot and patterns. But, you know, is it, the best that I could come up with, it was like, well, maybe behind, under the covers, it's like opening up that file could come up with it was like well maybe behind under the covers it's like opening up that file with a lock on it so that it is the only thing that could write to the file and then it tries to write the first thing to the file and then if it's writing the second thing to the file then it's still free to like undo its changes to one or more of the files,

you know, however many it has, but it has this exclusive lock on those files. And maybe that's either because it's like at the file system level, it's got the lock or it's just within the application in like, you know, a, what were the, the, to single thread, to serialize the lock, you, you, you would do the object lock in the code. And I can't think of what synchronize or synchronize. I didn't think that was talking about the two phase locking. Yeah. Yeah. Like a double, like a, you're referring to like a double check locking, but yeah. Um, you know, that, that was the best that I could think of was that like, it's, it's locking who can like how many processes could write. So meaning that only one thing could

be writing to that file at one time. But then that kind of boggled my mind because I'm like, wow, like you're like so limiting how many things could write to a given file. Now going backwards, remember like one table could actually be because of segments, right? Uh, and, and depending on the, the data structure that you might be using, like in the case of like a tree,

you know, one of the various trees, uh, that we've talked about, um, you know, you might have one file per each of those, those trees,

which I think that they had referred to as page sizes. So you might say, okay, well, it's fine for other clients to read and write to records that are not between 25 and 50. But the file that has those records, I've got an exclusive lock on that because I'm changing something on one of those rows, like inventory count, for example. Yeah. So that was the best way that I could come up with in my mind, like how you might implement a transactional kind of system at like the lowest level. Yeah, I think you're really close. We need to read further, of course, to get into the details. But I think that was you're really closely i have uh we need to read further of course to get into the details but i think that's that was pretty much where we're

going to get to and there's some problems that are going to we're going to hit like with like well what if i say lock these three files i locked the first one locked the second one and somebody beat me to the third one and meanwhile they're they locked the first file and they try to go to the second and i've got it locked and so now we have to decide like who wins and how do we give up and how do we know how do we tell you know the readers to basically stop and like hands off all these rows at the same time and so you get into like two-phase commits and write ahead logs and a few other kind of fancy features but i think that you kind of hit the nail on the head on the general theory behind it and now think think into like uh like just going like

sql server specific but there's, you know, uh, I'm sure that there's like similar keywords and other, uh, database systems too. But like you could do like a select with unlock or, uh, yeah, no lock, uh, select with, with no lock. Sorry. Um, what did I say? Did I say it was, yeah, you said with unlock, but actually it's never used for a select. It's usually used for an update, right? So an update with no lock. No, no, no. You would do a select from with no lock.

And then that way, you know, you weren't waiting for locks to be released. To be released. So you could do dirty reads. Well, you can also do with a no lock on an update, which means that you could be writing and not locking records, which is also dangerous. I would never do that. Like, can we erase that from our, we'll never talk about that. That's gross. But, but you could definitely do a dirty read. And in that inventory example, like, you know, I'm, I might have one process. It's, it's literally in the process of decrementing the inventory. And you're like, well, I don't need to display like

something that accurate. So just select it with no lock so that I can at least put up some semblance of like where we are without having to wait. Yeah. Yeah. Without having to wait on, on getting that, uh, that lock on that table. Hey, and, and I don't think they mentioned it anywhere in this chapter. So it's probably worth calling out. You do have to be a little bit careful when you're writing some of these transactions because you can put yourself into a state where you deadlock everything, right? And once you deadlock things, nothing happens in the database, right? Like the entire thing basically blows up at some point and throws an error and kills all the connections and all that kind of stuff. So, um, yeah. So a deadlock in the simplest form would basically mean that you have two processes that are each waiting on the other to finish because they're each

waiting on the other to finish. They'll never each finish. So they're in an infinite loop of waiting and that's a deadlock situation. Yep. So, yeah. So in general, like this whole point of a transaction is to make things easier for you as the developer to handle something when your writes or your reads fail, right? It's supposed to make it a lot easier.

You basically wrap a transaction around something and, hey, it should all work. And if it does, I'll commit it. And if it doesn't, I'll roll it back. and you don't actually have to do any extra work you know could you imagine if you didn't have a database that was taking care of the stuff for you and you were an e-commerce website and you go to place an order and now you've got to write all that logic that says like is this table locked is this table locked oh i didn't get the lock okay now we got a deadlock what do we do like you have to figure that out, and then you go to implement the address book. Guess what?

Same thing. Taxes, same thing. Just over and over again, you're re-implementing the same problems. Would it be nice to have all that stuff stored, all that logic kind of happening locally for you? Well, I think that was the point of the Google Spanner quote, right? The comment from, I'm trying to remember, they listed his name, James Colbert. I think that's right. Yeah.

That, you know, the whole point was that, like, it'd be better if the tool at least provided that capability. And, you know, if you don't need it for your particular application, fine, great. But, oh, man, to your point, if you had to like reinvent that logic. Oh, man, it'd be brutal. I mean, we gave a couple of simple examples earlier with the e-commerce thing, right? Like with having an order in an inventory or even moving money from one account to another. I mean, if you it. It doesn't seem all that valuable when we're talking about transactions in such a small scope. If you're talking about like a fairly complex ordering system, typically you're going to have an order table. You're going to have an order details table or order line items, whatever you want to call it.

Then you're going to have an inventory table. You're probably also going to have a payments table. You're going to have a bunch of tables that are all interrelated. And so you might have six or seven tables that you're touching just to place one order, right? And so that's where this transaction comes into play is, you know, the first thing you do is insert into that order table to get you a new order ID, right? Like just thinking out loud, something simple, then you're going to use that order ID in the other tables. And, and as you start inserting into these other tables, you're going to start using data from those tables to use in additional tables, right? And so as you're doing that, you're cascading this data all the way down across seven make sure those are cleared up. Oh, and by the way, there could be two orders trying to be written at the same time.

I need to make sure that I untangle all that stuff, like super duper hard problems to solve. Yeah. Re-update inventory or add it back to the inventory. Right? Yeah. I mean, and there's, I think that's the crazy part is like what you were talking about with locking files and all that, like that's kind of how it has to happen because you can imagine if you didn't lock certain portions of data that you're trying to update, imagine in your rollback,

you're going to have race conditions because somebody else was decrementing inventory. You're trying to reincrement inventory, like, um like which one wins out, like you're going to run into all kinds of weird issues. Right. So, so here's where, here's where my mind was going with that though. Cause like, again, I was taking it from the approach of like, because of this book, right? Like this, this book has made me like all the things that I've taken for granted for years related to, you know, my use of databases. And now I'm like, okay, well, let me think about this, like for real, like how, how am I to implement this? But what happens in that case where like, let's say, I think you called out, like you might have six tables, I think was your example.

So, okay, fine. So you have all, you have a exclusive, you know, you've opened up six different files exclusively, right? Nothing else can open those up because you're opening it up for writing. And now you're saving out those files. And the first five, you're able to save and close successfully. But the sixth one, you can't, right? It's still like even – it's like turtles all the way down. It's like transactions within transactions within transactions, like to know that like every little bit of that. So,

I mean, it really, it really goes to show that like, you know, I was thinking of this, this quote the other day, like we really are standing on the shoulders of giants because like every little problem,

like as a, an industry or a society that we, we solve, that thing becomes no longer a problem for us. It just becomes something we take for granted and we move on to something larger. Right. You know,

so think about that next time you take a look at your node modules folder. Yeah. Oh, geez. That's a giant for real. i did hello world how do i have 12 000 files in here because you didn't want to write your own array yeah so where we go here so yeah talking about like uh do do we give examples of not all applications that need uh transactions no i haven't talked about that, did we give examples of not all applications that need transactions? Nah, we haven't talked about that yet.

Did you have some? No, I marked it. Oh. I would think that things like stock tickers, right? Like that's one that I don't think you actually need transactions for. Tell that to the SEC. So in all seriousness, though, do you think that we're actually storing data for every price change of everything that happened every day from now? Like, just imagine how long these stock tickers have been online. There's no way they're they're tracing these movements every day and retaining it for years on end.

They can't be. Maybe they are. I've wondered that several times. I don't know. I mean, like I, I kinda, that specific example,

um, I don't know. There was a, there was a, there's an interesting documentary, uh, that I just finished watching about the Madoff scandal. And like part of the way that they were found was there was a

mathematician at a competing firm that was asked to buy his boss, like, Hey, look, we got to be able to compete with, with these guys. I don't know what they're doing, but here's, here's their returns. You figure out how we can make a product that competes with that. And the mathematician, like he's, he's looking, he's like, this is impossible, right? And part of his defense of we can't possibly do this is he tracks down, here we know exactly how many transactions of this specific type happen on any given day in Wall Street. And in order for their fund to be as successful as it is, you need a hundred times that the, what actually happens.

So I say all that example because I don't know, maybe they do have that kind of movement. They, they, they, it's big money. So you kind of want to hope that there's some documentation of it, some record of it. So I don't know,

maybe that's not the best example is where I'm going. So I can think of one that probably doesn't matter at all. Logs, right? Like if you're writing logs, you don't care about transactions on that stuff for the most part. Um, and,

and there's tons of things writing logs out nowadays. So you're not updating anything. It's right. Only stuff. You don't care. What about a poll? Like if you're voting on stuff and like maybe stuff comes in out of order, maybe you lose a vote or two. But if you've got millions of votes, who cares?

Yeah, possibly. Yeah, possibly. Moving along. Moving along. I don't know. I don't really touch on a sensitive subject there. Wow, JC. What a horrible example.

Yeah. That one was worse. What is wrong with the two of you? Okay. What's wrong with logs? Logs is good. Maybe. That wasn't your first one.

No, it wasn't. I was thinking that maybe a better example, and you may disagree, would be like a thumbs up kind of like like button on you know whatever on some platform right like you know but but that's a vote right but at the same time i'm gonna give my concession speech hold on oh god that's so amazing. I give. No, but that's definitely a good one. Or like a rating on a movie or something like that, right? Yeah. I get what you're saying.

Like, it's nothing like mission critical. It doesn't matter except for like our, you know, the chemical in our brain. What's that? I can't remember the name of it. Dopamine. The dopamine hit, you know, when we're like, oh, but it but it was five what happened i mean reddit is like that right like reddit sometimes you can post or you can post a submission on reddit and depending on uh you know what random server you might get uh be hitting on you'll see a different uh upvote count have you ever noticed that yeah if you if you refresh the page even it will change counts on you oh that's that's an

upvote so i don't want to confuse it with everything else oh you know what that reminds me of something all right So this is my tangent so far. Number one, I've found a community that is way harsher than even developers, which is crazy, right? Because if you say something wrong in a post onlineches and everything. You want to take a guess at the other, the other community that just is vile. They're just so mean. I'm so afraid of what you're going to say. I was going to say college sports, but I don't know. Nah,

I mean that, that's always been that way in sports. Oh no. Audio, audiophile people. Like, just straight up dirty to each other. It is like a mean industry. Well, as you were talking about Reddit in that regard,

like, I was literally in my mind recalling, like, audiophile conversations on Reddit. So now you, like, mix the two worlds, right? And it's, it's the worst. Yeah. It's, it's truly abysmal type stuff. Like it's, it's not even fun to read most of the time. So anyways, all right. So, so back to this thing. So I think your examples were great of, of types of applications where you just don't care, right? Like it doesn't matter. You don't need a transaction. You don't need to roll back anything there. It's not going to hurt anything. How do you know if you need a transaction there? Uh, I think you'd have to ask yourself,

like what happens in your application if you didn't, uh, if that, if that thing didn't happen, what's the downside? So going back to Jay-Z's example of the bank transfer, right? Like if you were trying to transfer money from your bank account to pay a bill and your bank screwed up and that didn't happen, right? Then you would, you could be in trouble like with whatever that bill was in that case. But also on the flip side, if they only did it one way, so let's say they, they failed in deducting the money from your account. So, you know, you win bank error in your favor, but, uh, they did pay out the money to the bill collector, then it's costing them. It's literally like that error is literally costing them. So you got to ask yourself, like, what's the, what does it mean if I don't get this thing?

Yeah. So, so like, what is the, the guarantee? How, how is there a safety net around this thing? Right. So that's, that's one. Does there need to be one? And, and that's a good one. Here's one thing to consider too, is just your number of objects. If you're only updating one entity, like one row in a table,

it's kind of less important than you have transactions. Assuming your database is smart about like having a Thomas city. So it's able to just update that row. If you have two objects that you're updating, like two tables or maybe two different databases a relational database and a no sql database or this system that system then your need for transactions becomes much higher even if it's a trivial or not trivial but like a not very important use case your chances of things going sideways just by having more than one object being updated at a time is much greater.

Yep. The other thing to consider though, and they touched on it in the Google Spanner thing is what's the cost? What's the trade-off of using a transaction, right? I can tell you in just in things that I used to do way back in the past where we'd update millions of records at a time, you put that in a transaction and it seems like a great idea because you want all that stuff to succeed. But if it does fail for some reason, your rollback can take three to four times as long to do. And now you've put your database in an unusable state for, you know, whatever the foreseeable future is. So you do have to consider like, what is my use case and can I afford the, the cost of running a transaction on this or do I need to figure out another way to do it?

Oh, and, and I think to wrap this one up, we said that, you know, this chapter, they're basically focusing on single node and distributed databases, things that sort of have transactions built into them. Although, you know, Joe Zach brought up the the new SQL thing, which I'd never heard of, which is really cool. But when we get into another episode, we'll be talking specifically about distributed databases and how things would happen in that world. My guess is a lot of people read this book and then they're like, all right, I'm going to make a new SQL database. I got it. Wait, are you saying that as new SQL as in the new term that we've learned tonight or new space SQL? No, no the the new sequel new term okay that's gonna that that this is why that term isn't going to take off no it's not gonna work well yeah it's

dead to me already it's dead is it you know i assumed it was new sequel but maybe it's new sequel new sequel what about out there what about next sequel that sounds that we should just coin that one do we have to be clever can we just admit that we're done with all the cleverness that happens on all these terms and instead just like put a product out there and be like yeah okay

yeah call it product Z yeah I like that I mean we just talked about Vim though I mean that one kind of stuck but I mean but but then there's you don't refer to them as like a classification of them editors uh you're right yeah all right see what i did there see this is why so the vim was a great example just make your product and be done with it. Stop trying to be cute with it. One and done. All right.

Somebody's going to come out. Michael, there is a term for the Vim editors and they're called Linux editors. Michael, I don't sound like that. Stop it. All right. So I think Jay-Z, you've been voted to do this next part. No. Who voted next part. No. Who voted for that?

No. More voting? What? Really? What? All right. Get ready. Why? Get ready.

Why? Was this a survey that I missed? How did this happen? This is what happens when the ballots aren't properly counted. Oh, God. A transaction didn't complete. I'll show you how it's done so hey tell you what uh we do a podcast here and you know what we really need reviews we need your review need your five star review really helps us out uh it helps people find the show and uh makes us feel really good and really uh it's a lot of fun doesn't even

have to be a long one it could be a short Uh, just make sure to leave those five stars. We try to make this easy for you at having a link on a website that will link you to sit your audible or, uh, you know, Apple or anywhere else that you might listen to on the podcast. And you can find that at HTTP, uh,

S www.commentbox.net slash review. And we'll have those links there. And, one star reviews are also fine so next what it was all going well you know i had i can't i can't not do it now i know i i don't even like it i thought that was not funny anymore like 20 episodes ago i just can't stop doing it now it sucks all right okay you heard what the man said. Yeah.

I was, it, I'm like kind of like the whole time. I seen it. Like, is he feeling okay? Maybe he needs to see a doctor. Like this sounds pretty normal.

Yeah. He came around at the end. We're good. All right. Yeah. All right. So, uh, well with that,

it's time for my favorite portion of the show. Survey says, all right. So, um, what episode my favorite portion of the show. Survey says, all right. So, um, what episode is this? Two or two. So Jay-Z,

you are first. I have a horrible track record with this, by the way. Oh yeah. Jay-Z, I think totally crushed you last time. Yes. All right.

So, uh, I'm trying to get my spreadsheets in order so I can keep track of the scores. All right, so name a place you might not get cell phone reception. Elevator. Elevator. All right. Alan?

Parking garage. Parking garage. Parking garage. Wow. Good answer. It does. That is a good answer. So first of all, I'm going to take the easy one out,

which was, man, I really, I don't know about that one either. I'm kind of confused on how to, how to answer both of these. So I think what I'm going to do is I'm going to give, I'm going to give you each a win here because elevator isn't technically on there but work slash office building is and i'm going

to count that because i think that i think that makes sense that's fair right i don't know i mean i was pretty specific about elevator i feel like and i feel like elevator should be on there and i feel like everyone who didn't say that is wrong so i'm willing to take a loss there okay you heard himself out of some points hey i'll take some help that would you know fairness is my middle name it would have been 13 but you know hey what wait 13 13 points i mean okay yeah 13 yeah same thing you you want the 13 points? Yeah, for sure. Okay. I thought you might. I thought you might. All right. And then parking garage isn't technically in here either,

but I'm kind of torn because like immediately when you said it, I was thinking of the type of parking garages that I've used in the past where they were in the basement of the building yeah like concrete and toilet and and and basement is on the list so but then i started thinking about like wait they're they're like literally uh you know probably the majority of parking decks are, you know, a building that you just drive through and so they're all up in the air. Yeah.

Yeah. I'll take a loss. I'll take a loss on this. That's fine. I'm winning. All right. Because that would have been 24 points. Where do they find these people? Where do these people live that don't have elevators but do have basements but not parking garages?

I don't know elevators but do have basements but not parking garages i don't know but they're killing me they're killing me slowly here all right i'm pretty good actually so alan you go first for this one name a type of vehicle you really wouldn't want to hit while driving a dump truck a dump truck whoa all right you really wouldn't want to hit while driving. A dump truck. A dump truck. Whoa. All right. I'm thinking of a sports car.

Maybe I should be more specific and say, I don't know, Lamborghini. Okay. So, obviously, Alan is thinking of Back to the Future and says dump truck because he doesn't want all of the manure falling in his car. That's right. Dump truck is not on the list. Oh, man. Who are these people? So you sure you don't want those other 24 points?

All right. And Lamborghini for Joe, not on the list. What about a semi? This is really going to irritate me because that was my other. Oh, you're really going to hate yourself when you hear these answers. Number one, police car, 44 points. Okay. Number two, fire truck, 16. Number three, ambulance for 15.

Then train for 10. A hearse for six. Now we get specific with Hummer for two points. And limo with two points. And I forgot. Whatever. I just realized that we didn't go back and call the other ones

out. So yeah, we can do that real quick. But yeah. Name a place where you might not get cell phone reception. Wilderness slash mountains was number one for 26 points. Basement number two, 24. Tunnel, 16 points. Oh, that would have been good. Yeah. Boat slash ocean, 14.

Work slash office building, 13. School, three. And court, two. Courts? What? ocean 14 work slash office building 13 school three and court two courts what yeah they'll call you in contempt you do that yeah so uh maybe it's a place where you don't want to get a cell phone reception all right so uh joe you have a commanding lead here with 13 points. Kind of. So pick a number between one and three, and I'll let that be the next question. One. One. All right.

So you're going to love this question. I don't know that I am. Dear boss, if you're listening now, it'd be a time to not at, at your job out of the 60 minutes in each hour. How many do you spend doing actual work? Oh, wait,

is this, are we asking me? Are we asking what people say? Technically, we're supposed to say what the people would say okay and now we've established already these aren't real people these are this is like chrissy tegan you know uh chrissy judge what's the name of the show you ever watch that it's amazing uh no idea chrissy's court is what it is uh anyway it's amazing uh that's but those are the people that they're asking the people in there to be judged um anyway um five minutes i have no idea what you're talking about there's no way no way all right so five minutes is what he said i want to say 45 45 minutes yeah all right here we go man i better get some points on this.

Come on, Christie's Court. Joe says five minutes. Five minutes is not on the list, so that's a zero there. Alan says 45 minutes. Number one answer, 30 points. Alan takes the steal. Yes, sir. Wow.

Yep. 45 had 30 answers. 30 minutes had 23. 50 was third answer at 16. The fourth answer was they spend all 60 minutes of it doing actual work for 12. They work at Amazon. They've got trackers on their brain. Then zero minutes, they work with us.

Six minutes, or no, I'm sorry, six respondents. 15 was next, what was that number, was that seven? No, six, six, 15 was the sixth answer with five respondents in seventh answer was 10 minutes with four respondents. Well, you're trying to say all those numbers and like they make sense and say them in a meaningful way.

That gets complicated. That was almost like us talking about graph, you know, algorithms, nodes and lines and all that. Yeah. Yeah. Here are the questions. Do meetings count do we figure that out i gotta change my answer now oh no no no no that doesn't count uh by the way the questions you left behind joe name something you never leave

home without was one of the questions questions and the other question phone was number one answer you should have picked that one dang it um the next question was how long does a typical new year's resolution last one day so mr pessimistic alan says one day but those were the other uh the other questions so yeah excellent i won yeah hey look at that even with cheater mccheaterton up there picking 13 points up at the beginning yeah yeah don't call out a comeback that's right oh yeah i'm so disappointed in the world right i mean between these answers and yeah i just looked on imdb for the link for because he's uh because he's court it's got three stars out of ten i've never heard of this show what is this he said it's amazing it's amazing it's like judge duty but chrissy tegan so like these people come in they got a

problem or like the roommate uh you know ate too much uh mayonnaise or something and like wants to be reimbursed and the other person doesn't want to pay because they're fantastic. And, I mean, there you go. There's that. Oh, it's a Roku original. How about that? Oh, that's why it only has three people, three reviews. No. Roku's great.

Don't get it twisted. He didn't say three reviews. Team Roku over here. It's got a three star. All right. That's pretty amazing. My people, if you're a fan of christy's court let me know in the comments you might never heard of it all right i'm gonna have to check it out now i've never heard it i i'd never heard of it but i won't be able to check it out her mom is the bailiff

dude it cannot be good jay-z like seriously 264 people have rated this a 3.2 out of 10 all of the viewers that That's abysmal. I mean, we heard the answers on the show, right? Where was dump truck, huh? Where was semi? I agree. I agree. Hummer, give me a break.

Get out of here, Hummer. Well, you're never going to catch the Lamborghini to hit it. But if you do, that was the answer. But it was. But you say that, though. And I know Alan might remember this, maybe. But you remember Tracy Morgan? Remember him from 30 Rock, for example?

Oh, yeah. So he bought a there was a story like this was years ago where he had bought a Bugatti. And as he was pulling out of the dealership, a Honda hit him. Oh, this was in this was in Manhattan. And as he was pulling out of the dealership, a Honda hit him. Oh, this was in, this was in Manhattan. And as he was pulling out, he got hit. Oh, it wasn't his fault too. Like they, they're the article. I remember the reading at the time they were talking about how it wasn't his fault, but you know, yeah, it was like awful.

A Bugatti. That's that's man. I'd vomit. But, but even worse though though you've seen the maintenance on those things outlaw and i love cars oh god an oil change on a bugatti is over 20 grand well you have to replace the tires every year and that's like 50 000 and when i searched that that was years ago so like who knows what it is now those tires were specially made for that car yeah the maintenance on that car is a hundred thousand dollars a year that is oh man that's insane

all right oh man there's so many act there's so many stories about this accident um i i will include a link to it but yeah this was actually newer than i thought it was because this article says uh 2019 i was thinking it was long much longer than that that's that's brutal all right so while he's putting that in there let's talk about the concepts of a transaction so as we spoke about earlier most relational databases i say most they're usually done in relational databases. I don't know if most do. I think that may be a, um, an assumption. I don't know. Some non-relational databases support transactions. Um, I think that new SQL link that we have for Wikipedia has several of them on there. So, um, probably worth checking those out just, just to sort of several of them on there. So,

um, probably worth checking those out just, just to sort of see what's out there. Um, so this is where they actually said in the book, the general idea of a transaction has been around mostly unchanged for 40 years, originally introduced an IBM system R,

which I've never even heard of prior to this statement. And it was the very first relational database. Shocking that I'd never heard of that being that I've been doing already BMSs for so long, but yeah, whatever. Um, so when things started going to no SQL databases, which means basically non-relational databases, they kind of just left transactions out because their whole point was the scale, right? And they didn't want to deal with that problem. So, um, outlaw mentioned dynamo DB earlier. Um, you got things like Cassandra, you got, um, man, I really can't think right now. There's tons of them.

Like other document databases? Yeah, document databases. Mongo? Yeah, Mongo. How did Mongo not come to mind? I don't know, man. My brain completely went blank on that for a second. So yeah, they kind of were just like, hey, we're not going to deal with it.

Elastic, would you count Elastic as a document database no it's it's a search index um well i mean they all are indexes right it's just a matter of like what their use case is for but at the end of the day it's written in a document type format not a relational format it's stored in the whole document it's got the secondary indexes kind of off-site. Yeah, I don't know. I don't think I consider it a database. I consider it a search engine, but I don't know. So what's

interesting is they did call out that in some NoSQL implementations, they sort of changed what a transaction meant so that they could sort of say that they had it, but they had weaker sets of guarantees wrapped around them. And this was something that I really liked that they called out in this book is a popular belief that was kind of put out there was you can't have transactions because

transactions means that you can't scale. Right. And that was that was sort of I even remember when NoSQL implementations were getting as popular as they were. That was sort of the thing that database like traditional relational database people would be like, well, it's not transactions. It's not real. Right. It's not a legit thing if you can't wrap it in a transaction. And that was the the pushback. Right. Oh, you have transactions you can't scale. Oh, you don't have transactions? Well, you're not a real business application database, right? Like you can't do anything real with that.

And the book basically calls both of these hyperbole, right? It's complete nonsense. You can do both in either situation, and there are tradeoffs in both, and that's really what it all boils down to. It's always that. That's kind of been one big takeaway from the book for me, and we've talked about it several times, is just how a lot of times you need multiple databases.

And Elasticsearch is a great example. Fantastic for search. Awesome. If you're running a large e-commerce website, you should have a search engine for sure. It's super fast to find things. It's great. You should not use it as your primary source of record,

whatever you want to call it, for inventory or price because of the lack of transactions and things like that. And so you typically will have some sort of relational database or some sort of other like primary store of data. That's the authority for things like inventory and stuff like that, that needs to be, you know, like for real. And sometimes that means that customers buy stuff that has just sold out and that's something that you got to be, you know, like for real. And sometimes that means that customers buy stuff that has just sold out and that's something that you got to deal with. And you know, you can either try to prevent that at checkout by checking the true inventory or you can just deal with some margin of error.

Okay. So I'm going to, I'm going to play devil's advocate here. Okay. So if elastic search is not considered a database then i ask you why is it on db engines.com oh case closed it's a database it is up there but i think they actually label it as a search engine well they talk about the different types of databases so like there's relational document key value and search engine is a type of database okay yeah that'd be i mean i guess technically the the whole term database is it's a place where you store data right so that's why i was saying like they're all indexes at some point yeah it's a database um not in the traditional sense but but i guess you could say that about key value pair stores and all that as well. So I think technically it's the LSM.

It's, it does the whole like kind of log based, you know, whatever. Um, I forget what it says for log structured merge and like that. But anyway, it's a,

yeah, it's good for some things bad for others. Um, consistency. Also great transactions. Nope. Yep. Hey,

I want, I want to mention something too, because it's come up lately and it's actually, it drives me insane every time I hear it. And I think this book is sort of what changed my mind on this. It's pronounced mommy. Mommy. Mommy.

No, mommy. Just kidding. Go ahead. So what Jay-Z just said is super important. It is okay to duplicate data in different systems if you are serving a purpose that that system solves, right? So, for instance, he just made a perfect example, right? I wrote an article a long time ago on coding blocks, and it was super popular. It's actually one of our more popular articles and it was talking about, because one of the only articles,

that's really funny. It might be. Um, but it was talking about creating a database schema for storing a product catalog in an e-commerce thing. Right. And, and when I went through and I wrote that article,

like I laid out the schema and was talking about like the entity attribute values and all that kind of stuff and, and what it's good at and what it's bad at, right? Like, and it's really good at adding new products to, to the system. But what it's really bad at is if you're trying to select information out, like, Hey, I want to find products that are blue or whatever, because you got to do a ton of joins and all kinds of things. Well, this is a perfect example of, hey, it might make sense to put that into a database so that products on a live site or application that belongs in a search engine. It is not wrong to move that data from one place to another to solve those different problems, right? One is for an admin interface. One is for a search engine.

It's not wrong. If you have needs for analytics, it's not wrong to then move that data into Druid or Pino or something like that. If you need a data lake so that you can do analysis over long trends of data, it is not wrong to throw that data in an S3 bucket and use it that way. I view it as an extension of Uncle Bob's, what did he refer to it as? True duplication and accidental duplication? If you have one system that's the

source of truth, the system of record for the data, and maybe that's in a relational database, but then to your example, you have that's in a relational database but then you know to your example you have the products in a search engine and you know that's just coincidental it's just accidentally that you have it in both places but you know they're serving different purposes right like it's not it's not true uh a true copy at that point but if you basically had the exact same data in one data store that you consider like your system of record and it's say SQL server, and then you have another one that's

Oracle and it's the exact same data and another one is Postgres and it's the exact same data. Like why? Right. Like at that point, that point, that's like true duplication that you got to figure out like, wait, why are you doing this? Although, you know what I find ironic behind this? And this is always driven me insane. People that have dealt with data warehousing for years, they're they're accustomed to taking data and pushing it out to like little data marts. Right. Because, you know, accounting needs a copy of this data and it needs to be fast and it needs to be true for accounting. Then customer service needs their copy of this data, right?

And so I don't know what it is, this desire not to copy data into multiple different types of systems, but it's okay to put it into multiple databases that could be used by different departments or customers. I don't understand it, but just know that if you hear somebody pushing back about, oh, well, we've already got this data over here. I don't want it over there. It's like, okay, well, is there a good reason to put it over there? If you have a business reason for putting it into a search engine, then that's not duplicating the data. And this is another thing I want to bring up that drove me crazy too, is people have no problems putting multiple indexes on a database table. That is copying the data. When you create an index on a database table, it makes another copy of the data and source it a particular way so that you can get fast searches. So if you have

10 indexes on the table, you now have potentially 10 copies of data and pointers to data and source it a particular way so that you can get fast searches. So if you have 10 indexes on the table, you now have potentially 10 copies of data and pointers to data and all that kind of stuff. So it's, that's how you solve a lot of hard problems is you move data into different formats that can be used in different ways. So, um, all right,

I'm off my soapbox now. I mean, I was thinking just to continue your soapbox for a moment though, like we've talked about the, um, all right, I'm off my soapbox now. I mean, I was thinking just to continue your soapbox for a moment though, like we've talked about the, um, Uber engineering blog and I was trying to find an example of it, uh, as, as you were describing. But, um, I know that I've read some of their blog entries where like they've talked about, um, they've actually like, they'll show you pictures of like, here's how we're moving data around. And, you know,

it's kind of like that example of accidental duplication because yeah, technically, they're copying data off to multiple little places, whether it be, you know, you want to call it a data mart versus just, you know,

a specific database for whatever its purpose is. But, you know, they have a specific need for that data over in the other place. And so not everything is going back to some giant data lake. Because the problem that I have with that is that if you try to engineer some one database to rule them all for your entire enterprise to to do all of your your use cases like that is so unrealistic it doesn't work it doesn't exist depends on your org size i mean if you got three developers you don't want to have like 10 different databases

right but if you had three developers would you be considering an enterprise like right no no right but but in fairness though jay-z if even if you had three developers and you're creating your own you know storefront would you run it all through a sql database well yeah i mean do you make a case for using as much open source software and like kind of outsourcing as much of that stuff so you don't have to build it yourself not even talking about that but i mean what i'm saying is a lot of people at least traditionally would start up with an rdbms and then and then they're like okay well i want i want to search for products on my site do you just tie into the rdbms which is now going to like really start crushing it because now you're competing with transactions for search

right yeah so stand up an elastic search or whatever and put it over there because you are serving the purpose of what what's supposed to be there. So yeah, I don't want to spend my limited resources. You know, we got three developers. I don't want two of them working on recreating a search engine, right? Right. For who knows how long and yeah, limping along and missing all the fancy features I haven't even thought about yet. Or, or trying to support a relational database that's falling over because there's too much traffic hitting it for you know 20 different needs so yeah i found one of the examples of the uh articles i was referring to where they had like uh they're showing database coming from like relational database and key value databases

and cough it's some of it's in Kafka. And then, you know, yeah. So then they put it off into like, you know, another format that they had written. Man, I really loved their blogs on Uber. Their engineering blogs were amazing. Yeah, and trying to find that article, I'm like, ooh, I'm behind. Right? All right, all right.

So who's going to pick up on this next piece here? Me. I can do it. So ACID. We've talked about ACID a few times on this show before. It's an acronym basically standing for atomicity. Oh, no. You know what I'm trying to say, right?

Atomicity. Tell me who the reviews are by. Go ahead. Oh, no, I can't do it. There's a reason can't do it. There's a reason that you do it. Atomicity, consistency, isolation, durability. And we'll go over those a little bit each at a time.

But it's a very old term that's stuck around for a very long time. And it's got definitions. The problem is that the different databases kind of adhere differently to those interpretations and those definitions and kind of, uh, so it doesn't really mean as much. If I tell you, I have a, you know, my,

my database is acid compliant. You know, it doesn't really tell you a whole lot. You get an impression, you know, it's more than nothing, but it's, uh,

it's a marketing term. It's absolutely, that's a great way to put it. Yeah. Yeah. And, uh, in particular,

you know, some letters are worse than others, uh, in terms of being kind of ambiguous and some you can almost kind of take for granted nowadays. But we'll get into them. Yeah. Oh, go ahead. No, go ahead. I was going to get into BASE, which is another acronym that originally started kicking around when people started talking about kind of NoSQL and getting away and kind of saying, you know, maybe we don't need acid. Maybe we could, you know, not care so much about isolation.

Maybe we don't care as much about atomicity and we want the things, we want to trade these for other things. So they started referring to this term base standing for basically available, soft state, additional eventual consistency. But the author makes the point that the term is basically garbage and all it really means is not that acid. So it's kind of like a watered down version of a definition based on an already kind of vague definition for acid. So all it really means is not very acidic. Yeah. So,

um, I'm going to help you out here. We'll, we'll start with Adam. I city. That's how you do it. That's right. The one thing about this, this,

this term always threw me. So at the, the definition of it is something that cannot be broken into smaller parts, but the author actually made a point of like, that maybe it should have been abortability. It would have been a better definition for a, and in acid. Yeah.

I like that. Yeah. Yeah. And, um, so, you know, we talked about like how things would work at kind of like a file level.

You can imagine like one way to kind of deal with like, um, multiple, you know, we talked about like how things would work at kind of like a file level. You can imagine like one way to kind of deal with like, um, multiple, you know, reads and writes to the different locations and file is that you have multiple readers that have handles in the file and you've got one writer and somebody tells the writer, Hey,

okay, go make a change here. And if the readers don't pause, if they still got a, you know, uh, they're not locked out, there's nothing preventing them.

Um, they could actually manage to read this as it's being written so um the file is being written to from some sort of buffer uh that's being shared by these readers and writers and you know maybe a part part of the record gets read before the whole thing is written and that would be bad in a lot of cases if you've got a database you know you imagine you're placing an order or something and the quantity got decremented before the uh you know before the payment got taken or whatever and that would be really bad but another way to do that would be to say let's write the data to a different location and then change the pointer to it all right oh here you go here's an even better

um example if you're like writing code have you ever done this where you are maybe like iterating through an object and mutating the state of it which is already kind of a no-no and you change the first name field on the object and then you change the last name on the object and then some sort of uh exit condition happens and you you know end up reporting that thing but it's only been partially modified and maybe something um you know picks up on that and the whole object has been mutated right because you're modifying that thing in place it was passed by you know reference or whatever but it didn't the operation didn't complete so it's an inconsistent state you know bad news a better way to do that sort of thing

is to basically create a new point of the object make any changes or you know make a copy of that object make your changes over there and then return the new object and do a wholesale swap so that everything is done correctly and it either happens completely or atomically or not at all yeah yeah actually i think we talked about something similar to that example in one of the clean code discussions where like you know why as reasons why you should not modify uh inputs to whatever your function or method is you know but as it relates to this that's a great example of you know why yeah i don't run into like situations like that with code very often anymore because i've kind of gotten away from doing a lot of things that lead to it like for example like global shared state is like a classic way where

that sort of thing happens where there's some memory that's shared by a bunch of different processes or different readers and part of it gets modified and it gets messed up or aborted and you didn't undo that particular change to it and yeah bad stuff happens yeah so i don't know if it builds on that but along these same lines with uh atomicity is or atomicity see it's tough right atom i city out of my city yeah atomicity atomic atomicity um so they call out in terms of multi-threaded programming, what they wanted to say is when it can't be broken in smaller parts is like, if you had two processes trying to access the same data, they could never see it in a bad state, right? They can only see it before the operation and after the operation. So that's kind of what they were talking about. It can't be broken down in smaller parts. So

there's no in-between state like what Jay-Z was just talking about where you could have something not in a proper state. They also say that in the database world with ACID, atomicity has nothing to do with concurrency. They don't care about if there are multiple processes trying to write to a piece of data that's covered under isolation. That's a totally separate part of the ACID description. Um, it does talk about what should happen if there's a fault when performing multiple writes, that's where the atomicity happens. Um, and then what do they have here? Oh yeah. This is just going back to the, Hey, if everything happened properly committed, if it didn't then roll it back. Right. Like that's, that is the atomicity on this whole thing. Um, and without having that, it's hard to know when

it actually completed or when it failed. So this is sort of the guarantee that, that your state is complete. I mean, this is why I really like Joe's example that he gave, because even though it's not necessarily a database, it really illustrates the point that if you are mutating that object in place and you failed for whatever reason and aborted, then how does the caller know which properties you have updated? You know, then it's on the caller, right? Yeah. No, all that it'd have to know what was the state before and what is it now? And is it proper? Is it what I thought it should be? So, so it'd be

really complex for it to do that. I mean, it basically had to keep like a copy of it's of a before. And then here, you know you know when when i get it back let me do i need to revert back which seems kind of a seems kind of wasteful for the caller to have to do um so yeah so without if without adam i city then you wouldn't know like what what was changed and what wasn't you know you know if it's up to the caller to to explicitly roll it back you can imagine like what if the problem was that the database got cut off from the rest of the network? And so it can't communicate back to the client to tell it that there was some sort of problem. Right.

So now it's like database has to decide how long to hold on to that or else it's, you know, how do you pick that back up? Yeah. And like we talked about at the beginning of this, like, I've been pretty impressed with databases in general. Right. And like we talked about at the beginning of this, like I've been pretty impressed with databases in general, right? Like they seem to be pretty, pretty foundationally solid in terms of just working. Yeah. So and they finish this up with saying atomicity is all about getting rid of any rights after an abort so that you don't have to think about it. Right.

Like that is that is the primary goal of this thing. Did we already say, though, about the benefit of it? Like, you don't have to figure out the special logic. You can just retry it if it fails. Yeah. I think we said it earlier. We didn't say it, but it is really great. Yep. All right.

Who's taking consistency? I got it. So consistency, that's the CN acid. Just basically means that the database is in a good, consistent state. It's a property of the application. I don't know what the next sentence means, so I'm just going to skip it. Well, okay, I can say this because what the author was saying was that consistency really had nothing to do with the database. What does consistency mean for your data?

Okay, yeah. It's like it doesn't – it's meaningless, right? Like you changed the – okay, let's go back to the bank ledger transactions, right? You know, if you, if you've decremented the account, right? Like, how do you know that it being, maybe it's should never be a negative number for where you like, you have some special bank account. Um, you know, maybe it's like a child's bank account that, you know, because of limitations, it's not allowed to go negative or something. I don't know. But, um, you know, so consistency in that regard, like that's all about the application's use case to say like the number that the value can't, can't ever be negative, right? It has nothing to do with the database

itself. Yeah. So, so since the Jay-Z didn't want to read that, I almost didn't put it in from the notes because I also, I was like scratching my my head like what it's what defines the invariance for its operations and so what outlaw just said is it's basically the rules around what the data is allowed to be right that's that's sort of it in a nutshell yeah so the rules of your application so we mentioned like you know the bank account example is that there should be uh everything should be balanced if there was. If there was a debit, there should be a credit somewhere else. And everything should always, you add them up back together, it's zero. And if not, boom, you're in a bad state. You are inconsistent.

Like Outlaw said with the example of a bank account that's never allowed to go below zero. If somehow it ends up below zero, boom, right? It's in a legal state. It's in a bad state. The whole application is bad so you know i i can tend to think of kind of consistency as being uh defined as being like the the same across all nodes because that's something we talked about a little bit with like eventually invention consistent databases um but that's really not what we're talking about here well yeah because that

that got into um the author calls that out that one specifically out as replica consistency yeah so that's where like you know this this consistency here is not getting to that level of granularity no um so what you said about like the negative balances and all that kind of stuff that is the invariant that you as the developer have to enforce, right? So you have to make sure that when you write your code, these transactions make sure that things happen completely, but it's up to your code to make sure that things don't end in a bad state. I mean, let's continue on with Joe's balance sheet example, right? We're like, you know, for every, for every, uh, you know, for there to be debits,

there had to be credits, you know, and they're, everything's supposed to equal out, right? Um, to zero. I think that's how accounting works.

Um, you know, the, the, the author calls out here that consistency as it relates to acid refers to an application specific notion of the database being in a quote good state it has nothing to do with the database system itself like microsoft sql server will happily store whatever data it wants to store but that doesn't necessarily mean that your application

is happy with what's in it and so that's where consistency here was weird. Yeah. So they call it out here in a second, but, but one thing that they did mention in the relational database to where, um, the database can help you keep things consistent is if you're using foreign keys, right? So it knows that one of the invariants in the system is you can't use a key in one table. That's a foreign key to another table. If that key doesn't exist, right? So it can stop that right from happening because it's like, hey, you tried to use ID 10 over here, but it doesn't exist. So it can force that from a relational database standpoint, but it can't force your credits and your debits because it doesn't know about them. It doesn't care about them, right? That's on your application. And then they called it out explicitly and said, for that reason alone, the fact that all this logic is on the application,

the C shouldn't even belong in ACID for database purposes. So it should just be aid. Right. So here I'm going to, I get to use my second tangent card of the night. If I could redeem that, thank you. All right. So you brought up foreign keys as an example here, right? Mm-hmm. But as great as they might be, true or false, there are definitely legit use cases where you might decide to scrap all your foreign keys. Man, that's such a tough statement to make.

Because then, are you working? Oh, man. Yes, there are situations, right? But are you truly working with real relational data if you can scrap the foreign keys or is it eventually relational is kind of what you're getting at in some situations right like hey i know i'm going to end up writing these things and eventually that key will exist in both spots um but it's not right now well if you think about it right like if you put that foreign key on there on that table then you're introducing a constraint which means that some overhead is being spent to ensure that that thing exists right and so you're you're taking a hit on that so depending on what

your performance needs are you might not want you not you might not be willing to take that hit and to your point like maybe it will become eventually correct you know like especially like if you were to think of a system where like um i'm trying to load example here but batch loading data is a great example of when you have it when you're when you're batch pulling something over from one system into another and you're filling up tables right and you don't necessarily want to do it in order you You want to do it in parallel. That is one situation where that can totally be an issue. Well, I wasn't even thinking of that.

I was thinking of like, it doesn't have to necessarily be large batch kinds of amounts of data. It could just be because of the cloud world that we live in now, the large networking world that we live in now, you could get data out of order. And so maybe, maybe you get, um, let me think of a good example here. Maybe you get a, uh, not an order. I don't want to talk about orders. Let's talk about like, you get, uh, like if this was a graph network kind of thing, maybe.

And don't get too hung up on the graph part of this. But, you know, Alan's contacts. Right. So you're going to store Joe as as being associated to Alan. But yet you haven't yet gotten Alan yet. So you don't know anything about him. So how can you foreign key to Alan yet? But you eventually will. It's just that the data came in out of order, right? And so you don't want to fail the whole process for that. You know that it'll eventually work itself out.

And oh, by the way, you could trivially write scripts to say, you know, to write, you could trivially write code. They could do cleanup on some period periodic basis to say, Hey, do I have any cases of like where somebody is associated to someone else that doesn't yet exist in the system? Let me either remove that record in the, in this, in this example, it would be, you remove, you're finding Joe's record and removing his record because, you know, he's a contact of Allen, but Allen doesn't exist in the system.

Right. So, I mean, there are, there are reasons why you might want to avoid it. So I say all that because I wouldn't so that I say all that so that we don't get hung up on foreign keys as the like go to reference, you know, the canonical reference for consistency and what consistency means in a database because it it doesn't have to be. No, I think I think the only reason they included it is because if you do have a foreign key, it's like you said, it automatically puts a constraint on those two tables or on the table that the foreign key is defined on. And so if you try to put something in there that doesn't have a key in that primary table, then it will force the invariant on you, which is you can't have a record here that doesn't have a related record over here, right? So it's like one of the only things that, I mean, unique constraints are another one, but there's not a ton of things that the database can automatically force into an invariant for you. Yeah, but not to belabor this point too much, but not everything is going to be a foreign key. So

going back to Joe's ledger example or balance sheet example, you're not going to be a foreign key. So going back to Joe's ledger example or balance sheet example, you know, you're not going to have a foreign key to every dollar amount, right? What if he did? How weird would that be? All right. So let's get into the eye of acid,

which we've already hinted at before, which is isolation and isolation is all about handling concurrency problems or race conditions. So this is going back to, we kind of hinted at this when we talked about it out of my city. I'm definitely saying that correct. Where, where we were talking about like, you know, multiple processes that might be trying to read the same piece of data. Yeah. And so an example here would be like two different clients trying to increment a column.

So when you say increment, we mean basically it reads the value out of it, say the number 99, and it updates it by one to 100. And you imagine two clients trying to do this at the same time, both of them read 99, one of them updates to 100. Next one comes in, updates to 100 again. That would be bad. Those changes weren't isolated, right? What should have happened is one should have gone first and the other should have gone second. And that takes some brains, right? Something has to notice that changes are coming to the same thing and say, wait a second for one of these to finish. And that would be an example of a database that supported good isolation.

Yeah, I think in this part of the book, they gave a very similar example with the two users. And they actually show visually what it would look like with lines drawn. It was very complicated drawing. No, I'm just kidding.'t complicated but um you know it visualized the whole thing that you were just describing and so isolation is is a means of trying to like prevent that uh so that that type of situation can't happen right yeah that diagram in the back uh reminded me of sock sock shoe shoe right and that was i know it's a totally different situation but it's just there's something about that with the actual diagram itself. Anyway, are you, are you referring to a sock shoe sock shoe? No, that's, that's incorrect.

Um, the way he said it was the correct way. Well, you know, Dave looked it up in a super good day, looked it up in a chat GPT and we have a definitive answer with a great explanation on why. No, you already lost me. Chat GPT was your answer. Then like, I know that it can't be trusted i wonder if you'd have chat gpt write you a good resume and then you go shop that around i bet i've heard people you like using right tests and stuff like write me a test on the

colonial war or whatever you know like 20 25 questions and bloop. It's unreal, man. And then the kids just take that and they put the, each question in the answer and loop and chap GPT grades it. Oop. And then why are we doing this for again? Right now? Yeah. Um,

so on this isolation thing, the book actually didn't deep dive onto this because they're going to go into it further and later, later sections like week isolation levels and all kinds of other stuff. So, um, but yeah, I mean, what, what outlaw said is you kind of have to isolate them. They did say though, that, you know, one approach could be, you take the serial approach. And so one, one request comes in, you increment that number. And then serially you take the other requests that came in and you make it piggyback

on the first one. The problem is you get into some real performance hits when you try and start synchronizing all that stuff and doing it serially. The book also mentioned atomic operations here. That's an example in a database. Imagine if the database had a call called increment that

took care of reading the value and incrementing by one all in one single operation so that nothing could ever kind of get in between those steps i mentioned like cpus having instructions that do that sort of thing too to prevent the kind of problems you can imagine where you're like you don't want something being in an inconsistent state and so it lumps kind of a couple steps together for you. Turns them into one. So the last one, the D here is for Duracell. I meant durability. I said it right.

Nothing outlasts. So durability, just meaning that, that once the database has committed a right, the data will not be forgotten. Even if there's a later, a database failure or, you know,

hardware asterisk, you know, put an asterisk on that, but hardware failures occur. And what I mean is if there's problems with whatever medium you write to, then all bets can be off there. This is basically just meaning you've committed it, it's guaranteed that if you started that database up again, you'd be able to read that same value back out. So this is easier. This notion of durability typically means in a single node database that the database has been written to a drive. The data has been written to the drive, and the same data is written to the write ahead log or some similar

implementation. The, and we've talked about the write ahead log before, but this is to ensure that if there is any data corruption, the database can be rebuilt if necessary by reading back from that log of, you know, transactions that haven't yet been committed to whatever their various pages were.

Right. Yep. But in a replicated database, durability means that the data has been written to other nodes. And even that one can really be an asterisk, right? Because that can be an implementation decision, or I should say a configuration decision, that you make as part of whatever your system is. So if we talk about Kafka, our favorite for a moment, you can, as the producer, decide how do you want your data written. Do you want to just write it, write and forget, fire and forget,

and you're just going to trust that it got there? Or do you want to ensure that it got to every replicated broker? So that can be an implementation detail. The performance implication here, though, is that for the database to guarantee this durable, it must wait for those distributed rights to complete before committing the transaction, right? So again, going back to our Kafka scenario, if you have five brokers and

three of them are the replicas for one of your topic partitions or your topic, and you as the producer want to guarantee that your message has been written to every replicated broker before you're willing to consider it done, then you have to wait on all three of those network transactions to happen, right? So, you know, you're introducing additional network latency and IO there for that particular use case. But if, depending on like the mission criticality of your system, that might be perfectly valid and you might be perfectly willing to wait for that. Right. But then you could also think of other extreme examples where like, you know, you might only get one shot to send that message.

And so you just need to like fire it and trust that it got there. So like if you were to think about like maybe a message to Mars Rover, right? Maybe they don't do it this way. They probably don't do it this way. But maybe their window of time to be able to like make that connection to the rover is so limited that they might just like try it and have some kind of authentication mechanism to make sure that it got there complete before it does anything. But they might not be able to wait for like, hey, we're going to try it three times to make sure you got it kind of scenario. I mean, that's a horrible example because there's only the one rover that they're going to send the message to at one time.

But you understand what I'm saying, though. It's like there's situations where you're not going to send the message to it one time, but you, you understand what I'm saying though. It's like, there's, there's, there's situations where you, you're not going to have a large window. So. Yeah. The author also makes the point that that perfect durability doesn't exist. And second law,

third, third one dynamics, right? Uh, entropy, like, uh, the,

there's nothing you can do to, uh, ensure that your data is going to last forever. It was actually a fun article on bbc.com. I saw a saw a while back about uh like what it takes if you really wanted to store something important like uh so there's like there's so articles like this but i don't remember if this is the one that um mentioned this particular example of like people trying to like save information for future generations so should there be like a nuclear war or something crazy that happens that people are able to kind of find some information and start over again with like a head start?

They talk about like what it takes to actually preserve data for, for example, a thousand years. He's like, what are you going to put it on a CD? Like who's going to read that? First of all, you know, like second of all, CDs don't last forever. Hard drives, magnets fail, electrical charges fail, everything fails. Like what are you going to carve it into stone? You know, it's like sometimes the prehistoric kind of things work. So there's some lists. Sometimes the old ways are best.

Yes. Yeah. I mean like, uh, I remember back when, you know, when CDs first came out, you know, like you just thought like, Oh, I'll be able to write, like, I'll be able to archive my stuff here and it'll last forever. Like why would it ever change? Right. It's on this, this piece of medium.

And then, you know, later learning about, well, are you talking about a CD that you bought, uh, like the, you know,

at the record store, you know, like sure that one might last for a long time, but none of them are going to, are guaranteed to last forever. And so the, the CDs that you as a consumer were recording onto had a much lower lifespan. Oh, a few years. Yeah.

I didn't know. So like I just Googled it cause I was, I was curious to find out what some of it was. So for a DVD, a dual-layer, writable DVD, it's five to ten years. Well, actually, even for the erasable DVDs, they were also five to ten years. 10 years so um you have to go to the the cdrs cd writable cds with the gold metal layers those are greater than 100 years of average longevity and they were called archivable um cds if i remember right like you paid a premium for those things. I don't even know how they make such a claim, though, because the CD hasn't even been around that long. How do you test it? But the typical CDs

that you would buy, you're like, oh, hey, Metallica's got a new album coming out. I want to buy that on CD. Those are 50 to 100 years is the average lifespan. Now, why are you buying an LCD? I don't know, man.

I mean, like new cars don't even come with CD players. What are you doing? You want the lossless man. So if you really want your data to last, you have to encode it into your DNA and then pass it on to future generations. That's right.

That's right. Oh, Hey, so the last thing that outlaw sort of hit on this earlier is putting the asterisk on the hardware failure. If somehow all your hardware that your database backups and everything were on got destroyed at once, it doesn't matter what kind of durability plans you got in place. It's gone, right? So you can only do so much. And that's why perfect durability does not exist. Yeah. I mean, you could think of it as like,

okay, I, I, I have a single database, single node database, and I write it to the hard drive. Oops. But my hard drive failed and it's now unrecoverable. Well, then you've lost that entire database. So you're like, okay, well, I'm going to have a database cluster and it's over here in my data warehouse or let me not confuse that term. It's in my tech center. But now a tornado came

and wiped out the entire tech center and so your database cluster, out the entire tech center. And so your database cluster, all the copies are done. So you're like, okay, we're going to multi-site, you know, back that up. So, you know, here's tech center building one and tech center building two is, you know, few miles away, but, you know, maybe maybe an emp takes them both out or something like that's the kind of the point is that uh you you can try to do your best but you can't assume that it's that you're going to be able to get perfect durability right that's something i read uh the book like a bruce snyder book on security a couple years ago and he kind of said the same thing he

said there's no such thing as secure it's a dumb question even ask if something's secure is it secure from what like heat death of the universe is it secure is your sister system secure from someone barging in with a grenade and you know or gun to your head or uh you know from you forgetting the password is it secure from you know it doesn't it's a nonsensical question to say is something secure because there's so many different things. It's an infinite universe of things that could go wrong. hardware and technology progresses, right? So, you know, we try with like ideas like perfect forward secrecy to like combat against something like that. But you know, that's all you can do is try. Yep. All right. Well, so

we'll have plenty of links to some of the stuff that we talked about in the resources we like section. And with that, we head into Alan's to some of the stuff that we talked about in the resources we like section. And with that, we head into Alan's favorite portion of the show. It's the tip of the week. All right. I got another music one for you. Do y'all have instruments that you just hate?

Like musical instruments that you just, every time I don't know, you hear a banjo or something, just like, Oh, did I, a musical instruments that I own or that okay just instruments that like great on my nerves when you hear them you're like oh yeah like why why are nails on a chalkboard a musical instrument i don't understand exactly yeah i can't think of any that i hate okay that's that's good. I mean, it's a good thing.

You know, piccolo is kind of high-pitched. So, you know, maybe I'll give you one. But I'll tell you, one of my deep, dark secrets is I had an instrument that I just hated. It was a saxophone. Really? Every time I saw a saxophone, I heard a saxophone. Just the sounds and images I immediately conjure thinking about saxophone, I just hate all of them. Like, you know, ugh. Drive me nuts. So, not a Kenny G fan? Well, he playedophone. I just hate all of them. So not a Kenny G

fan? I don't count that one. I'm talking about the one that's curvy. The one that you hold on to? Sorry. Give me a second. Aren't they all curvy?

Not the alto. It looks like a clarinet, sort of. Honestly. Oh, really? Yeah. Huh. Yeah. The curvy one is my favorite. It's the jazz one. Yeah. That's the gross one.

That's the one I previously thought of as gross. Well, I tell you, I just celebrate his whole catalog. That's a Michael Bolton reference. What was the guy from um parks and rec um red guy oh he played the saxophone um parks and rec yeah ron oh yeah yeah swanson yeah swanson yeah so he played the saxophone anyway so i for years my entire life i've just hated the saxophone. Even seeing it, I'm just like immediately checked out. Well, I stumbled upon a band on Spotify that features in their last album, at least, a lead instrument of saxophone.

And I don't hate it. In fact, I love it. And I realized that I've been wrong all this time about the saxophone because actually it can be pretty great. And the band is called The Bad Plus. And they're, I don't know, they're kind of a fusion band, I guess. There's some rocky influencers, you know, like rock and roll-ish. There's definitely some jazz kind of stuff in there. They're kind of weird.

The latest album actually reminds me a little bit of like an instrumental radio head, except it said Tom York, it's a saxophone, which if you would have told me that a month ago, I, I mean, I would just immediately shut my brain off and maybe, uh, delete you out of my contacts on my phone or something just for saying that even it was a possibility, but now I've come around. I no longer hate saxophone. Uh, and i think that you should check this out it's great programming music it's a little off kilter it's a little weird uh i don't really know how else to describe other than to say you know maybe kind of like some radio head with a saxophone instead of tom york but maybe uh that sounds great to you i don't know you should try it i like it i actually listened to a little bit of it well while you uh started talking about

it they are they're good yeah so now uh now my least favorite instrument is the piano because what that's like my second favorite well yeah that's that's the problem is it's everyone's second favorite you know it's just it's so good at everything you know it's just not really fair no i okay other instruments okay this is gonna be my tip of the week then hold on i'm gonna find this uh too good it's too good is my problem with it like come on make some room it's not fair no cello also too good it sounds too good the cello is one of my very favorite, man. Yeah, that's why I can't take it. Like, there's not many instruments that can, like, bring out an emotion in you. Like, just hitting a chord on a cello is so good.

I'm going to find it. I'm going to find it. I haven't stopped finding it yet. It's kind of like it's got the beauty of, like, the singing qualities of a violin, but it's got those lower pitches, too too that can really just dig in there. Yeah, they resonate within you. They're amazing. Okay, I found the YouTube channel,

and I'll have a link to this, and this is the correction to Jay-Z's tip of the week. So there is a YouTube channel called Piano Rock. And the artist, and I'm going to mispronounce his name, but I'm going to say Cristello, maybe? Cristello? Is how you might pronounce his name? Sometimes. But he has a truckload of videos where like name any song that is not piano. And he plays a piano only version of it where he's not only playing like the rhythm and the lead instruments of it, but also the vocal parts of it.

He's so ridiculously creative and how he's playing it so he's got videos of him like at you know like a mall right and playing the piano in the mall this this guy you're gonna you're gonna take that back you're gonna take those words back jay-z you're gonna eat those words you're gonna be like piano is definitely one of the greatest instruments it's one of the most versatile one of the greatest instruments. It's one of the most versatile, one of the greatest. And Michael was right, and I will never doubt him again. Those are the words I want to hear. That's the problem with it, though.

I've got to be a contrarian. It's so good. It can do so much and capture. It's just not fair to everyone else, all the other instruments, all the piccolos out there. They all secretly hate the piano. I think the guitar is extremely versatile, too, though. Yeah, I hate that, too. I was going to say, I mean, you as a player of it, there well they all secretly hate i think the guitar is extremely uh versatile too though yeah i hate that too i was gonna say i mean you as a player of it i mean you've heard of tommy emmanuel

right yes oh yes if you watch that guy play a guitar you will forever be mesmerized by somebody who can who can just like master something so well because he uses the guitar as the lead, the rhythm and the percussion line. And he plays it all at the same time with two hands. It's incredible, man. Yeah. I had to start over again.

I would go with the bassoon. I was just watching, there was a video of Tony Emmanuel that popped up that popped up uh just because uh within like you know recent days jeff beck has passed on right and so tommy emmanuel there was a performance that he was doing and he played over the rainbow and dedicated it to jeff beck and i'll have a link to that as well and it is just phenomenal because even when you hear him starting with it, you're like, where's this going?

Like, I don't, I don't recognize this quite at first. And then you hear it and you're like, Oh man, that's so good. How he just did that. Hey,

so just so you know, he does tour around and he's come to Atlanta several times. I've seen him twice. It absolutely go to it. It's, it is jaw dropping stuff. This was a, this was at a recent show,

by the way, the, the, the video them that I'm talking about, this is just days old. Um, obviously that's how he was able to dedicate it. Right.

Um, so, uh, yeah, so, so I think our takeaway there is that, uh, Jay-Z is wrong.

Huh? That was weird though. As a tip of the week. All right. It's never happened before. Weird. Yeah. So, so my tip of the week, other than being correct is, uh, so I don't know how I never

discovered this command before, but now that it is in my arsenal, I will be forever amazing at all things Docker. So the command is docker builder. And specifically why I needed this was you can pass in another, uh, you know, command. So Docker space builder space, and then whatever the command is. And in this case, I wanted to use prune so that I could prune all of the builder cache that I had. So here's the way my use case worked. We've talked about Kubernetes many times. We've talked about our love of Minikube many times. So when you're using minikube as your local kubernetes cluster you're spinning up a vm to house that minikube i'm sorry to house that kubernetes cluster and you know the beauty of that is that with minikube it allows you to from the command line just super simple spin up a different

kubernetes cluster at a different version of kubernetes using different drivers be it docker or parallels or hyperkit or whatever or or even no driver uh at to to um to run that cluster under, right? But the problem that I was facing is that, you know, you also give it a size for like, this is the amount of CPU, here's the amount of memory, and here's the amount of disk space. And I was trying to wrap up some code

for a feature that I've been working on. And in my local Kubernetes cluster that I was devving against, I already had a good amount of data throughout all of this orchestrated system. And I didn't want to lose the data in any of it, right? But at this point, the 150 gigs that I had associated to my mini cube VM was starting to get sparse. of all the different images and, or let's face it. Cause we were, I was using scaffold, which is another thing that we've shared our love of, of before too, as it relates to all things Kubernetes.

So I was using scaffold to do the, the builds, which under the covers is calling Docker build. But, um, because of the, how low my space was getting in order to do the scaffold builds and, or the scaffold runs,

I would literally have to go in, in another window, start juggling like, okay, which Docker images do I no longer need? RMI, those Docker RMI to remove those images. So I could like prune as I go kind of thing in order to like build up the images that I need and then run it so that I could like wrap the, put a bow on this whole change set. Right. And, uh, what I found was we've talked about the Docker system command. So you could do like a Docker system DF or a Docker system prune as an example of space. Um, and so with Docker system DF, you could see like how could see how much space is being used by images, containers, volumes, and build cache.

And you can prune, and there is a filter option. And I'm sure that using the filter option, I could have probably found, the type equals cash or something. But, uh, you know, I was like, I did, I, I, it was easier to just delete the, to delete the images as I, as I went, rather than trying to, like, I was too, I was being lazy and didn't feel like going to look for the, um, uh, you know, what the filter command would have been to use system. Cause with system doc, Docker system prune, you can prune as much as you want. You can get specific. And even as a previous tip of the week, I've talked about how you could give it a date range and you could say like from two weeks and, and all the different date formats that it'll take because dates are hard. But, you know, so you can say like,

hey, you know, everything that's, that's, you know, with new within the last two weeks, I want to keep but the older stuff get rid of, right? Like, that's the kind of cool stuff that you can do with Docker system prune. But it's, it is you what you what what I was concerned about was while i'm perfectly willing to get rid of the volume the i'm sorry the the images i wanted to make sure that none of the volumes were were going to be removed because i didn't want to lose any any data again that was the whole point of all this right so one of that one is system df lines though like i said is the build cache so it's like oh how can i delete the build cache docker builder prune allows you to prune just the build cache from your docker system and in my case i reclaimed like 70

gig from my vm right so disgusting like i i, man, how many times have I just been building over and over and over without, without realizing it? But yeah, so, uh, that, that Docker builder prune, you know, now, now it's in your arsenal too. You can use it as you need. Uh, if you aren't already using Kubernetes, um, you know, it's fun. It's awesome. You'll love it. as you need. Uh, if you aren't already using Kubernetes, um, you know, it's fun. It's awesome. You'll love it.

Once you get to it, it's very hard. Uh, so be prepared to like have a, you're basically like deciding to take on every job in the data center as well as every application developer job too, for everything like front end database, all the different technologies. Let me administer all of it as well as become like a networking guru.

Like, yeah, but, uh, and with all of the ammo, by the way, yeah, all of the ammo,

big deal. No big deal. And the ammo is going to melt your brain at first. You can be like, wait, what? All I did was go to a new line. Why is that an array?

Um, so, uh, the, uh, the, but yeah, so so minikube scaffold these are all great technologies if you haven't already looked at it uh you know you can i think i gave another one where uh in a recent tip of the week where using minikube where you could specify uh the profile using the dash dash

profile so like you could start multiple um minik mini QVMs. And actually, as it relates to this effort that I was going through, like I was actually taking advantage of that ability with mini cube so that I could keep my, keep my pristine data the way I wanted, but I could, I could build up entire other test clusters in mini cube with, with a different profile name that I could dev against and everything. So if you're doing Kubernetes development and you aren't using Minikube, I would love to know if you have a really good reason like, oh, well, because there's this better product or better platform. Man, we should do a whole other episode on this.

I actually looked at MicroK's micro K eight S at one point, and it looked very promising and had problems with that. It looked like a good alternative to many cube, but at any rate, I digress. All right. So mine, funny,

you mentioned YAML. So as, as Jay Z mentioned, one of the things that kind of sucks about Kubernetes is a lot of it is you can do JSON, like you can actually create JSON files, but it seems like YAML is the way that everybody prefers to go. Man, I had a YAML on a deployment the other day that I was trying to wrap my head around. And for whatever reason, I mean, when you're looking at indents and dashes and all kinds of stuff,

like sometimes it's really hard to visualize exactly what that structure is supposed to be. And I find JSON much easier to read because it's all in curly braces. Sure, it's a lot of extra stuff, but it's all grouped off, right? So I found a plugin for visual studio code and I've got the text in, um, in the show notes here for what you can search for. If you're just in visual studio code and you go to the extensions,

you can plug in this hillier.yaml dash plus dashson, and it will allow you to easily convert back and forth between YAML and JSON. So in the case of what I was doing, I needed to take this deployment file and add another container to it to run it as a sidecar. And I just could not figure out where one began and the other one should begin. So I just converted it to JSON, made my changes there and then switched it back to YAML. And it was beautiful. Like it worked perfectly. So super easy to do. You access the features through the command palette, or you can just rename the file from.yaml to.json and it will change it for you, which is super cool. So highly recommend that one. It worked out great. And

then this last one. So you brought up Spotify and it triggered a memory that I just had recently. So I don't know about you guys. I'm sick to death of all the subscriptions that I have to everything. I've got like a hundred of them to everything. Like I don't even know what I have subscriptions to anymore. Um, cool. Then I'm going to send you an affiliate link to subscribe to me, right? There we go. Oh, well paid subscriptions. I don't mind. I don't mind subscribing to coding blocks, right? Like that, that that's free. No, no, no. To subscribe to me. I have my own. Okay. You're not 99 a month. That'll blend in. Well. So I had Spotify. I had Apple TV. I had Apple Arcade. I paid for some space on Apple for backups. And like I started looking like, you know, Spotify is, I love Spotify's interface. It's by far the best out there, period. It's the easiest to integrate with home equipment. If you want to cast it to a Roku or to a TV or to a sound system, like they've done it right. And every piece of

hardware out there has integrated Spotify's protocol, right? So just about anything you buy nowadays, you can play Spotify on. So that's amazing. But I started looking at everything and I was interested. Apple Music actually has lossless. So that's one big one that's amazing. And actually their sound quality,

I swear to you, if you just listen to the Spotify track and then you go listen to the same track on Apple Music, it sounds better. It's definitely a higher fidelity thing. Is Spotify sounds better or the Apple sounds better? No, Apple sounds better. Okay. And so I was like, you know what, let me look into this.

And so I started doing some research and it turns out that Apple also pays their artists better. So I'm not going to say people are making a fortune. I want to say it's a penny per play. Something like that is what Apple will pay the artists. Spotify is a fraction of a penny, like a very small fraction of a penny on what they're paying the artists. Well, you know, we're all Swifties here. And just saying, you know, Swifty isn't on there. Taylor needs her money.

You know, she needs her Spotify plays. She's really not on Apple One. I thought she wasn't on Spotify. She is now. She is now. Oh, she went back? You know who's still not on any of them is Garth Brooks, which is pretty interesting. I wonder why he's a holdout.

I remember Metallica used to be a holdout on a lot of them. Tool was as well, but they also are now there oh is tool on spotify now yeah i looked not too long ago and i couldn't find them yeah when taylor and tool and the beatles went to spotify i was like okay well i'm signing up now yeah so so here's what i'm going to say here that was interesting so the thing that apple pays their artists better i was like oh that's cool i like that right like i like it that people are going to get paid for what they're doing. And I liked the better audio quality. And I found out about this thing called Apple one to where you can roll a bunch of these services into one and save money. So, um,

if you use Apple one, I have a link here in the show notes. Um, you can get Apple music, you can get all the Apple news stuff, which is tons of magazine subscriptions and magazine articles and newspapers and all kinds of stuff. You get Apple TV, you get Apple arcade that can be shared with your family. All of these can be shared with your family. Um, for like 30, I want to say 32 bucks a month, maybe 34 bucks a month. I don't know. High end version.

That's the highest end you can get. And I still ended up saving money over what I had on all the various other subscriptions. And I believe I've got a few hundred gigs of space now for backup that comes to boot. Whereas I was paying for like 50 um you get 200 uh right isn't it i can't remember i think only had 50 and it was always complaining at me that i was out of space so here i i've got it now if okay so you did the premiere for i'm gonna round up to 33

per month okay that's two terabytes of uh storage, storage. Yeah. So two terabytes, but here's the question that I have with this. Okay. So here's another cool thing for the, for those that don't use Apple or don't know, like, and I don't even know if you knew this, Alan, this is the one of the things that I think is like super kind of like respectful that Apple does is they will tell you, Hey, you could actually pay us less money. You could actually save money if you switched to Apple One. Instead of having multiple transactions, you could just have the One and save money. But I say that, I know that because I keep getting those things and I never do it. I did. The reason why I haven't done it though. And I'm, and this is my question to you is,

is that storage plan per person in your family that's on the plan? Or is that total? Cause I have, yes, it's total. Yeah. So, okay. That's your homework assignment is you find out if,

if, you know, are you, are you, are each of your, your family members sharing that storage? Or if, if it's each family members sharing that storage? Or if, if it's each family member has that storage? Cause then I was like, well, how does that work?

If you like already have upgrades for each family member for storage? Yeah, I'm not sure on that. I mean, the interesting thing for me was this allowed me to, so I was planning on upgrading my Spotify to a family plan because my kids all now want to listen to music and they're always stealing my, my, my songs. And I'm like, yo, I'm trying to listen in my truck. Quit stealing it from me. Wait, what does that mean? That's a Spotify thing.

You can steal a session. Yeah. So, so if you were on a single plan, I'd imagine same thing. Well, I don't same thing. Well, I don't guess you'd do it with Apple music cause they wouldn't have access to your account. But on Spotify, if, if you're all logged in as you,

if I'm playing it in my living room and then somebody goes downstairs and, and plays a different song, they take over that stream because you only have one plan, like for one individual. So your wife was like headbanging to tool and then you're like you know what i'm gonna put some jay-z on right now right yeah okay yeah totally and so i went i i was actually gonna upgrade my spotify and then i saw the apple one thing and it was like hey you can save some money i was like all right let me try it out

and i will say as somebody that has a lot of apple devices like like my kids have iPads, I have an iPhone, works out great for that. I won't say that Apple software is the greatest. It does not touch Spotify, right? In terms of Apple Music, the application is nowhere near as good. However, it works on my wife's Android phone, and she's able to do basically everything that I can do. She can do on her phone as well with this Apple one subscription. So it works out for families that have mixed Android and iPhone or iOS Mac users. So I've been pretty impressed with it so far. Um, so, you know, for what it's worth, I actually like the service right now. I've had

it for a couple of months and, and overall pretty happy. Now here's another thing too. Um, I don't know. You tell me, cause I honestly don't know. I, I have been a long time, uh, Apple music, you know, uh, subscriber, but, um, you mentioned that Apple had the higher fidelity audio and the Apple lossless versus what Spotify offers. Have you also noticed that Apple has the spatial audio, Dolby Atmos audio? I have, they only have a couple of albums or not a couple albums.

They're very specific on what they have, right? Like I think there was a Whitney Houston one that just came out you have to pretty good there's a lot in fact they have they have playlists specific to where you can find it it's yeah and some of it is like old stuff too so it's not, but, uh, you combine that with like the new AirPod pros. Yeah. I might try the AirPod pros. I just always felt like their value for what they were compared to a lot of things out there. It didn't measure up. Um, I, I, I, I used to agree with you, man. Uh, but I'm telling you these, they're so amazing.

The new AirPod pros there, they were in my, uh, shopping thing, right? And shopping spray. But so, so, uh, I don't know if it's like maybe a combination with the, the spatial audio, but you could do head tracking on the, on the headphones. Yeah. It's, it's pretty cool, but it's do head tracking on the, on the headphones. Yeah. It's, it's pretty cool, but it's also,

it's also insane. Like how they do the spatial audio with just the headphones. Now I've, I have listened to the Apple music, the spatial audio, the Dolby Atmos audio. I've listened to it both with the Apple AirPods and in a, for real Atmos theater setup.

Right. And it is, it's amazing. Like some of the songs that you can hear where like, you'll hear chimes like go from behind you all the way around you, you know, or whatever. It's just,

there's so many cool things, man. I don't know if Spotify has anything like that. No, Spotify is not even close. So that, that was one of the things that drove me away is Spotify has been talking about for a long time going to lossless or at least better quality.

And it's been on their docket for years and they haven't done it. And it was like, man, you know what? I'm going to make the change and try it out. And I've been happy with it. Like I said, the audio quality is stellar and you can go full lossless if you want to, but even their AAC recordings are, they sound better than the, and the best way I can explain it, if you've never actually listened to a lossless sound versus like an MP3 compressed, listen to something like a drum hit or a cymbal hit, and you'll hear the distortion or the smearing of sound. If you're using a decent set of headphones

on like an MP3 compressed versus a flack or a lossless type thing. And it's a similar type effect when you listen to the got that crunch to it. It sounds very, um, smushed and Spotify on Apple. It sounds very well defined and that, and, and it was super clear when I would listen between the two. And I always felt like three days grace never sounded great on Spotify. And it drove me crazy because I knew that it had sort of like that hard guitar lines, but it never came through. It always felt sort of, um, undefined. And so at any rate, like all that said, if you are interested in saving some money and if you are in the Apple ecosphere at all, um, then check out Apple one, cause it might save you some money. And

the part that got it for me is all of those services are shared with up to six people. So my kids can get Apple arcade games for free now, or we'll include it in my price. They can all read magazines that are in the Apple news plus stuff. They can all listen to their own music without taking over my stream. Right. So like all of that fitness plus, which I don't have an Apple watch probably will never get one. So, um,

that doesn't do anything for me, but it is included in there. So I've, I've used that too. It's actually really cool. It's basically like, how would you like think if you had to compete with Peloton, but without a bike, how might you do it? And so, you know, you get to watch and you can do it.

You can have that kind of experience. So, I hate to do this. Let me give you, like, two bonus tips to go along with this, though. Because since you're talking about, like, high fidelity, like, lossless music, these two are going to come in handy. One is you can go to storage, like on, on your iPhone or whatever your, your iOS devices, you can go to storage and you'll see the music app and how much it's, it's taking up in that. If you click on that, you can actually scroll down to the bottom and you can see everything that you've downloaded and you can choose like, no, I don't want that one downloaded or maybe I want to get rid of all of them. Right. But here's the other thing to be aware of. If you

go to the music app, I'm sorry, not the music app, the, the settings for the music app in the, the settings app. Does that make sense? Yes. If you scroll down, there'll be a section, there's a section called downloads and you can optimize your storage to say how many songs, how much space do you want to reserve for downloads from the store? And you can make decisions about like downloading over seller and more

specifically do you want to download in dolby atmos right so you could turn that on and you could say like hey because the reason why i say that is like i ran into a situation here in the last uh couple weeks where you know my phone was claiming to be out of storage and i'm like how is that possible like i don't have that many apps Like most of my stuff is like pictures and video or, you know, music kind of stuff. Right. And, uh, I looked and the, the music app was taking up like 106 gig on my phone. So I'm like, why, why are you not being smart about this? And it was because I didn't have the optimize turned on. So it was just like, why, why are you not being smart about this? And it was because I didn't have the optimized turned on. So it was just like,

well, no, you wanted to download everything. So we're downloading everything. And it would have, it had stuff that I'd never listened to. I'm like, why would that even be there?

Um, but yeah, so, uh, yeah, I ended up limiting that. So, yeah,

bonus tip. Excellent. All right. Well, we've said enough and, we got to get jay-z out of here it's past his bedtime oh not tonight he's gonna be not tonight oh that's right are you about to start about to pick the theme well you know by the time you're listening to this that's already been picked and the topic was no um so so like I said earlier, subscribe to us on iTunes,

especially with your Apple One. Pour one out for Spotify, but you can still find us there. Stitcher, hey, we're only there once now. That's progress, right? Pretty good. ACID compliant. That's what Stitcher's database is. Now it's ACID compliant.

So yeah, if you haven't already left us a review, you know, Jay-Z, he almost got it, but you can find some helpful links at www.codingblocks.net slash review. Hey, while you're up there, make sure you check out our show notes, examples, discussions, and more. And like we said at the top of the show,

if you want a chance to win this amazing book, leave us a comment on this episode. And, yeah, send your feedback, questions, and rants to our Slack community. And Slack is the best place, but we also got a Twitter at CodingBox. You can see updates on events and stuff like CodingBox Game Jam. Woo-hoo!