Coding Blocks - Understanding Serial Transactions for Databases like Redis

Episode Date: April 3, 2023

We’ve got a new / old opening…Allen goes off / on script? Michael denies Joe the “swing” vote, and Joe is all in on AI assistance Testing for concurrency issues is hard because it’s non-deterministic – basically you get unlucky due to the timing of things Serializability Common Implementations Actual Serial Execution Encapsulating transactions in […]

You're listening to Coding Blocks, episode 207. And we're back to our old intro. Oh, wait. I guess saying the old intro would make it not the old intro, but a new or revised intro. This is the intro, and welcome to it. And so we hope you enjoyed the show. Hit us up at, yeah, have a nice day. I wrote all that down so we can do it next time i compressed it i

super compressed the show there yeah excellent all right we're done we'll see you next time okay you know i'm good i gotta read them i can't i can't do this hey so subscribe to us on itunes spotify stitcher more using your favorite podcast was literally twitching over there like i can't what are we doing it hurts i thought i had bad ocd but i guess i'm not alone though you know the thing that gets me that's really funny is my wife cdo oh yeah that's yeah that's right why isn't it but but there is one thing that in this life i cannot deal with and my wife will do it to me on occasion is she'll open all the cabinet doors and open up all the drawers and i just because she knows that i walk in and i'm like why why would you do this wait she leaves them open on purpose because she knows it drives me insane like i

if you leave a drawer open i will close it like i can't i can't not close a drawer or a cabinet door like i have to do it anyways oh that sounds evil it's wrong it's totally wrong and i'll walk in and i'll try and just turn around and walk out i can't i can't do it so anyways hey yes um so hey also leave us a review if you can if you haven't already you can find some helpful links at cuttingblocks.net slash review and hey uh we got a website with a bunch of sausage links to the top of the page uh cuttingblocks.net uh did hear from the comment section thank you very much that alan was in fact right about everything oh really including yeah including what we said so i don't even remember what what was i right about uh i said sausage links and

outlaw said something i forget what it was that was like it's something funny uh okay yeah i thought i just lost my mind i think you're referring to uh that new podcast distributor uh oh yeah yeah that's right i i don't remember them now but i think it was like stitchify I think you're referring to that new podcast distributor. Oh, yeah. Yeah. That's right. I don't remember them now, but I think it was like Stitchify. And it's an up-and-coming platform that you should probably check out.

If you haven't already, you can subscribe to us there on Stitchify. And, you know, while you're there, hey, leave us a review. Leave a cabinet door open and let us know. That's so wrong. See, I'm already twitching thinking about it. Hey, with that, I'm Alan Underwood. Wait, you're out of order. Yeah, I know I did it on purpose.

You messed me up this morning. I'm Joe Thriving on Chaos Zach. And I'm Michael Outlaw leaving all the drawers open. That's right. All right. So, hey, before we get into the news today, we are continuing on Designing Data Intensive Applications, one of our very favorite books. And this time we're talking about executing transactions serially. Seriously. All right. Serious transactions. Serious. Serious. Serial. First, thanks for some reviews all you want to

take this sure uh so from itunes we have power hungry pygmy so uh huge thanks also i wanted to say like i couldn't remember so uh in that uh review he or she was talking about, um, that the meltdown voice. And I went back listening to it. I couldn't find it, but I didn't listen to the entire episode to find it, but I did go scrubbing around trying to find it and didn't. So, uh,

you know, maybe some help there and I'll, I'll relive the meltdown voice if I can remember what I did during that. But also that he, the episode in reference here was 161. Anybody care to guess why episode 161 might

stick out in our minds for any reason? I don't remember that number. Nothing, dude. I didn't mean to say that out loud. It's the one that started the infamous. It's definitely infamous, by the way. The infamous sock shoe sock shoe debate.

Oh, yes. Which we've clearly followed on my side on that one. Yeah. Sock shoe sock shoe. No. Oh, wait. No. You were shoe sock shoe sock.

That's right. That's right. in case you're in an ice storm yeah oh man what are socks oh golly yeah florida boy down there yeah it's those things you wear with your crocs yeah that man that is such a trend now i see i see adults walking around like that is that a trend now i think that's the trend forever i don't know yeah all right so hey real quick before we get into this one so there is a database technology that's brought up in the book that i'd never heard of i don't know if you guys had it's called volt db you know I hadn't heard of it. So I went and did a little bit of looking on it. And what's really interesting about it is it looks like it's pretty sweet, but they call out that they have an AGPL V3 license. And hearkening back to the days of old, when I think outlaw found this site called TLDR legal, that basically will give you in layman's terms, what the heck these licenses all mean.

This one's really interesting and would be really hard for me to use. So it's, it's basically GPL, except I know you guys, I don't know if we've ever talked about it on this, on any of the podcasts before, but a lot of companies got really irritated with companies like Amazon, who would basically take open source software and wrap it and turn it into a service and be like, okay, here it is. Right. Like elastic search, for example. Um, there was a big uproar about them changing their licenses a couple of years back and they were like, yeah, it's, it's garbage, right? Like we have this, this license in place that says, Hey, you can use this if it's for your personal or commercial use or whatever. But GPL, the whole thing was always copy left, right? Which is basically, Hey, if you go using this or if you change it or whatever, you have to distribute the source code, right? It's the hard copy left type thing. Well, there was a loophole with services because technically, if somebody was using a service, they weren't distributing the software, right?

They were distributing the results of the software. And so there was a loophole for companies that would wrap things and create a service around them. So that's where AGPL came from was it was an addendum to that to basically say, hey, if you're providing a service that is using that software, that is also distributing. And so you have to provide the open source code. So this company decided to go with that. So it's really interesting. You can't just use this thing.

And again, there's the gray area between what you actually have to open source and not. And so for instance, they all, I have a link to an article here from the blog that is basically Google won't allow anybody to use a GPL within their company. If the software is licensed that way, they can't use it. And it's because there's that gray area of, Hey, do I only have to open source the changes I made to that software? If I modified volt DB, that's fine. I'll open source that, but do you have to open source the rest of your

application? And that's where the gray line is. And my guess is that's why they don't allow it. So I just wanted to bring that up because it was a new learning on open source licensing and a technology I hadn't even heard about before that apparently is pretty cool. But yeah. Yeah. Hey, free tip apparently is pretty cool. But, yeah. Hey, free tip, by the way. So, in my personal projects, I've recently started putting the title of MIT license up top and then using the text of AGPL in the license. Which is a great way to get people who aren't reading your licenses to use your product and you get them. Get them.

Yeah. I'm about to retire now. That's a tip that that's his quote unquote tip that's the plan though oh that's great all right so on to the show uh so today we're talking about uh serializable transactions um and continuing on with those at chapter seven yeah i probably should look at uh yeah chapter seven and designing distribute just gbd uh intensive apps that's what that's what uh uh stitchify uses is distributing yeah is there is there notes so yeah all good uh and that we talked a little bit last time uh we're talking about again how um testing for concurrency is really hard because it's non-deterministic i can run

the same test you know multiple times and depending on which one kind of you know hits first i get different results and so how do you know you know if you tested the correct uh condition So one way to get around this is to just stop that from happening. Make your, make the transaction serializable. So when, you know, we talk about serialization, we talk about basically taking some sort of, you know, code or data, whatever, and turning it into, you know, serializing it into another format in order to uh in this case in order to order it the term serializability does keep throwing me off because i just keep thinking about like jason serializability and well as you can code objects too yeah that's what i was going to say like really i think that we're when we're saying serial here we mean like serial versus parallel like

we're talking about doing things one single thing doing it instead of having like multiple things that could be doing yeah and but i do uh you know there's we've got this transaction log in play here too so i like i kind of wonder if it's a little bit of both but i i don't know well i mean i think it it is what outlaw said it's all about the parallel versus the serial. And it sucks because the word serializability messed me up too when I was reading through this. Because it's like, man, it's an overloaded term for, I think, what we do day to day. I just keep thinking about the transaction log and putting things in order.

But that's really not what we're talking about here. Yeah. So, I mean, really, the answer is, as crazy as it sounds as it sounds it's like oh you're having a bunch of concurrency problems stop doing things multi-threaded and just do a single thread and it sounds so anti like how to solve the problem right it's like oh my gosh i have a problem at scale well then don't do things at scale yeah you know kind of thing but that's literally what what the what the answer is here and like it's it's it's trying to figure out like okay well that sounds great right but what's the reality of trying to make that actually happen like how do i do that right so it is oh go ahead well i was

going to say the interesting thing is they say that this problem has been around and being discussed since the 1970s. Of course. Yeah. That's crazy. And doing things this way is the strongest level of isolation. It's the best solution. But I was going to kind of hint at that. There's some problems with it.

So in this case, the upside is the database prevents all race conditions. So your application code doesn't have to worry about it. Even if your transactions run in parallel, they're guaranteed to act and result as if they had run one at a time, one after another. And what's the downside here? So in order to talk about the downsides, we have to talk about the ways that you can do this. And I think there were two or three different kind of implementable, three different ways of doing it. First being. there were two or three different kind of uh implement three three different ways of doing it uh first being well that's that yeah that's what we're gonna get into yeah sorry get in i'm still waking up did anybody else when they heard three not think of like the holy hand grenade now shout do it yeah not once not twice not four sorry yeah all good um but we do uh Not once. Not twice. Not four. Sorry. Yeah. All good.

But we do. The book kind of dives in. Why am I. Why. Hold on. What? You know what that reference is from, Jay-Z? Yeah.

Yeah. Monty Python and the Holy Grail. Yeah. Monty Python's quest for the Holy Grail. You do it. I mean, I assumed it was a Monty Python reference, but I'm trying to remember the scene. Oh, man. But he's reading off the thing about how they must throw the holy hand grenade that shall not do it once or wait three seconds right it's you shall not wait one second that

shall not be the number yeah three no more than three i you know what internet i have is right you i i am coming to a blank like i think this is i think what we are witnessing here is this is when like mark this date this is when the dementia started like this is this is the beginning of the end for me because i'm wow yeah you're usually mr obscure reference like you get them all well i mean that's not even so obscure i did i as soon as i heard it i was like okay i'm pretty sure it's a monty python reference but i don't remember the scene oh man oh it's so good all right that's depressing watch out yeah and now we know where uh which which table alan sat at at high school you know

alan was with the nerds and outlaw outlaws doing cool kid stuff i guess that's right that's right cool Cool kid stuff. Whatever that is. I don't know either. Good stuff. You should watch that whole movie again. For punishment. I will take that as homework.

Yes. This chapter doesn't really dive into the particular solutions that I mentioned too deeply. We just talk about things in terms of a node database, and then it actually dives into multiple nodes in Chapter 9, so it skips a chapter and then kind of picks back up with distributed systems. And I don't know if you looked at the title for Chapter 8 yet.

That's like the problem with distributed systems. And yeah, I feel like we've already been talking about the problem. So I'm kind of scared about that one. I haven't read it yet. Have we been doing this as we've been going along? Right. Yeah. All right.

So, Hey, the first one that, that they actually talk about is the one that we're spending this entire episode on because there's actually a lot here and it's true serial execution, right? Like run one right after the other so yeah the easiest way to get rid of race conditions is to just do it that way no concurrency whatsoever right like everything that comes in you're like hey get in line and we'll do you next it's funny

it kind of sounds like this would have been the first one that he would have started with you know things being single threaded and whatnot but it's not um the book says that this was only really uh people only really got around to it in 2007 and the reason is because uh things were just too slow before then you know the computers the lack of memory all that stuff made it so you just really couldn't uh scale up enough in order to support this so 2007 was the first time do you remember what database it said was the first one that i didn't see it i don't think they mentioned a first database just that did it um yeah they linked to a paper but um yeah i didn't see if the paper was tied to a particular database well i can look to see the paper but the i kind of was like reading between the lines here and

just kind of assumed that, well, maybe that, cause the author didn't go into this part of the detail. So this is where I was like reading in between the lines was that maybe like we, as you know, a community we're getting like thought we were being creative by trying to do things and, uh,

you know, multiple threads and, and, you know, thought like, Oh yeah, this, obviously this will be better.

And, you know, realized that it wasn't, but you know, maybe at the time, yeah, obviously this will be better. And, you know, realized that it wasn't. But, you know, maybe at the time, to his point about being able to do things in memory and whatnot, that, you know, based on the hardware that we had at the time, that that was, you know, maybe a more performant way based on what we had. But over time, they learned that like, oh, I guess that's not actually so great and creates more problems than it solves. Well, I mean, that's that's where they start talking about like, hey, why did it start becoming a thing in 2007? Right. Like so the problem's been discussed since the 70s.

Why did it take them almost 40 years to start doing this? Right. Why did it take us 40 years to do everything that was described in the 70s? And what were they doing in the 70s man like i mean let's see we had kiss coming around aerosmith black sabbath i mean it was a great time i guess but you know like what like why was it all the great computer like thoughts were made then or thought of then but they didn't get implemented until now yeah who knows but yeah this is where they say hey ram finally became cheap enough to where you could store an entire data set in memory like it was it was only about them when we started to have that flip to where you you didn't have to have everything on disk and they said basically if you could have it all in

memory it's way faster, right? Because you're not having to wait on the I.O. to the drive in order to do some of these things. By the way, that reference was not to, didn't reference a particular database. It was just a white paper. Yeah, I got that. The name of the paper is pretty funny. I just closed out tab, though. But the end of the architectural era, it's time for a complete complete rewrite and i was kind of wanting to see if there was some database

that was um inspired by it i was gonna ask chat chat gpt oh so here we go yeah confidently wrong well i don't i've i i don't know i'm pro i'm still loving it uh and as wrong as it is uh asking humans and google also very wrong i was gonna just say like our have any databases been inspired by this paper so we can keep going it's probably not going to turn out well so basically ram became it became cheap enough to buy ram a lot of ram and we could get ram in large quantities on a single stick so therefore you could have an obscene amount in one machine which you know obscene amount by 2007 standards is like what you you probably have more in your iphone today right yeah but but anyway so that that was part

of the thing is that that it was becoming cheaper that you could avoid some of that overhead of trying to manage the multiple threads now just do it all in memory yep and then they also at the same time they had people looking at the the designers of databases looking at, hey, what actually happens in a lot of these online transactions, right? And they found that they're usually short-lived and they're really small reads and writes. And so they figured, hey, they can just be run on a consistent snapshot using snapshot isolation which which we talked about in a previous episode outside of the serial execution loop so so basically if you start from a steady state then you can do these updates and it's real fast and then you can move on to the next one

well they also talked about i think it was this part correct me if i'm wrong where they talked about like uh the author talked about being able to push readers off to replicas and, and the writers could be focused on just the, you know, whichever one had that right. I remember seeing that somewhere. So that way, that way, like in terms of like analyzing your traffic that's coming to the database, right? Like, you know, if,

if 90% of your traffic is just readers, then you don't have to have them go through that single thread, you know, on the one that can control the rights. So that way you can kind of improve that performance. But that's where part of that snapshot isolation comes into play. I think that comes in on a little bit later, but we'll be getting to it today though I believe when we get into partitioning

by the way, ChetGPT did confirm what I was saying which is there was no direct descendant with this paper kind of like the Dynamo paper there was the Dynamo paper and then Dynamo but it did mention three databases that were heavily inspired by it and mentioned it was Cassandra

Bigtable and dynamo big table yeah awesome uh yeah so um sometimes single threat systems can perform better than concurrent ones which is pretty interesting like you just kind of think of concurrency as being ultimately faster in terms of the clock time you know something taking you know our like humans perception of time so maybe it's taking more time than cpus but generally you know in terms of observed time it goes faster but it's not the case um and not always the case i should say um so in the case where uh we've got a single core there's not as much overhead like things just kind of you know do the instruction in a very simple way which is nice but if you've got a transaction that needs to or sorry not transaction um if you've got uh concurrent um

concurrency going on you've got thread switching you've got management overhead you've got uh bad stuff um that potentially you know happen has to be dealt with if things line up just right we're going to talk about that a little bit more ahead of time but just wanted to kind of point out that um single threaded doesn't mean slower you know what's kind of crazy here though too is that um in the time that this was becoming a thing in 2007 right i just searched it up because i was curious. The first multi-core processor was invented in 2001, or at least introduced in 2001 by IBM. So remember, we used to have physically different gigantic processors. It was a big deal to buy a multi, uh, processor motherboard. And then, you know, you'd have to buy the two processors, multiple fans or whatever your cooling system was,

was probably fans, but, uh, or heat sinks, I guess it was probably just heat sinks too though. Right. Like that was a thing. So anyway, um, but they were much smaller than that, but yeah. But then as it became available to where you could just do this all on a single chip we're like nah we're gonna go back to single yeah thread all the things it turns out that's faster you know my cpu heatsinks are like bigger than my video card now yeah they can be for sure so So you were talking about the single core and that's what you've got, right? The problem with that is you're bound by a single core. You can't take advantage of more if you're doing the serial transaction.

You can't scale up. Yeah, you can't scale up in that regard. And they say that you also have to set up these transactions differently, which, which is interesting. Like you have to think about it different, but we also didn't mention which databases currently like the popular, I guess,

ones, which I don't know, are they all popular? Volt DB, which we talked about at the opening in H store, Redis and the atomic. Yeah. I mean, I've heard of Redis.

I hadn't heard of Datomic or VaultDB. No, I don't really think of Redis as being a database either, but I mean, it stores data. You know what's funny about that? I had the same exact thought. I actually typed into Google, is Redis a database? And it takes you to Redis' page where they say that redis is a database yeah and they're getting into streams now there's redis streams that are kind of like lightweight

kafka streams it's kind of everything yeah i mean it's just a very simple you know because it's a key value type storage but yeah i mean I mean, storing data and extending their boundaries a little bit to kind of grow into different spaces, which is cool. I mean, it's a smart move on, on their part. Hey,

so real quick, this, I always love an opportunity to bring this back to the stack exchange, their performance thing, how they set up their infrastructure and talking about having a database and memory. So if you go to that page, which is stack exchange.com slash performance, and you scroll down, they have two Reddit servers that they use to serve the caching part of this. They each have 256 gigs of Ram, right right so that goes back to the you could

store entire data sets in ram and that's why they can do some of this because you can have servers with just obscene amounts of ram nowadays so pretty cool i always like going back to that because anytime i see it i'm like man, they actually run super lean in terms of hardware. And it's pretty interesting to watch it. Oh, yeah. So on to one of our old favorite topics, store procedures. And we actually just talked about this last episode or maybe two episodes ago. We just kind of hit back onto it.

We used to talk about store procedures a lot more back in the day, uh, when we were dealing with less databases. Uh, but, uh, in this book, they talked about how in the early implementations databases had the

intention of making the entire flow part of the transaction. So, um, if you were booking a flight, for example, the person might be shown a list of the flights. They kind of click in, they look at the seats,

they, you know, add their baggage options, you know, add their baggage options. You know, they check whether they want like aisle or whatever. And then at the end, they do it and commit. And then transaction would be committed, which if you think about it, you know, it's kind of crazy that we might be holding up a transaction all this time. But, you know, there's queries going back and forth.

We can't do that all as part of one single store procedure procedure because there's you know a human evolved at multiple parts of that step and so we don't know even what queries we're going to run if they're hitting back button or whatever so uh you know nowadays we talk about transactions like that's not what we're generally thinking about at all but that that was the original kind of thought process there which is um you know which is pretty interesting and you can imagine that being single-threaded terrible like it's almost like uh there's a like a you know a starbucks or something you've got someone on the counter and there's only one barista working and everyone trying to buy a plane tickets like waiting for whoever's first in line to kind of pick their seat which is just crazy and then if they decide not to or they go you know get the

laundry out of the whatever move to the dryer uh everyone's just sitting there waiting uh for that transaction to commit which is unfeasible especially so for a single core uh serializable i just when he was describing that part of the you know where you were keeping it the transaction open you know like maybe there's like okay okay, we already established that some dementia is setting in, but I was like, wait, have I ever worked on an application where we did something like that, where we would like open up a transaction, query the database for something like, oh, okay. I need more input from the user. Let me keep this transaction open, go back to the user get some more information

then go back i'm like i don't think i ever did anything like that but then also i was questioning like well maybe that's because like maybe this was a thing like if you were a mainframe developer or something maybe that was more of a thing then that's exactly definitely in a web world i'm like i just i how would you do why i think it does kind of do something similar as a ticket master um when you like kind of go pick hold a seat we're like you've got to check out in two minutes you're gonna lose these seats right yeah i mean that's sort of like a forced way around it at this point but yeah i had the same thought outlaw like seriously same thought you don't have to do that with the open transaction though for the ticket master thing though that's not no like they're not doing an open transaction i'm sure

they're sort of faking the same type thing that used to happen back in the day yep this is more like what we're talking about here is more like hey sorry the computer's in use right now so you gotta come back right there's There's already an open transaction. Hold, please. Mom, get off the phone. I need to use the internet. Right. Yeah.

Exactly. You should try buying your plane tickets in the middle of the night because it slows traffic. Isn't that crazy? That's probably the type of thing that would have happened. Yep. But, yeah. Yeah, so where were we?

So, yeah, it seems totally crazy in the modern kind of web world, but there can still be situations where a transaction can occur with multiple interactions, you know, kind of like we mentioned with like Ticketmaster, the seat picking thing is a good example. But these are now up to the application to kind of query and see if it's available. And then, you know, send another query and see if it's still available, you know, have the customer kind of switch around.

We know we covered that, but doing it this way in a serial transaction database would be slow. I'm just repeating the same stuff, sorry. I mean, but that makes sense. So what they called out here that's important is the one that we talked about a second ago that seemed insane, right? Like let a person start a transaction and then finish it 10 minutes later. With an in-memory database and the way that they have to handle transactions, even though your application server can talk to the database way

faster, still having those interactions to where, all right, let me go see if it's available. Okay, it's available. Let me lock it. All right. Now let me try and let me try and add it and then make sure that it didn't conflict with something else. All those interactions take time. There's latency across network and everything. So it's still a problem if you're trying to do these serial transactions. So it is important, even though it's way faster, it's still an issue. But I didn't read this part as like having necessarily anything to do or any dependence on the memory comment like the amount of memory because they actually referred

to you know this sequel i'm sorry um store procedures becoming a stand a sequel standard in 99 which was you know eight years before that article that talked about rewriting everything to to do things serially so really the point that i got out of this part part was that by doing everything in this by pushing that logic into the store procedure then you were reducing the latencies needed to like get information. You could just do everything really quickly in one transaction and be done with it. And, and so like the, the example that the author gave in one of the diagrams was like, okay, you, you go back to that on-call thing, right? Where like you're, you would select that doctor's on-call schedule. And then, you know, so, so one transaction was for that. Then you would do some

if logic, okay, well, I guess I need to update that thing. And then you would go back to the database again to do the update. And instead they consolidated all of that into one inside of the store procedure where it's like, okay, let me select the value. And as part of the store procedure, if the value is greater than this, let me do the update and it could happen you know in a much tighter time frame yeah and but that was the important thing right is having to do it in a stored proc instead of um your application managing the transaction like you can do in some databases right like you could open up a transaction in in sql server in your app code and and do a bunch of things there and then commit it at the end this like the redis and the volt db and those they want it in a stored proc so that you give it everything it

needs at once and then it can do everything it needs in that stored proc yeah i mean it actually i mean we've talked about before you know or at least i've talked about before where like i'd worked on applications in the past where you know all database logic we pushed into stored procedures so that you know you the developer it had several advantages right like you the developer are less aware of like the underlying schema for like tables and columns and things like that you only care about like the stored procedures and the inputs and outputs type of thing so it abstracts some of that Plus you were keeping the data logic close to the data, right? Which I don't know, over the years, we've kind of like grown, you know, maybe drifted away some because we're like, well,

it's easier to horizontally scale some of that logic if you put it in your app code, because you can't scale the database code as well. But this part of the chapter kind of like, I don't know kind of reignited that that old love affair that i had with the the other architecture i'm like oh that is kind of neat you know you know good points about uh why you could do that but he also going back to the volt db talked about like some of the distributed systems where i i thought this was such a cool idea where in order, instead of like writing to one and then letting the change replicate out,

it would do the exact same stored procedure call on every instance. And then that way, you know, there was no networking sync needing to be synced between the replicas, but it did rely on whatever that stored procedure does has to be deterministic and like calls for like if you needed to get like a date time for example that they had specific calls or apis that would produce the same exact result on every one of the replicas and i was like oh man my mind blown like i want to see how that code works

that that code i want i want volt db to distribute that code so i can see that code so hey and and also for those following along in the book the there is a really good diagram showing the difference between like a regular transaction versus the serial one with the store proc where everything has to get passed in that's figure seven nine in the book so um if you have it go take a look at that i didn't realize we were gonna start referencing the figure numbers well this one was good and you know obviously the notes then yeah i don't want to screenshot it. Put it on the page, right?

Like we're going to get in trouble. Hey, so it's that time. Hey, Jay-Z, you going to do this one? Yep. Oh, man. So, yeah, no. I would like to ask you all to please consider leaving a review with whatever number of stars you feel is appropriate. Preferably five.

It's definitely five. Five. Definitely five. We love those fives. Yeah, it's kind of silly that... That they even let you do a four? Like, that's silly. Yeah, let's just do ones or fives.

Like, ones are for bombing and should all just be kind of removed algorithmically. And then fives should be, you know, just kind of a count of your popularity. Let people know how cool we are. Let people know that you like us. and then five should be you know uh just kind of account of your popularity like let us let's let people know how cool we are right let people know that you like us let us know it's great are you sitting seated at alan's table or yeah yeah sit down hang out with alan's table like screw alan's table let him know dark turn fast man like first of all first of all this became so weird because suddenly like i'm the cool guy is what you were saying and that and now it's like also screw me so like oh wow that's it got like really good there for a moment and

also took a really quick turn this is why i'm not allowed to ask reviews it's not even so much the one star thing as it is like my uh whenever i do it's like this weird fever dream of like words that are kind of loosely related to reviews um so if you want more of that i guess you should you should leave a review and you can do it we try to make it easy couldn't watch that slash review the word salad salad Jay-Z. That's good. Jumbo. It's a mix. Toss it around. Oh, man.

All right. So with that, it's time for my favorite portion of the show. Survey says. All right. So to Tuck Co.'s trademark rules of engagement, Alan, you are first.

Yeah. Let me see if it works. You can try to win this time. I try to win every time, man. I know it's sad. All right. So name a specific piece of furniture that might be handmade.

Table. Okay. Let's go with a piece of furniture. Swing. A swing. That's furniture, right? It is. All right. It's so bad if you get more points than me on making couches let's face it yeah right so uh alan says table joe says swing so the number

one answer on the board you know what no let's go backwards oh man So there were seven answers on the board. Number seven answer is bench for two. Cabinet for two is the sixth. The fifth answer, bed for two. What? Fourth most popular answer is dresser for five. Third most popular, shelf or bookshelf for 12. Swing number two. All right.

Now we, we can, this is where there's, there's, there can be debates made, but I don't know. I think I want to just shut some stuff down, but I'll let you guys have your debates.

Number two answer on the board is 31 points chair and the number one answer for 41 points is table.

Look at that. I'm sorry. I meant table. A swing is really kind of like a table with see i was expecting you to say it's really like a chair no were you really expecting that what a win outlaw okay well um you're off you're off to a rocky start there my friend swing is a chair with ropes attached you know honestly i think they should have said bed frame not bed because who's stuffing mattresses right yeah i agree with that well they didn't say mattress they just said bed but yeah sure uh

but yeah honestly i expected some some pushback on the chair you know like a swing being a type of chair that's just on on strings no. Chairs suck. Swings rock. Right. Way better. Way better. You're not wrong. You're not wrong. So, Jay-Z, name something you find

in a breakfast buffet. Oh. And your choices are... No, I'm just kidding. Yeah. I I mean I think I know the obvious answer but there's also the things that I want to see in a buffet and so I kind of intended to say one of those anyway so I'm going to go with sausage even though

it's not number one it's from the heart and that matters this is tough do i go eggs or bacon um i won't put eggs let's say eggs they're gonna be cold though they are gonna be cold or usually runny too because they don't cook them well yeah yeah no matter what you know those eggs gonna be bad be there but they'll be there yeah that's not from the heart all right number seven answer on the board melon for two points number six coffee for six points fifth most popular answer orange juice for seven

number four answer on the board potatoes slash hash browns for 12. Sausage is the third answer for 19 points on the board. And it's way better than bacon, by the way. Ooh. Bacon is the number two answer for 24 points. And bacon just slaps your sausage around and is like, get out of here. This is so basic. So basic. Not if it so much cooked right not if it's cooked right but it never is on a buffet never is you know what though they make up for it with quantity they do so you just tell it off you just

you just take the tray and you dump that on your plate and you might have to get a second plate just like sandwich it to hold it together to get it back to your table and then you got like you know eight pounds of bacon yeah yeah and a heart attack uh number one answer on the board waffles for 25 points are you serious no it was eggs i just wanted to give alan a little bit of a heart attack but honestly why is it waffles or pancakes somewhere on the list waffles should be there it's always the people are thirsty right like coffee and orange juice come on getting greedy yeah yeah where's the business i dude i actually thought biscuits gravy clearly they've never been to a breakfast buffet in the south right yeah like we have liquid heart attack we have heart attack from a pig we have heart attack from a chicken

that's right all right well uh as is Alan, you get to pick the final. You ready for this? I am. Where are we at? We are two answers in. Alan has a commanding lead at 66 points to Joe's 19. So just a short, small, a little bit of a deficit there from the heart. So I'm going to go out on a limb, and I'm going to say that,

Jay-Z, I think you lost this one. I don't know that it's going to matter, because for whatever question he picks, you pretty much have to nail the number one answer, and it has to be so ridiculously popular. It has to be like so ridiculously popular like it has to it has to you know and you are really hoping that he messes up entirely this is where strategy could come into play right like you're getting ready to give me three categories and i can choose the most obscure one which almost guarantees that nobody's gonna have a good answer for them yeah exactly yeah yeah

all right so enough movies to know that I'm going to win, though. Good guy always wins. I'm the main character. Have you seen that Reddit? Wait, Deadpool, is he the good guy? Obviously.

I don't know what he is. Obviously, he's the good guy. Alright, so here are your choices, Alan. I don't know what he is. Obviously, he's a good guy. All right. So here are your choices, Alan. Name a planet. Name something you have to do that would give you a reason to set an alarm to wake up to. Or name something you haven't done since high school gym class. Let's do the gym class.

All right. I skipped that. Well, then that could be your answer. I haven't skipped it. Climb a rope. Okay. So your answer is climb a rope. Alright.

That was seriously your answer? I wasn't sure because you jumped in so fast there. Okay. Jay-Z? Change clothes in front of a bunch of classmates? True. I've known that a long time.

All right. You know what? It's either change or climb a rope or it's play kickball. I guarantee it's one of those two is number one. Kickball's coming back. Well, but it's climb a rope. That was my answer. Yeah.

Yeah. Yeah. Exercise is the number nine answer on the board. Just generally, I haven't exercised since high school gym class. That is sad. That's something about our people. All right. So that was only two points, though. Okay. That is sad. That says something about our people. Alright, so that was only two points though. Okay, that's good. Only two people. Tennis was number eight on

the board for two points. Oh, fancy. Yeah, exactly. That's what I thought. Like, who was playing tennis as part of high school gym class? We had square dancing instead of tennis. Yeah, we had tennis also. Wow. Tennis.

Like, not because you're on the tennis team, but because the coach is like, okay, today we're going to learn tennis. Yeah, even how to score it and everything. Wow. I forgot. Alan is from the UK, so they do things different over there.

Oh, yeah. Well, we had our tea and croppets. Yeah. Somebody across the pond is going to take offense to that. Wait a minute. Please don't take offense. We would never have tea and crumpets that early in the morning. I don't even know what crumpets are.

Is that like a little biscuit cracker or something? What is it? Okay. Wow. I'm going to get some hate mail now. You can direct that to uh hit me up on slack at joe so what what number are we on uh number seven answer on the board for three points

change in a locker room wow look at jay-Z getting an answer. Jay-Z on the board. He is now tied with Allen. Oh, wait. No, I'm sorry. He is 44 points behind Allen, so he needs Allen to get a negative score here. I'm just saying I called it. Number six answer on the board is swim laps for five points. Lift weights is the fifth answer for eight points.

Number four answers sit ups slash crunches for nine. Push ups is the number three answer for 21 points. Number two answer on the board for 23 points is dodgeball. And the number one answer for 25 points, run a mile. Oh, man. I got zero. So I would say you had a chance, Jay-Z, but really you didn't. Okay, let's do the math.

Let's do the math. Oh, my gosh. Could it be? Hey, you know who's on a winning streak now, right? That's right. Wait, did you win last time? I thought Jay-Z won last time. I'm starting a new streak.

Oh, this is the start of a new streak. Okay. That's where I messed up. You know what? That's my bad. That's right. I should have recognized that. That's my bad.

Okay. Well, yeah. So with that, Alan is on a winning streak of one. Yes, yes. How can you sleep at night with those answers? You should be ashamed of yourself, sir. Okay, well, let's get back to the book. You know, I was reading a good book on the history of glue.

I just can't put it down we can get back we can get back to this book though in the meantime so all right weren't we already talking about this part though the pros and cons of uh yeah we just talked about like we didn't talk about the pros and cons of them we talked about the fact that they sort of need to be used in these um serial serial transaction dbs and this is funny so they say that they get a bad rep and i know all three of us have heard people complain about stored procedures at some point in our careers and they're they actually named out some reasonable reasons in the book one of them is each vendor has their own language their own implementation right like t-sequel looks different than pl sequel looks different than p-sequel and you know so on so on so that's one

of them and i agree with that like the the the syntax the, all of it's just different. Well, they adhere to the ANSI standards for SQL, but then they add on to it. And so they'll have like their own little ways of doing certain things. Yeah, totally. And then they also mentioned, and this one was pretty interesting, that the SQL language hasn't kept up with other programming languages. So basically what they mean is like, you're not going to have all the functions out there that you're going to have in a regular language, which makes sense.

Yeah. Yeah. No generics, right? You can't type T everything. Um, and there were some other things that I actually, um,

agreed with here. It's hard to manage code stored on a database server. So there's ways around it like we found out ways to to get stored prox and to get and all that kind of stuff or choose your i actually thought that part was a little bit dated when you're talking about the difficult to keep in source control like that was the that was one of the complaints that was made here and i thought it was seriously dated. It's a little dated. There are tools out there, so many tools out there to do things in the migrations kind of format.

Like Roundhouse is one, but there's FlywayDB is another. And that's assuming that you even go that route. Microsoft had, what was that other thing that they had that was like that the dax the dac packs dac packs are using database projects and visual studio yeah i mean so there there have been ways to do this for a long time that i just it was hard to give credit to that one so but, but you know, that's being super nitpicky. Yeah. I was going to say,

I think in fairness though, like using roundhouse or fly away or something like that is learning a new process just to be able to get your stuff to deploy in a good way and, and to put that stuff in source control. They're good solutions, but it's not as easy as, Hey, just learn how to commit to get and be done.

Right? Like it's, it's not that brain dead simple, but I don't know. I mean, that's like saying you could make the same argument about any other language, right? Like, Oh, I just want to write my, my, my Java class file or my C plus plus class file or C, you know, routine. And I don't want to care how to compile it and make it part of the library, right? Like doing the migration is just a way of getting it compiled into the database. So I don't know.

It's to me. No, it wasn't like it's no, it's no harder a stretch than to like, you know, get up to speed with Maven or MS build or.net build or whatever, you know what I'm saying? Like,

I don't know. That's sort of fair. I guess, I guess the, the only line in the sand I'd say there is as somebody, if, if you just work in.net, then visual studio does everything for you.

You probably don't even know what's going on behind the scenes. If you haven't been doing it for a long time and don't have to deal with build systems, right? Like it just is sort of magic, sort of the same thing with intellij to a certain degree you have to know a little bit more if you're using maven or gradle or something like that you have to know a little bit more about it but they can actually hide a lot of that stuff with roundhouse and flyway you are forced into the world where you have to understand how that

stuff works so i don't know i, you make a valid point though. This one's definitely thinner in terms of how, how important this is. This was the weakest of all the reasons that they get a bad rep was. Yeah. It's kind of my point. The next one is legit, harder to,

to debug. Um, it's a database, right? Like it has changing data so so you have to have something set up to be able to do your test properly and know that you're going to be modifying data all over the place so that's one not saying that like i remember in sql server um it was a sql server management studio came out with the ability to debug storm prox and you could actually step

through them and it was really nice, but you have to have your data set up in a way to where you can actually test the stuff that you need to test. You couldn't really do it line by line the same way you can with code because the whole query was whatever. You could mouse over and see data and stuff. It was still much better than trying to put

a bunch of print statements in. Totally. We already said it's more difficult to keep in source control also pretty pretty lame uh more difficult to test so you can't like unit test something kind of the same way that you would um i guess more of what you'd be doing here would be integration tests this is the thing where i feel like we have failed We as a developer community across the globe have failed because I have, like every time we bring in like a new database engineer, you know, one of my first questions is like, hey, what kind of, have you,

are you aware of any like ways to test your database or any of the routines or like how do you how do you validate things like and it's always you know there's never a really good answer for it yeah it's like it's like total state manipulation all the time so you can lock out stuff you can make the table name dynamic you can create fake data and run tests but it's just all so awkward yeah like you talk about the code unit tests being hard to write and maintain and people pushing back on it like database tests. I mean, it's almost to your point earlier. I think Jay-Z said it a few minutes ago.

Like it's almost entirely integration test. So it's and we going back to past episodes, we definitely prefer unit tests over integration tests just because the automation around trying to spin up a database or you just assume that this database exists and let me just go connect to it. And yeah, so that's awful, too. Yeah. Yeah. I really wish there were. Society needs to solve that problem. Yeah.

It's hard when the whole purpose of a database is just keep changing data, right? So it definitely is a harder problem to solve. This one's interesting. I think it's gotten better over the years, but they say it's more difficult to gather metrics for monitoring. Like in your application code, you can just throw a metric in there and have it spit it out to something, whether it's Prometheus or whatever, right? In a database server, you're kind of tied to whatever they expose. And,

and that's true. I mean, like I said, it's gotten a lot better. And if you're using anything in the cloud, they have all kinds of monitoring stuff around it, which is awesome. Well,

I don't know. I mean, if we're talking about like, I feel like we're, we're not comparing apples to apples there because on the one hand you talk about like comparing writing metrics out from your application and then compared that to metrics from some other application that's outside of your control which really if you were going to compare apples to apples they'd be comparing you putting out

metrics in your application versus what your like web server like if you're using an engine engine x for example like what metrics is it pushing out yeah sure you're beholden to like whatever apache has decided to output from that metric server i think the the more to the point here would be like if you wanted metrics spit out as part of your like if you're going since we're in the store procedure section here if you wanted metrics being spit out as part of your like if you're going since we're in the store procedure section here if you want to metrics being spit out as part of your store procedure to say like hey this many records were read this many were written blah blah blah you can do that but there's definitely like a latency hit that you can possibly take depending on like how those are you know like

if you're logging those to another table, for example, you know, that's kind of, that's kind of gross. And, you know, there are some like a Postgres where you can output things to like standard out or standard error. But,

um, I'm not sure about like, you know, what the abilities are in like an Oracle for something like that. But I would assume that it's there. Yeah, I totally messed up. I made the comparison between the engine and,

and another application. And we are talking about store products. So yeah, it'd be exposing metrics from your stored proc is, is more difficult than, than from like an application code. Um, so the other thing,

and we've seen this, we've seen this so many times because databases are shared by tons of applications and tons of different developers somebody who puts together a store product that's not performant has or can have massive repercussions on the entire system as a whole right like you can be blocking things you could you could completely tank the database like there's all kinds of stuff like uh i remember back in yeah deadlocks you guys remember like orms back in the day as much as people hate stored products there's probably more hate for orms from the standpoint that people don't know what they're doing when they're like hey give me everything from this table and then do some operation on everything in the table you

know like it's easy to it's easy to make something really bad yeah orm's definitely got a bad rep yeah this one's pretty funny so um so some of these issues have been uh you know fixed over the years so modern serializable databases some of the ones we've talked about like volt and atomic and redis actually let you use regular programming languages to for your store procedures so you can do you know kind of like those constructs that you're used to like loops and maps and stuff which are typically not you know around in uh sql but uh that means you could unit test those store procedures then i didn't think about that when i was reading that part of the book yeah it'd be kind of like that that would make it easier to swap out the table and say use a virtual table or something without having to

like entwine that in all of your every single statement in every line um which sounds pretty nice but it's kind of also like the people who write a lot of sequels sometimes don't necessarily know closure or lua or groovy so there's you know it's kind of it's risky bringing on a different language to do one aspect of your work. So, you know, I'm not too crazy about it, but. Well, I mean. It's good. Yeah, that's a good point.

But when I was reading that part of the book, I just thought that like, well, that maybe SQL wasn't the standard way to like query that. But maybe you are right. Maybe there's like, there's this SQL layer on top. And if you wanted to do something more fancy, then you could do like a Java or groovy to query volt DB, for example. Well, I don't think so. Like with Redis, isn't it mostly just, um, a lot of times if you're doing HTTP type stuff, I don't know what you do for their procs. I've

never looked at it, but I don't know. I mean, the, I think the only thing that you said there, Jay-Z, that I'd almost be like, well, is that really, do people that work in databases, do they only know SQL? Because I mean, if you're working in something like MongoDB, you're not using SQL 90% of the time, right? Like it's, it's more of a declarative type thing. So, you know, I don't necessarily tie maybe 10 years ago, I would have said databases, you're looking at SQL nowadays, you know, depending on the type of data storage you're using. It's JSON. Yeah. It's JSON or, or, or probably God forbid, it'd probably be YAML at some point, but. Well, I mean, to, to, to Jay-Z's point though, where I was thinking what you what

you were talking about was like if we take postgres as the example right like you know you can use a sql like be it uh you know plpg sql or or whatever to query that but also you can do things in c if you wanted to now how many of us are going to break out C compiler to go write, let me write this as an extension into Postgres. If my choices are Groovy, Clojure, Lua, or C, then it's got a shot.

Okay, fair. That's fair. I think I'd do Groovy over C. I think C Sharp you couldovy over C, I think. I think C Sharp you could use to write Proxen for SQL Server. I think I remember that as well. As a matter of fact, they didn't even introduce being able to do things in R and maybe even Python at one point. They did.

Yeah. R came out first in, I think it was 2017, and then in 2019, SQL Server 2019, I believe is the one where they introduced support for running Python. And it was pretty cool where like you could, uh, you know, execute, execute the query and,

and manipulate the results in Python. But you want to, you guys remember the funny part about all that though, is nobody actually allowed it because that means you had to turn on some sort of flag in the database that said that you could run external code. And so database administrators the globe wide were like, nope. Not happening.

Yeah, it's pretty scary. Especially with C-sharp like that, you don't deliver the script you would deliver the DLL. You have to load it into where the SQL server is installed, which is scary. Right. Yep.

So when databases are in memory and the transactions are single-threaded, then SOAR procedures can actually be really fast, especially having memories really quick. And that's one of those things that wasn't really feasible in the 70s because of the limited resources. But with no I.O. networking overhead, the transactions

just kind of run through. We talked about VoltDBv and yeah so now uh fun part uh so we talked about things as if they're running on kind of a single you know like a single processor uh single transaction but there is one trick that we can use here uh that lets us do do a little better in terms of being able to kind of scale out while still also running on a single machine, and that is partitioning. So we've talked about data partitioning before. We've talked about several episodes on partitioning data. But what we're talking about is basically kind of splitting things out. So if you have a multi-tenant solution, you might have a partition by tenant. Or if you have like a multi-tenant solution, you might have like a partition by tenant, or if you have like something with,

you know, IDs or something, maybe you have like one through 100,000 over on this partition, you know, or maybe partition by the first letter of the first name or something, or just something to kind of split up your data in a way that lets you kind of do things, you know.

But this is where like, it comes back to the to the the thing that this book has like just beaten into our heads is that there's all these great capabilities out there partitioning being one of them right but you have to truly know what your use case is going to be so that you could set that partition up the way that you're actually planning to use it because like you just said like okay well you could partition by the first letter of the tenant name, right? So every company that starts with a Z is over in this partition. But we know that like, well, the English language, at least, words aren't distributed evenly across the alphabet.

So you're going to have hotspots on certain letters of that partitioning scheme. So, okay, that already might be problematic, even if you were querying it by that way. But what if the way that you're actually querying the data, the majority of the time is by a username or a tenant ID number or something worse. And so now you might have to like, uh, you know, query, you know,

join two other sets that aren't part of the same thing. Like you, you truly have to know what, what your use case is going to be to make sense of it. Yeah. You get it really wrong. And before we go past this, for anybody that hasn't heard the term hotspotting before,

basically, if you think about it, like, you know, like he just said with the with the alphabet the letter z there's not going to be anything that goes into that partition but let's say the letter s says mr zach this is mr zach my bad okay there's one thing in that partition so right this stuff is always going to be fast, right? But if you're the letter S, right, for some reason, you're behind, you're in line with a million other people, right? So anything that happens there, if you're the millionth person in line, it's got a whole bunch of stuff to go through before it gets to yours. Whereas anytime Joe put something in his, it's getting done immediately.

So that's hotspotting, right? And anytime you're designing a system, especially sharding or transit or partitioning or anything like that, you kind of need to know how your data is distributed so that you sort of get an even load across those partitions. So that's, that's hotspotting in a nutshell. So I was, I bring all that up because as I was reading this part of the book, I was kind of trying to think like, okay, well, what, like, you know, it's 2023. In reality, like, what does this mean? Like, how do you write your partition? How do you partition your data to where you do have single partitions for things?

And maybe the best that I could come up with, and I don't know if this is a good example, but maybe like the keying in Kafka, you know, the deterministic type of keying that you would do in Kafka to make sure that it would go to a, you know, specific broker, specific topic, you know,

within a topic, a given topic, you know, specific broker, specific partition kind of thing. But I wasn't sure if maybe you guys had a better example of like where this single partitioning kind of idea, like to visualize it in the,

in, in the real world. Well, I think, go ahead. Sorry. No, I was going to say,

I think, I think you sort of said it. If you were to envision your partitions is like a, um, a phone directory by, by the first letter of your last name, that's a great, that is a great way to do it. Yeah. But I'm talking about like, let's put this into like practical, like real application use now. Like let's not talk about it in theory, like a phone book, but like what's something in real. And so the closest thing that

I could think of that might, you know, be the fit that what the author is talking about here, I think, would be like Kafka. Right. So, yeah, I mean, I think Kafka does really interesting stuff. Another example, a system that I like for kind of thinking about partitioning is Elastic. So it's common in Elastic sometimes to partition by date. So like every day has a new index and that means if you do a search this is like uh you know give me everything with the keyword red or something then it can search each of those partitions you know like keyword red doesn't really make sense for like sensor data or something like that iot type stuff

that you might see with uh daily partitions but you just got to kind of imagine it goes like search each of those partitions or you know like we're talking about transactions here so it could run a transaction on each partition at the same time and it also buys us the ability to have like a nice retention period so i can only keep the last 30 days of data because you know every 30 days you like kind of lop off in that case you're very susceptible to hot spotting like imagine the weekends are really quiet but the weekdays are hot but there's also elastic has the ability to roll over indexes at certain intervals like size for example so like every 50 gigabytes create a new index and use an alias that points to all of them and when you query it can do two transactions and if you have two part two they call it shards but

it's basically partition so it's like okay so we're running the transaction on this shard and we're running a transaction on this shard and we're running a transaction on this shard. And then we put our data together at the end and we just have something to kind of collate that. I mean, the partitioning by date, we've done that same trick in multiple databases, the three of us. We've, you know, we have been part of teams

where we've done that at the database layer and, you know, is a performant way of like, okay, here's today's worth of data and I want to age data out of the system. So I just dropped this entire partition for the oldest one. But this part of the book felt more, uh, database kind of heavy, like, like traditional SQL kind of database heavy. Cause you know, I guess, I guess the, I, in my brain went, you know, down a very specific path because like all the stored procedure talk. Right. And so that's where I was trying to like wrap my head around, like, what, what would it mean

to partition the data in a single partition like that? Like, I just couldn't come up with a good example for a SQL type database. Yeah traditional type database seal type is kind of throwing me off so i think i don't maybe understand your question but how about this what if we're an airline uh sales ticket sales company and when you're setting up the system it makes sense you have each airline be in its own partition and that way they can go in there they can modify their flights they can do all the stuff that they need to without worrying about affecting others the transactions are all going to work great but we have this one use case where the customer goes to buy a ticket and they search across all and you know they're buying those

tickets and maybe they're doing you know a flight from here to bermuda or you know somewhere somewhere far away and it involves stopovers and so my first flight is with southwest and my next flight is with uh i don't know air hungary or something and now we've got transactions that are going to roll across two different partitions okay i see what you're talking about now but yeah traditional sql i meant like you know your your sql server oracles postgres whatever but in this it's it's very similar to that right like what they're getting out here is i mean let's let's say that each customer was sort of evenly distributed in their data here. Because that's what they're talking about.

And if you partition your data in one of these serializable databases or serial transaction databases, then you're not bound by a single CPU at that point, right? So you have the Outlaw, the Underwood, and the ZAC customer partitions. And then if all three of us have something to come in at the same time, each core of the CPU can handle transactions for, for each of our, our customers. Right. So, but basically have three serial, uh,

yes. Uh, processes per partition is what you're trying to say. You could have one, one process per partition, right? So instead of only being bound by that single CPU thread that you had on the entire database, if you partition

your data the way that they said it is you're bound by the number of cores that you have on your cpu right so if you have a 32 core cpu then you can have 32 partitions and so you can parallelize um you know up to 32 times and maybe if we're talking about like distributed databases, like the Kafka example, then it's, you can horizontally scale the problem, right? The interesting thing that they talk about here is they even suggest you could move in. I think you had mentioned it a little bit earlier. You could move the read only transactions off to its own thread. And then that way your rights are the only thing that you're having to deal with that that you really want to multi-thread but your reads could be because they're fast you could put those on their own off to their you know move those off to

replicas right yeah um so yeah we already talked about the cores and everything so that's good now this so jay-Z mentioned this a minute ago. There can be somewhat of an issue if you have some sort of proc that needs to go across multiple partitions, right? At that point, they said that you sort of have to manage. You have to know how to manage this stuff to make sure that everything sort of stays in sync, which is sort of weird, right? I would imagine that's a place where you can sort of get yourself in trouble in these, but you have to know how to write the stored product to know how to go across these partitions properly. I'm trying to think of an example, like a real world example. Well,

let's say, let's, let's go back to where we've done this type of thing. Uh, us, I don't think we have, but I mean, we haven't actually worked in one of these serially,

um, transactional databases either, but let's say that we had our three customers in there. Right. And all of a sudden, what was it? Southwest that recently had the thing where everything got shut down like they couldn't they couldn't get anybody. December morning it was. Yeah. They couldn't get anybody on a jet. Right.

We need to update all customers and delay their flight three days. Right. To the same flight number, but three days out. All right. So now I've got to update Outlaw, Allen's flight company, and Jay-Z's flight company. So now they're doing that across everyone. So that would be a good example of where typically you're probably not even having to deal with that, but now you've got to go push everybody back three or four days and give them all refunds. So that could be a situation i mean we i know we've definitely done some you know we've definitely been part of teams where we've done some i don't know if you call it nasty tricks

you know we're like you you just go ahead and don't rely on the system determining what the partition is for you you go ahead and calculate that out yourself and uh query it directly so that you can uh skip to the heart of the matter rather than, you know, taking any overhead for letting it figure it out and bottleneck its way through it. But I,

I couldn't come up with like, I was struggling to like, try to find, like think of like real examples that we've done. So we've never worked in a single threaded database either right but no we have though that's the point that's the point that's the point of this book or this chapter is that like this is a thing since 2007 no no but i'm saying we haven't like worked in any one of these databases that forces anything serially right like the no that's the point is which ones have we done

sql server postgres that's the point is that once we've done SQL server, Postgres, that's the point is that like, when it comes down to the writing, that was my takeaway from this part of this chapter of the book was that, that, that pretty much across the industry, they had discovered like, Oh, Hey, if we serialize these rights, we can, uh, you know can get better performance. Oh, so yeah, they mentioned that, but most databases don't do it. Right. So that was kind of like the reason why they called out VoltDB and Redis and the other one is most databases don't do it because of the performance issues.

Or they don't do it by default. So SQL Server supports it if you set the transaction isolation level to serializable, but it's not the default and you'd have to go out of your way to do it. Right, yeah, that's sort of the big deal here is it's almost like most of the ones that do support this are very specialized in that the whole,

like Redis, right it's it's keeping everything in memory i've actually heard of problems where people use redis sort of as their database and if the thing restarts they lose everything right and and it's a i'm sure there's ways around that kind of thing but but that's that's what this whole thing is about is like it's almost a very specialized thing because the reason why it's not more of a normal practice is because it's a big performance hit if not implemented very specifically on hardware that can handle it. Okay. One thing that was interesting here is out of all the databases that they talked about, VaultDB is the only one that they say could handle multiple partitions. So I'm guessing that means Redis isn't in there because I would have thought for sure that they would have mentioned them.

And this is also why I went up and looked up VaultDB because I was like, man, like man they're talking about a lot but i've never heard of this thing but they took a huge hit cross partition rights like it was eight times slower i think was the the number that they say how much slower i could have sworn i looked at a magnitude um so order magnitude so. So if they support 1,000 cross-partition, it's 10,000 normal. That's right. Well, plural orders. I don't know where I got the 8 from. It's that dementia. I swear.

And, hey, like we were saying earlier, this whole thing of trying to find out how many transactions you can support per partition, you have to know your data, and you have to know how things are coming in across those partitions. Otherwise, you could really mess yourself up with this. Yeah, totally. So key value data is likely to do well in single partition transactions. It's a simple lookup.

Data with multiple secondary indexes is probably going to require cross partition transactions. A good little rule of thumb there. Yep. All right. Well, we will have a link to this book and our resources we like. And with that, we'll head into Alan's favorite portion of the show. It's the tip of the week. All right.

And I'm starting here. so i am still on my ai you know whatever you want to call it uh kick so i'm using copilot heavily um grammarly is another thing i like to um it's not i don't know if that's doing ai stuff but i like it it feels like the kind of productivity boost that i'm seeing from copilot but remember copilot's the tool from github that lets you kind of plug into your browser and you can write comments and it'll write the code for you or it'll just kind of suggest stuff almost like autocomplete.

I love it, huge fan. I forget how I found this, but I was just kind of looking around and found an additional extension you can install in VS Code called Copilot Labs, which has some experimental features that you can install and it puts a new sidebar icon

on the left like next year navigation all that other stuff and it has four major windows and they're like just top to bottom when you click on the icon it's kind of the ux is a little bit weird hey that's why it's experimental that's really cool though so the first one is code explanation so what you do is on your window panel like with with the file, but you can select like a block of code, like a function. And it will you click the little button on the left to say explain this and chat GPT or sorry, GPT three, whatever Copilot uses is going to summarize that. And the way it does it is pretty well. It is pretty good. It does it like a bullet format.

It doesn't do it in a paragraph, which I really appreciate. And I did a couple of examples just to kind of try it out. And it did really well. One of the examples I used that I just tweeted about actually on the KillingBuzz account was I had some code in Unity that was figuring out what the normal vector was, which basically just means like normal. It's hard to explain. So, it kind of feels like if you're standing on an angle, then the normal is 90 degrees to the plane that you're standing on.

Which is very easy to see and kind of hard to explain in words. But anyway, it's kind of a mathematical function where it's like if it's between these degrees and these degrees and you get the tangent from the angle. And it very quickly figured out that it was just getting the normal and kind of mentioned that. So that was nice, kind of a cool example.

And it also called out specific things. Like there were some build flags and stuff. So what do you call it? Like preprocessing directives in the code. So what do you call it? Preprocessing directives in the code. So I called those out separately in a different bullet point. And it was just really nice. So you can imagine if you've got a function,

either that you are just having a hard time reading, you can kind of do that and just get a nice little summary of what it does. Or it's kind of a cool way to check what you just did. If you've got a function that you just wrote that's like a little tricky like you could maybe try explaining it and see if like it explains it in a way that you know was what you intended uh which is pretty cool uh live rubber ducking live rubber ducking it's rubber ducking talking back to you uh so uh next one is code translation so if you want to take that block you just select it and convert it to another language really great for reading stuff so if like you've got some stuff in like python you're like not really sure what some of those functions do and you don't really want to go look

them up and you don't want to get it wrong try converting it to a language that you're more familiar with it's pretty cool um the next one sounds really weird but in practice it works out really easily they call it IDE brushes. And when they say brushes, they mean brushes like you would use in like a Photoshop or a paint or something. And what you do here is you select the brush you want to use. For example, add types. If you're working in TypeScript and you've got some code that's using any, and it can figure out the types from the functions calls in some cases so what you do is you select the brush for add types and you paint with it over

in your window and it will automatically apply or suggest the uh types for you well that's when you hit tabs except yeah it's pretty nice and it's also got things for fixed bugs so you can like say hey fix bugs with my paintbrush here, and then you kind of paint in the file, and it will fix bugs for you. It also has brushes for improving readability, resilience, documentation, with JS docs type thing. It will document the function above it. And there's also a way to add custom brushes.

I didn't see how you could actually write those custom brushes, but you can imagine how that might be a kind of cool thing where like maybe you've got some custom tools that you want to like implement at your workplace to say like make sure this code isn't from stack overflow or you know something and you just kind of paint the file that's really neat and then the last one is also really cool uh test generation so select a couple blocks of code and it will generate a test for you on the left. That only works in JavaScript and TypeScript right now. So you can definitely tell what kind of demographic they're going after several things kind of leaned heavily towards JavaScript and TypeScript.

All of them are cool. It's totally free if you're already using Copilot, which is only $10 a month. And if you're not using Copilot or you haven't tried it, super recommend it. It's a great productivity boost. And the thing I like most about it is that you don't really have to change your workflow you don't have to learn some big crazy thing you just install the extension uh click authorize and it'll take you to github to you know do the rest like enter your payment whatever and kind of set it up and aside from that like it just starts suggesting stuff and so if you want to kind of take and go further and write comments and kind of explore some of the other ways you can interact with it cool but you're going to start

getting that benefit immediately after installing it just going to start suggesting stuff for you you hit tab to take it i actually had somebody mention this to me the other day like they just bought it because in their mind they were like at 10 bucks a month if it saves me an hour it was worth it yeah right i mean that's that's a real easy way to look at it and i assume jay-z you've been using this for a while this probably saved you more than that right yeah absolutely and you know what um aside from actual like tangible time that it saves it saves context switching uh i wrote a little script the other day where like i you know got a list of helm list of Helm charts and I did some awking to kind of extract some stuff.

And then I did a little loop to go through and then do an X args to perform it. So at the end of the day, it's very easy to say, hey, I want to get all the Helm charts, massing them, pattern, and delete them. But to do that, it was like four or five bash kind of constructs or programs that I don't really use that often. I have little flags and stuff that need to be entered and that's stuff i would probably googled you know at least half of those like even just the while loop at bash i probably would have googled for to make sure that i got the syntax right and stuff and i didn't have to do any of that it just got it right the first time i just had to test it you know could easily see and read that code but it just saved me from having to go out and google a bunch of times and just kept my focus there on

the single pane i was already working in so whether or not it saved me an hour like no probably saved me 10 minutes but it felt really good and helped me feel like i didn't lose the thread on the task yeah you can move on to the next thing quicker sounds like intellisense plus plus right totally totally so i think it's i think if you are not trying it out you're doing yourself a disservice like maybe hate it you know fine but i do think that like fast forward five years from now i think everyone's gonna be doing it so as we'll hop on so in the meantime uh Jay-Z shared with us what a crumpet is. And is it just me looking at the picture of it on Wikipedia? It kind of looked like an English muffin, the way it had all the little very porous top about it.

Yeah, that makes me super love crumpets. Some kind of bread thing. English muffins may be one of my favorite things on this planet said no one ever no dude are you kidding me the only sad part is they've gone up 50 at costco which really irritates me i really jay-z are you a fan of english muffins is it my alone you know if i have any other choice i'm gonna take the other choice but right now i'm hungry so it sounds pretty good. Are you kidding me?

You toast an English muffin and all those little holes are like little carriers for butter. They're amazing. But my choices are like a pancake, a waffle. Well, that's different. That's not fair. You went sweet to bread. It's just at the bottom of the bread bucket for me. Yeah, it was always the bottom of the bread bucket for me. Yeah, it was always the bottom of the bread bucket for me.

And your butter comment didn't help any because nowadays, Michael today versus Michael years past, I eat so, my menu of foods is so dramatically different now than what it used to be. Yeah. I'm not allowed to eat a bunch of butter. It's probably why I don't have any English muffins in the house right now. Well,

first of all, I'm like all the carbs of the muffin and then the butter, like that sounds like a good way to just stop your heart from beating. Oh, it's so good. I mean, you got to die somehow, right?

Death by butter. Doesn't sound like the way to go. Oh, man, sounds great. But yeah, OK. So all right. So for my tip of the week, we got this one in as an email and I thought I would share it. So Andrew wrote in. He told us of Alt Tab.

So have you ever wanted a Windows like alt tab experience on your mac though yes so you know how like on windows if you alt tab to something you'll see first of all uh let's say let's say that you had two chrome windows open right right? In Mac, you, when you command tab between it, you'll see the, just the application Chrome, just the one windows, you'll see it. You'll see like, Oh, Hey, here's what was on this window of Chrome. Here's all this other window of Chrome. And for, you know, and it could be like word docs or Excel docs, whatever, instead of seeing the one application, you see a picture, like you can literally see like what that page was doing, you know, in the case of like a Chrome thing, right?

Like you see like a miniature screenshot of it. So alt tab is a utility that you can install under Mac OS. And I think it goes back to like, yeah, 10.12 and newer, right? So it goes pretty far back. And you can brew install this if you wanted to. And it brings in that capability to your Mac. So now caveat this, I will say the one thing that made me super cautious is that there's like, you got sometimes you love and you also hate when now this security-minded world that we live in

where Apple and Google, especially at least on mobile devices, but even Apple on their laptops, will like, hey, do you want to give this app this one very specific permission? And you're like, whoa, that sounds awful. No.

So in order to do this functionality, it needs to be able to – the way the operating system refers to it, like when it asks you, like, hey, do you want to allow this, is it refers to it as screen recording. Right. So you're like, well, gosh, no. Not when you put it like that. I don't want to do that. So that was the only thing that gave me pause. So I'm throwing that out there in fairness that maybe that doesn't mean anything to you.

And you're like, whatever, I'm going to go on. I'll do it anyways. It definitely gave me pause. But I am super curious to know like how widely used, you know, if, if a lot of other people were like, no,

no, no, this thing's amazing. We've, we use it in our company or whatever. And so. Hey, but check this out.

So on that, I think I know why they have to do the screen recording because just like in windows, if you all tab, it's going to show you that shot. Yeah. That's what I said. Yeah.

Yeah. But they did at least on the page now granted i don't know if you can actually they've got a github project so maybe you could go look at the source code but they have a privacy section directly on the page that you link to that says they don't send or receive any data without explicit user consent and basically only on crashes right like hey you want to send us a crash report so this is the honor system right right um which is i also get like i don't necessarily always um believe the honor system and so this is yeah this is where like oh man like i'm so uh i'm such a skeptic because like even even if even if they are doing

the honorable thing like let's just let's start from a positive place, right? Because this looks amazing. Like you look at the screenshots of what this thing is doing, like you super want to use this. Yeah. And by me, and I bring this up as my tip of the week

only because I'm hoping to find like other people also, like I want to hear like other people's usage of it and everything. But the problem I have, not just with this software, but like anything like this, is this like, well, that's fine today, but who's maintaining it that I know that it's going to also be the same tomorrow.

So like when it's the big back projects like Apache or, you know, backed by Google or Microsoft, then you're like, well, okay, there's like a big player that's maintaining that thing. But the smaller ones, it's just hard when you have to give up some pretty serious, you know, when you have to give up some serious permissions that you're like, okay, well, I just hope that somebody doesn't submit a PR three months from now. But that's the skeptic in me.

So, yeah. So, I don't know't know yeah it's impossible man i'm right there with you i anytime something asks for permissions i'm like no sorry but yeah so if you're using this thing like right in now because that one's not a complaint i use a different slack handle so at michael and you can complaints use my my complaint complaint Slack handle at Joe, but at Michael for this one. And yeah, cause I would love to know like, you know, other people's use of it. You know what, you know, how it's worked out for them, especially like if it's, you know, approved it like company wise, like that'd be awesome to know too. So yeah.

Cause it looks amazing. It's got, it's, it's very full featured. It's yeah. So we'll, there'll be a link to that in the, in it looks amazing. It's very full-featured. So there'll be a link to that in the show notes, and thank you, Andrew, for the submission. Yeah, and for any lukewarm feedback, you can add Alan. So I know Jay-Z has all, like, I've seen him work, and he is a heavy user of Term inside visual studio code. Yes. And, and I've, I've always sort of done it sometimes. And I just like, I don't know if I forget about it or I just like, I term so much better on Mac that I just have always gravitated towards, you know, doing my terminal stuff in that. But there is a super compelling reason to use terminal in Visual Studio

code. And this isn't anything that I found on the internet. It's just, I happened upon it the other day and I was like, okay, this might've done it. If you do any get commands or anything in, in terminal, like a get status, right. To see a list of the files that have changed the way that I used to do it is in iTerm, I'd see that list, I'd copy the link to the file. And then I'd be like, all right, let me go open this up in code. If you do that in code in the terminal down there, you can command or control click the actual file and it'll open it up in an editor in Visual Studio Code. So you don't have to take any extra steps. Like you're right there. It's super easy. So that is one that I love. Like it's just a convenience factor that saves me 15 seconds,

right? So love that. I do it so much during the day. Thought that was worth calling out. And then the other one, I don't know that we recommended this as a tip of the week. I think it was mentioned in passing that, you know, with all the last pass stuff that's happened, I think outlaw has, has done a lot of reading on it. I haven't paid much attention to it. Um, I know Jay-Z, you also looked into a bunch of it and what was happening. Um, I think so Jay-Z moved over to Bitwarden. I've been using Bitwarden, but for a different reason, I didn't want to pay for a multi-user last pass thing. And in some situations I wanted to sort of keep things separate. And so I had them both installed side by side and I can say using Bitwarden, I really liked the UI

and I really liked some of the stuff that they've done. And there hasn't currently been any leaks or anything, but the thing that might be pushing me over the top to switching is LastPass has basically jacked the price from $12 a year to $36 a year over the past several years. And it just seems to keep going up bit warden is 10 bucks a year for a single user so it's less than a third of what last pass is and you can get a family version of it for up to six users for 40 a year so i'm thinking about maybe doing the the whole switch and they've actually in maybe last pass even has some of these features. I don't know. But Bitwarden had something that was super cool.

If I can find it again. I don't, I don't remember exactly what it was, but at the bottom of their page for their personal thing, they had a comparison of the different ones and it wasn't security report man doggone it i can't find it there was something super cool on it and i can't remember exactly what it was but it was like a feature i'd never even thought about that i was like oh man that's a that's pretty amazing so i can't it's really easy to switch to you could just export from my last pass and then

you import to bitward and it's like hey is this last pass check yeah i mean that's that's absolutely beautiful oh emergency access this was what it was so but my wife and i have actually talked about so if you've ever had anybody have a major family event go on, like, you know, somebody passes away, you know, something super bad, it can be really difficult to try and help get them into a good state. Like how did they get to their banking? What, what are all the bills they have? How do you do this kind of stuff? And it can be, especially if they're not using a password manager and things are written down all over the place, like trying to get to where you can help them out is really difficult. And so we've talked about it because I have a lot in like almost all my stuff in LastPass. What if I were to get sick or me and

Reagan were to, you know, die in a car crash or something, right? Something awful. How could somebody get to this? They have this feature in here called emergency access to where you could actually designate another Bitwarden user to be able to request access to your thing in the case of some sort of emergency. So like that to me is like pretty... LastPass has the same feature. Yeah, that's what I was saying. I didn't know if they had or not. I'd never even thought about it. You can set it up. But when I saw this what I was saying. I didn't know if they had or not. I'd never even thought about it. You can set it up. But when I saw this.

I'm sorry. No, you're good. It's just when I saw this, I was like, oh, that's really interesting. It's something I've thought about, but I'd never really looked into if there was any implementation behind it. Yeah. So that family plan for LastPass versus Bitwarden, it was like $48 a year for LastPass, I think, and $40 for Bitwarden. So Bitwarden was definitely cheaper. But they both have the family access. And the way it works, at least on the LastPass side, is you can set up like, hey, here are the people. And what happens

is that as a security measure, you can set what the timeframe is that you want to have expire before they can get the access. So they'll make their request and then last pass will give you as the, as the account holder, a period of time to say, no, no, no, I'm still around. Don't give them that access. Right. And, and, and you can set it up to like um i think the default is like three days like you can say like hey wait i you give me three days to respond beforehand and that way like you know in the case of a death for example you know your family member would have to wait three days before they would be able to get into your accounts oh that's actually really interesting that's that's good to know how it works.

So here's what's interesting on the pricing, and this is what's really annoying. So LastPass is $3 a month per person for the premium, and if you're sharing or anything, you have to have that. So you're $36 a year for one person. Well, you can share free. Did they change that? It used to be you couldn't. No, here, i'll share a link

oh they have free sharing on the free tier so but but you're at 36 bucks a year for one user but then for families it's only four dollars a month so it's only a buck more a month which makes i mean that's actually a good deal bit warden is 10 bucks a month for the premium you can do the free one which is actually pretty good too, or you can go to $40 a year, which is still a little bit less, but not significantly so than LastPass. But at any rate, I have been using it, and it is really good. And Jay-Z, I think you like it as well.

Yeah, it's great. The one thing where LastPass gets you, and I've never understood this, is that their free thing is for like one device type. So, you know, it's pretty fully featured as long as that's going to be the device type you're going to use. So in other words,

like if your device type is going to be, you know, your computer and, and so you're going to use it as like a browser application or whatever. Yeah. Do you remember when it was a separate app? Do you remember that? I do. I do you remember that i do i do so last past vault yeah um but you know if but then you you're limited in what you could do on like ios for example or or android right like you lose that integration capabilities

and that's where you like you want they they try to force you into that premium but i just i don't understand why there's like such a jump from the free to the premium in comparison. Cause I agree. Like, you know, especially if you, if, if your family like, well, you know, go ahead and jump to the family. Like you'd be silly not to. Yeah. But like you said, the, the fact that last pass made it incredibly difficult to use on your computer and then on your phone, like you can't, it's basically like, Hey, you don't have the premium thing. You can't auto fill on, on iOS, or you can't auto fill on, on your computer. Well, you pick one, right?

And it's like, man, you used to be a buck a month and that was reasonable. Right. Um, but going to three bucks a month, which is still not a whole bunch of money, but the fact that they tripled it on you in the time that I've used it, Bitwarden is looking way more appealing now. So, well, in the time that I've used it, it was originally free,

right? And everything like it just did everything free, but you know, that was years ago. So like a lot of capabilities been added since. Right. But, um,

yeah. Yeah. So at any rate, um, if you are looking looking if you don't use a password manager first do it right get the free version of bitwarden or last pass and just use it please um but second if you need that extra functionality uh the bitwarden one is actually really nice yep and so uh with that i will leave you with this uh lasting deep thought for the show do you think the ocean is salty because the land never waves back i like that hey there was there was another one that i heard uh recently uh oh yeah

i'm not gonna do it well so i'm gonna skip it right now i'll get it for the next episode recently. Oh, yeah. I'm not going to do it well, so I'm going to skip it right now. I'll get it for the next episode. What? Yeah, I can't. I'm going to mess it up, and I can't mess this one up because it's actually pretty fun. Wait, you shouldn't do things and you're going to mess them up? That's right. Somebody should have told us 207

episodes ago. Okay, fine. Then I'll fill in the blanks for Alan. Why do photons always go on vacation without any luggage? Because they're traveling light. That's really good. Last but not least, can trees poop? I mean, my back porch says yes. A car says yes. It's something give me the answer

of course how else do we get number two pencils thank you mike rg for all of those um all right so with that uh subscribe to us on stitchify uh itunes you know wherever you'd like to find your podcasts. Wow. Just real quick, I just want to interject. The slendity of the ocean is not related to the behavior of the shore, but rather to the natural processes involving the erosion of weathering of rocks on land.

Thank you for that. ChatGPT did not get my job. Not understanding the joke. Thank you, ChatGPT. ChatGPT. Hey, while you're Yeah. Not understanding the joke. Thank you, chatgpt. Chatjizzy. Hey, while you're up there at the website, codingblocks.net, make sure you check out our show notes, examples, discussions, and more. And send your feedback, questions, and rants to our Slack channel at codingblocks.net slash Slack.

Yeah, and hey, follow us on Twitter at codingblocks, where I just tweeted a minute ago. And head over to codingblocks.net and you'll find all our social links at the top of the page.