Coding Blocks - Designing Data-Intensive Applications – Multi-Leader Replication

Episode Date: June 21, 2021

We continue our discussion of Designing Data-Intensive Applications, this time focusing on multi-leader replication, while Joe is seriously tired, and Allen is on to Michael's shenanigans.

You're listening to Coding Blocks, episode 161. Subscribe to us on iTunes, Spotify, Stitcher, wherever you like to find your podcasts. And leave us a review if you can, because we love those. And you can check us out at Coding Blocks. And then we can find show notes, examples, discussion, and a whole lot more. And hey, we have email addresses and stuff that you can find there. You can ping us. I don't know that you can, but you can follow us on Twitter at CodingBlocks or you can head to www.codingblocks.net and find all our social links there at the top of the page.

Sans email addresses. With that, I'm Alan Underwood. I'm poorly prepared. And I'm Michael Outlaw. This episode is sponsored by educative.io learn in demand tech skills without scrubbing through videos whether you're just beginning your developer career preparing for an interview or just looking to grow your skill set so last episode we talked about replication and we mentioned that we are strictly talking about data that is small enough to fit on a single machine. So each replica has all the data.

And we talked about the reasons why you would want replication, like fault failure, tolerance, scalability, geolocation, getting things closer to make them more performant. And we talked about three old from the 70s techniques for replication, single, multi, and leaderless. All right, now, and leaderless. All right, now you're caught up. So stand alone. And old didn't mean they're not used. It just means they're old.

Yes. They're literally very old. Well, I think what we discovered was that everything that was ever good about computing was developed in the 70s. And 50 years later, we're figuring out like, oh, that's pretty cool stuff that they did back then they just didn't have the horsepower to run it they could get to the moon but they couldn't do like you know anything across the internet you know what that reminds me of you

remember when everybody was like oh man cell phones are gonna be the next big thing for like the internet right like it's gonna be big but for years and years and years, it just wasn't there. And then, and then it, then it was, they were all right, you know, 20 years prior. So yeah. Yeah. Good times. Yeah. Everything's been like all algorithms come from like 1980 or before and the rest has just been integrating with advertisements. Yeah. Basically the rest has been figuring out how to turn off tracking. When you figure that out, could you let me know? That'd be,

yeah, no kidding. No kidding. All right. So this episode, say what? This episode, we're going to talk about problems with replication lag,

and we're going to talk about multi leader replication, which is the good stuff. Cool. All right. Well, how about first we say thank you to everybody who left us some reviews and i'm gonna let alan take this first batch oh look at this see it's so great when you do the intro for the the reason you don't have to be the guy to say some of the names right like oh that worked oh, that worked out perfect. Yeah, we got lazy this time. We didn't put initials by anything.

So, yeah, I've got, we're going to say Google Reed here, T. Bednarek, JJ Henasia, and Katie Crossing. And some excellent, excellent reviews there. So, thank you very much for those. And from Audible, we have anonymous and anonymous. Squared. And one of those anonymous has said that they've been with us since day one. That's been a lot of days. I'm so sorry. Yeah. I call shenanigans because I'm like, no way, nobody would still be listening to us after all these years. Surely you would have given up, right? Like we're, we're approaching a decade, right?

Like that's, that's crazy. Can you imagine we're going to have to do something special when we, when we cross the 10 year mark? Yeah. Um, I'd imagined it'll involve walkers and canes and stuff. Well, obviously, obviously, but I meant something else like because because we're like barely uh like less than a year and a half away right if i remember right it's 2013 yeah 2013 yeah it was september of 2013 if i recall or maybe we're closer to two years sorry we're just over two years right at two years yes that's why you got to leave that to the math of my chicken. Yeah. All right. So with that,

I don't think we had any other news, right? Not this go around. Nope. All right, cool. Just thanks. Yeah.

So let's dive on in. Cause we got a lot of notes. So buckle up. So first we wanted to kind of continue on with a part from, it was kind of from the last section, but it really applies to all replication strategies. And we want to talk about sync and async rights now that we know a little bit more about things. And we're going to be doing a little bit of review here. So don't worry if you missed the last one.

But you should go back and listen to it. Yeah, don't worry if you miss it, but just go listen to it. Download it at least. I don't want you to worry. No stress needed here. Yeah, I just need to it. Yeah. Don't worry if you miss it, but just go listen to it. Just download it at least. I don't want you to worry. Don't don't no stress needed here. Yeah. I just need to see the numbers. Right.

But what are we talking about? Single or multi-liter replication? Remember that we said all rights go through the leaders. So if your application is read heavy, then you can add multiple followers to increase your scalability, right? That doesn't work well with synchronous writes. Remember, that's where you have multiple followers and the leader makes sure that every follower has the data before it returns to the client and says, I got it. So obviously, the more followers you have, the more latency you have for your rights, the more options you have for your reads.

Yeah, because what you're saying is if you have one leader and you have 10 followers, that one leader can't return back and say, yo, I wrote this record until it knows that all 10 of the other followers had that same record written. So it has to wait for it all to finish. That's the safest way to do it. But, you know, like I mentioned, it increases latency. Also, what if one goes down? Yeah, I was about to say, safest but also the most error prone. Right. Dangerous, right?

You could actually bring your whole system down. Yeah. More dangerous way to do it. So that's the problem with synchronous writes. And then if you allow asynchronous writes, whereas the client says, hey, update this or insert this and leader says, OK, we basically got it. Don't you know, don't worry about it. Then the problem is the data can be scaled. So the client come back and say, OK, fine, well, here's a query.

And that query goes to one of the replicas. It doesn't have the data yet. And so it returns incorrect data. This is much faster. But the problem is that you can see inconsistencies and uh there are certain use cases where this is like more prominent and uh well i think we got some examples coming up here we call this eventual consistency so if you've heard that this is a term that was associated for a long time with no SQL, but it's a problem anywhere you've got replicas.

Yeah. One of the things that they called out here that I actually liked is they said, eventual, they kept purposely vague because they didn't want you to think that there was some sort of time associated with it. Right? Like if,

if you do your right is eventually going to be consistent. Is that a second from now? Five seconds. It could be an hour from now. Right. Depending on if there were network problems or anything in between the, the followers. is eventually going to be consistent. Is that a second from now? Five seconds? It could be an hour from now, right? Depending on if there were network problems or anything in between the followers. So eventual is truly just,

I don't know, sometime in the future. You'll get it when you get it. Right. Yep. And like I mentioned, some use cases can be particularly bad. The example I was trying to remember was like updating user information.

So if you're trying to update your, say your address and then check out, well, wouldn't it stink if you update your address and went to check out and the address wasn't updated uh things like that can be confusing the user goes back and tries to update the address and now it's there what the heck i hate the site crappy programmers or worse you you you think you've updated your address you then you place that order and then realize that the address still shows the old address after you place the order. Yep, and all that stuff, it happens. You should know what you just said is interesting, though.

So there's one thing for the user to see something that is misleading, you totally could have a system that when that order went through the, the fulfillment phase, if it was pulling from an out of date, uh, replica, now you've got major problems, right? So like we were talking about what you experienced as a user,

but man, if the, if the other systems aren't following some sort of consistency, um, purpose, you could have a really bad setup on your hands. I mean, this, this is kind of, Other systems aren't following some sort of consistency purpose. You could have a really bad setup on your hands. I mean, this is kind of, you know, tangent alert. Here we go.

First one of the night. But that's where like that type of use case, specifically to e-commerce type transactions, depending on the size of the organization and the amount of traffic and whatnot, where it could warrant that, um, the, the order itself, like there's no like foreign key reference to an address. Instead, it just has a copy, you know, like you could, you could envision like, you know, if you were to think about this in like a relational database, kind of like, or just a table kind of schema, what that might look like, like that, you know, that row might have everything, you know, for that needed for that, you know, payment information, shipping and billing all there instead of foreign keyed off

to something. And then that way, you know, whatever it was that you displayed on the screen at the time is what got written to the ordering system, you know, maybe, but, but they did, I mean, speaking of e-commerce systems though, um, later in this chapter, there were specifically Amazon was using as an example of where they had talked about, uh, like updates to the carts update to the shopping cart, not always, uh, reflecting the change depending on like, you know, merge conflicts and eventual.

Yeah, that's a problem. Especially if you're on Amazon scale and you're selling however many orders per minute or second or whatever. Even a small percentage of failures like that is a big, big problem. The big takeaway for me from this book was that it's important to know what your use case is and what your storage mechanism is, because then based on those kinds of things, then you could make the decision that like, oh, hey, you know what? I'm going to remove an item from the shopping cart. And, you know, until you actually go to, you know, it's okay if it got like mistakenly added back,ly added back as long as you hadn't placed the order. If you're still in the shopping cart mode, that might be acceptable that it get added back as part of a merge conflict thing, which we're getting kind of ahead.

But there might be other situations where something like that would be totally unacceptable. And so knowing those, uh, the limitations of that, that storage architecture and its capabilities, especially as it relates to replication, um, you know, it's, it's, it's really important to, to be able to fit that within what you're trying to accomplish. And there's no perfect solutions. Yeah. So that's why it's really particularly important to know what you got. And so there's a couple of ways of mitigating this. So luckily we don't see these kinds of problems too often anymore. And that's because organizations have figured out how to kind of partially solve these problems.

One interesting way of doing it is to read your rights. So we call it the read your right consistency and the deal is that you attempt to read significant data from the leader or uh only in sync replicas that you know have your data the one that the data that you wrote the data that you just wrote right so yeah that this only helps you in some scenarios like right example if you know it's your address that you just updated but if you're a bank account and someone else is withdrawing that doesn't help you there that's someone else's rights but if you know that you're the only one like writing to this data like the address example then uh you can basically make sure that you only read from uh the leader which always has an or a replica that has your changes and

we'll talk about how you tell that for replica has your changes in a second in general the idea is to make sure that any uh significant changes that you made to the data can be read back so for example you could uh query the data back to make what wait am i getting ahead well how does any personal notes so soon so to to kind of go back to what he's saying let's say that you update i'm trying to think of something um let's say that there's just some sort of product in your database right let's just go the e-commerce route and and you update the description on it when you go back and you view that product it route. And you update the description on it. When you go back and you view that product, it'll know that you're the one that wrote that change.

And so it's going to go to the same place to bring that record back. However, if Jay-Z or Outlaw goes and looks at that same product, they're not guaranteed to get that same record back from the same place where it was written. And so they may have a stale version of the data. So that's what he's talking about with that. Read your rights thing is I get back what I wrote from the same place, but other people may get something else. I want to, well, I kind of want to say this in a different way.

Um, it, it, it's basically like picture. You have a three server set up for your database replication or whatever. I don't even want to call it database because I don't want you to think relational. Whatever your storage mechanism is, you have three servers that are getting the data replicated among them. It's basically like an affinity.

Like when I make a connection, if I make my connection to server A and that's where I'm reading and writing my data to, when I write my data to A, I read it back from A. But Alan and Joe, while they're still, you know, while I'm working on it, they might be reading and writing from servers B and C at the same time. So they might not yet see my change until it gets replicated over. Correct. But I have my session kind of has this sticky session affinity to, you know, server a,

in that example. Yep. And, and so they mentioned a couple of ways to do this, right? And this is, this was one that was kind of interesting that they were talking about is you can read important data

from a particular server. And what they said is that's kind of very user specific. So an example they gave is if you are modifying your user profile on a website, you know, maybe it's Facebook or something, they know that you're the only person that can modify that profile. And so when you go retrieve that profile information, you'll always get it back from the leader, right? So that was like a way of prioritizing important data based off who you are and what you're capable of doing. Yeah. Another way is to basically after you make an important change that only to do that you would be changing you can read from a leader or a replica that you know has that particular change for some period of time we call it five minutes i'm gonna just stick with this one because i know that it got

my change because i just read right back and verified i got some sort of checks on or something that let me know that this replica has it so this is who I'm sticking with and talking to until I'm pretty sure that things have gone and got around. I just thought of an example that's going to like totally kind of derail things that I'm like dreading giving, but I just go for it. I think we've, I think we've kind of talked about this before though,

but this totally isn't what, uh, Martin Kleppman was the author of the book was going after when in this particular section but when it comes to like the the important data like one thing that came to mind as you were describing that was like well it could also be like data that like real-time data versus not real-time data and so like the in my mind where i went with it was like, okay, building your Elasticsearch index, that's kind of costly, right? Like updates to that thing are not easy. It comes at an expense, right?

But you're willing to pay that expense because Elasticsearch searching, you know, it's indexed in a way that makes searching fast. So you're willing to pay that cost, right? But there might be other data that's like more real time and you don't want to take the hit to have it in that index. And so the example that was coming to mind was like going back to your product example was like, Oh, well you could easily have all your product description and attributes in your elastic index. And that way you could search it. And when you build, when you render the product page, you know, you could have a bulk of that data coming back from, uh, the elastic

search index, but you might also want to go and query a relational database to get like, Hey, what's the current quantity and what's the current price? Because those two things could fluctuate in real time. And, you know, you might want to be able to display like, Hey, we have this in stock, or maybe you want to like give your customer some kind of sense of urgency and say like, hey, we have this in stock. Or maybe you want to give your customer some kind of sense of urgency and say, hey, there's only five more left in stock. So it's totally not where he was going with this. But the idea of reading, quote, important data from one particular place is what made me think of that example? Well, you know, one place where I've actually seen this happen, where you get different data back on even different page refreshes is Reddit used to be really bad about this. Oh, yeah.

If you submitted something, you'd see a certain number of upvotes. And if you refresh the page, then it might be way higher or way lower. And it's like, wait a second, how'd that happen? If you refresh again, then it'd be a different number it's like no look i know that five people didn't just do this within the past five milliseconds you know like that didn't happen even better than that like you'll you go to reddit you submit your post and then you refresh the page and your post isn't there you're like i know i did it yeah yeah wait what happened so these are real problems right like they're they're not easy.

Yeah, so the read your rights strategy would be an example where we say, you know what, the votes, that's other people doing that. We don't care about that. People are going to see some fluctuation. But if you submit data, then we want to make sure that after you submit, you always read from the leader or you only read from an NSYNC replica. So you don't end up double submitting or getting confusing. So that might be an example where we might say this data comes from one source and this data comes from another. Yep.

Another one that we had here was the client can keep a timestamp of its most recent write. And then that way, when they go to read this back, they can check that against the data that's on the server or whatever to make sure that they're getting something back. And I'm sure that you guys probably remember this from like SQL server. You have the, the row, uh, man, what's it called? Um, what was it? Row version. Row version. Yeah. It's actually a way when you're not talking about replication, it's a way to make sure that you're not writing. Um, you're not, you got data back and then you're trying to update it, but you don't want to update with stale data. So you can check that row version to make sure that you're not trying to write with an old thing that you had. Um, but this, this is very similar, right? Like that whole notion is

you have two pieces of data, one that's, you know and then the other one that's the other one that you can sort of verify that, hey, I'm at least getting something as new as what I thought I was. You know, it's funny. You kind of hinted at something else a little bit more. I know it's all skip ahead. But they call it monotonic reads, which like the Reddit example, you refresh, you see five. You refresh, you see 100. You refresh, you see seven. It's jumping up and down.

It's a really bad user experience. What if we use something like a row version and we got that row version back and said, if we see a read that's older than the highest row version I've seen so far, just use the one I've seen before. So you're doing some sort of session caching. And that's a way of making sure that the number only ever goes up so instead of going five and then 107 it would go five and 100 and then when it sees the seven it's like whoa this is an old read so i'm going to stick with 100 because it's a

better user experience neither one's right you know what's funny you know what's funny about that is is that sounds good but then we will talk about in a little while where that also falls apart. Yes. It's unreal. Like these are really hard problems. Anything to do with the timestamp though is immediately like, as I was reading it, I was like, Oh,

I could think like, that's so easy to like make fall apart. Right. You know, like I just changed the time on my clock, you know, like, or,

or what happens if, you know, maybe I'm not doing it to be malicious, but it's just like legitimately like uh daylight savings times for my time zone kicks in or something like that and now it's like all the problems that we've discussed about like the server time and my time and you know it has to know my time zone as well and like just it's just it's just, it's just problematic. And then, and then it's like, okay, well, what if I'm on multiple devices? Like, you know, and you know, who's to guarantee that those devices are synced up time-wise.

Right. I mean, they're not mostly, you know, there was definitely a period of time where like, yeah, maybe your phone would recognize that you had gone to a new location. And so it would update the clock, but your laptop didn't used to. Right. So you got off the plane and your phone's like, hey, local time is this, but your computer's like back at home.

So speaking of which, you just hit on, what if you're on multiple devices, right? So you're on your laptop and you update something on the site and then you get on your phone and you do it. Yep. So did we see your user and always make sure that user reads from the same replica i do that all the time by the way yeah i do too honestly like like important emails i'm like okay let me make sure you know if i can see that if i can see it in the sent uh box on my phone then my OCD will let me go to sleep knowing that I sent the email.

Yeah. Yep. Or I'm sorry, my CDO, I had it out of order. You know, um, you mentioned the clock thing too.

Um, when we say clocks specifically, like a lot of times in this book, when they use the word clock, they could mean like clock on the wall, like time clock, or they could also mean like a logical clock which could be something like the number of reads or number of rights that have happened ever in the

history of the database so it could be like we call it a clock but it could just be like some sort of number that means something else yeah um consistent prefix reads this is another uh the name is always confusing to me but the the idea is if you think about causal data, so something happens, something else happens because of that first thing. So like an order shipping. You can't ship an order until the order has been placed, right? And the order maybe can't be placed until the user has created an account. Those are three things that have to happen in order. In that case, they happen at least minutes apart in practice, but you can imagine other things that have to happen in order. In that case, they happen, you know, at least minutes apart in practice.

But you can imagine, you know, other things that happen a little sooner. What happens if your replica gets information about the shipping before it gets information about the order being placed? Right. And this goes back to what I was saying falls apart on the one that you were saying earlier. It's like, hey, what if I just check to make sure I'm always incrementing, you know, my number, my timestamp, whatever. The problem is exactly what you're just saying. Uh, you know, order shipped. Okay. So that's

newer than what I had before. But, but now when I get order, you know, order placed comes in after it, that's an older record. So it's going to be like, no order ship. That's it. So it's, it's just a weird state that you can get into where it's like, well, you couldn't have shipped the order before you placed it. So why did I not get the notification that the order was placed? Like it's just a weird state that you can get into where it's like, well, you couldn't have shipped the order for you placed it. So why did I not get the notification that the order was placed? Like it's, it's hard. Well, unless,

unless you sign up for the Amazon yesterday service. Well, well, that used to be prime. Now it's Amazon next week. So check out this idea. So, uh, the idea of consistent prefix reads, the idea is that essentially that you partition the data.

So you could say, for example, all order information from a single user always gets routed to the same replica first or the same partition first. And that way it's impossible for a reader to get one of those items without the others, because we know that all information is always going to some replica or is always being read from the same replica.

And this is something that Kafka makes big use of with their keying strategies. And so they'll basically make sure that information that is keyed the same way always ends up in the same partition so that you're always reading things in order that they happened. And that's basically the idea here. And from our experience, like this is just real world stuff that we've run into. Like that sounds good and it does solve some problems, right? But when you start writing particular data to certain partitions, that actually introduces other problems. Like if you need, you know, some sort of global,

uh, uh, keen management or windowing time, windowing stuff, man, it makes everything hard. Like it's like you solve one problem really well. And then you introduce a monkey wrench into some other problems and it just constantly goes on. So, I mean, the, the windowing issue actually became a real problem for us when we were doing partitions for, for different, you know, types of data on different partitions. And then it's like, Oh, well, you can't, you can't join them. Like Kafka kind of won't let you do it in a good way. Well, where I was going with in my mind was like hotspotting.

It's like, you have to be careful that your, your keying strategy doesn't, um, start moving more things into a single partition at a time. Cause then you lose all the efficiencies of having everything partitioned. If you're only, you know, if you have like 10 partitions and you're only ever writing to two of them, great. That doesn't do you a lot of good though. So again, you solve one problem, but I mean,

so just what Joe said is like, if you have all reads for a certain customer going to a single partition, well, what if that customer is just a massive customer, right? Yeah. Like now they're destroying that one partition. They lose the benefit of partitioning in the first place.

So it's like, ah, nine, 99 of your customers are like little small mom and pop shops that might have fewer than, you know, five to 10 employees. But one of your customers is a large enterprise that has,

you know, a hundred thousand employees. And that one is being, is only being written to the one partition. Well, their, their read and write time is going to be severely impacted in that case. Yeah, it's hard.

Yeah, it's funny. It is really hard. And that's the kind of thing is like, you know, we said these algorithms are from like the 70s. However, as you've seen, every single one of them is like, oh, you want replication? You just blah, blah, blah. And then someone says, well, what if?

And then it's like the algorithm thinks for a couple seconds and says, well, you just... And then real world says, what if? And so these things just go on and on forever. And like we said, there's no perfect solution for any of this stuff. All these things have

mitigations. The mitigations aren't perfect. There are trade-offs involved. Every single bit of this is hard to get get right and that's why we say the algorithms are from the 70s but no one's really got it right yet hey by the way man this all right so tangent number two i guess here we go you're right i i think this is worth a talking point is it is easy to what if things to death, right? So an example, like what we were just talking about is, well, what happens if we have hotspotting? What happens if we have this? Like what's easy to do as developers and managers and people doing this stuff is it's easy to be like,

well, we can't do this because that's going to happen. And that's like pretending you live in this unicorn world where what you have already is perfect. And that's not the case. So I think the important thing to do is to look at what your needs are, right? Like what are your most important needs? What are the most impactful things you can do? And then say, does this solve that problem? Right. Instead of, well, what about this? Like you can die by a thousand cuts when you're trying to come up with edge cases, but really what you need to focus on is what is my 98%, my 95% that I'm trying to hit and go after that. Just as a thought. Really good point. I

tell you, it's easy to kind of read this chapter and be like man replication is awful we should not look at it's like well there's some things you can do if you're willing to deal with some of these problems that you can't do any other way right yeah yeah oh man i i don't want to skip ahead but i kind't want to skip ahead, but I kind of want to skip ahead. So I'm going to say this now, but don't listen to it until later. Okay. But, you know, because there's a portion later in the book where they're talking about where you have multiple replication points, right, under the leaderless concept. And they're talking about like a calendar app, right? As like, you know, one, one replica is your, your phone.

One might be your computer. Another one might be a tablet, whatever. But like each one of those could do rights concurrently. Like no single thing is the, is the leader in that case. And I was trying to think like, man, like I never really thought about the complication. But like think about if you had to like write it from scratch, how you would deal with any kind of conflict resolution in that type of scenario. Right. So, yeah. So, I mean, I don't have the answer for it, but it was just what came to mind.

That was like, you know, one of the cool things that I mean, really, I've said it before, but it's things like that that make you think about things that you wouldn't ordinarily do in your like day to day job, which is why I love this book. By the way, I should mention, too, like I'm actually rereading the book. That's how that's how good this book is. So if you haven't read this book, you need to get a copy of this book. That's how good this book is. So if you haven't read this book, you need to get a copy of this book. So leave us a comment, and you can get a chance to win a copy of this book because you need this book in your life. Codingblocks.net slash episode 161. Yes, leave a comment. But, yeah, you're definitely right. I know that I'm personally, uh, bad about the,

um, like the what isms, you know, like, well, I don't know. Let me think. What about if this happens or what about if that happens? And like, I catch myself doing it sometimes or like, especially I catch myself doing it sometimes when it's in like a public forum. I'm like, oh man, why did I say it now? Cause now I just derailed the whole conversation by accident. The problem is, me and Bobby, one of our friends, we were talking the other day. That's what you do. You bring up things that have been painful to you in the past. So you bring it up.

You're like, well, what about this? I've spent a lot of time dealing with this stuff in the past. Right. So, so you bring it up and you're like, well, what about this? I mean, I've spent a lot of time dealing with this stuff in the past. And so you bring those things up. So it's not bad to do, but it's always important to keep in mind what do the benefits outweigh the cons, right? Are the pros way heavier on the scale than the cons are? And that's where you kind of, you have to do it with everything. So it's hard not to, to, to try and attack everything and be like, well, does it do this? Does it handle that? Does it handle this? And the answer is always no, it doesn't do everything. It can't. Otherwise there would be one system on the planet that everybody used and none

of this other stuff we'd even be talking about. Right. Well, and that's, that's another thing, another like eye opener about this book. I mean, between this book and the Uber engineering blog, for example, it just illustrates so perfectly why you cannot think about having just one data storage as the only place that your data is going to live. And that one data store is going to solve all your needs. Because even in this book, you know, in one of the previous chapters, they talk about, you know, online transactioning databases versus OLAP databases and, you know, why each one is structured for a particular type of purpose. You know, the indexes are structured in such a way for a different type of query, right. You know, I mean, that's just one example. And we've talked about the Uber engineering blog,

which if you've never checked that out, we'll include a link to it. But I mean, you see the, um, the, the journey that Uber has taken over the years and the, the,

um, transformation that they went from like one, you know, just building their platform out and the way the different data systems were incorporated into it. Just a great example of why, like there are legitimate reasons why it is okay to have multiple copies of your data for specific purposes.

And don't try to fool yourself into thinking like i don't want to copy my data around this is the only place i'm going to have it and you need to do everything from that one database yeah it's not even just okay in a lot of places it's preferred right um so yeah let me say we're saying like um there's no one database to rule all use cases. No. If you're a mom and pop restaurant or something, you could probably get away with MySQL. Just wanted to clarify. Yes, totally. Yeah, no, not everybody is an enterprise, right?

Not everybody has the data needs for an Elasticsearch and a SQL Server and a Cassandra and, you know, yeah, totally. That's pretty cool if you do have those needs, there are certainly interesting problems to solve with those needs. Uh, yeah. So onto multi leader replication, remember we talked about single leader last time where you have one thing that takes all the rights and then you can read from any with some caveats,

uh, multi leader solves the problem of a single point of failure for writes, which is pretty nice. And the deal is that it allows more than one node to receive writes. So I didn't really think of that the first time. So just to put some illustrations around this, like as you're thinking through this in your head, right? Our canonical reference that we gave to the last episode for single leader, I'm going to say it was the canonical, I don't care if you agree or not, was Postgres, right? Where you could have a multi

replica system, but there would be one node, one primary node that would be responsible for all their writes, but the reads could be distributed among the prime, among the secondaries. And our canonical reference for a multi-leader replication, and we discussed this even in the last time, because I questioned like, hey, what would be a good example of this,

was Kafka. Because in the case of Kafka, you can have a topic spread across multiple brokers, across, you know, you can have that topic split into partitions. Those partitions can then be spread across multiple brokers across there. You can have that topic split into partitions. Those partitions can then be spread across the brokers and there's an NSYNC replica. There's a leader elected for each partition of the topic and you can see

which replicas are in sync using some of the Kafka tools to, to view the NSYNC replica account or list rather. Yeah. And one of the point out too is that the replication still basically happens the same way. So everything we said about the last episode about kind of the leader publishing some sort of like, you know, operation log or something. We talked about a couple different strategies for that still happens. The only thing is that you have multiple leaders and each leader can act as a follower to another leader. So maybe, you know, in the Kafka example,

you can be the leader for this topic partition and a follower for this topic partition or another one. So the only thing that throws me off about the Kafka thing is I think in true multi-leader replication is, I think if we were going to say Kafka was a true multi-leader replication thing like that, then you could write to any one of the brokers for any of the partitions. And then that would also be replicated across the other ones, which I don't think is the case.

I think, I think it's just what you said outlaw, which is, you know, a broker is chosen as a leader for a particular topic, partition, whatever. So,

so I guess here's what I'm getting at. Okay, fine. I think, I think the, the way that multi leader is set up is if you were to have in, and I can't think of a database system off the top of my head. I know that Postgres will do it with other tools, and there are other databases that will do it with other tools built on it.

But let's just say, for example, you have three MySQL databases, right? And they're all leaders. That basically means you could write to MySQL A, Joe could write to MySQL B, and I could write to MySQL C, and they're all going to replicate their changes across to each other, right? So that's what the true multileader is, is the same. Each one of them can take in the writes, and then they'll replicate those changes out across. When you're right, you're right. So another way of saying that then in the Kafka example that I gave, why that doesn't work is because it's still a single leader replication strategy just at a more granular level.

Right, right. Fine, Alan. It's fine. I didn't want to mislead anybody. No, sure, sure, fine. Yeah. Can we have a tangent instead? Yeah, no.

We probably could. I'm just kidding. I am trying to remember because they did call out some other technology. I mean, you are right. They did mention specifically like GoldenGate for Oracle or BDR for Postgres or Tungsten Replicator for MySQL that would allow those traditionally single-liter relational databases to operate in this multi-liter fashion. But I thought there were some other ones, like a Cassandra or something maybe, but I can't remember.

There is one, but we're not going to say it right now because it goes back to something you guys said earlier. But we're going to wait until we get there. So we can guess it until we get to it? We could, but that might be boring for everybody. So when do you use multi-leader replication, right? Like this sounds like a win-win to a certain degree. It sounds like a complication. It might be, right? Well, they say you usually don't want to have multiple leaders in the same data center, right? So I know the three of us, we work in cloud applications and stuff, and we were just recently talking about disaster recovery type things, right? Like you typically want to have your data in multiple different regions, right?

Let alone data centers, but at least multiple multiple data centers so you wouldn't have two leaders in the same data center it just doesn't make sense you're complicating things for no good reason so you might have one in the southeast and then you might have one on the central east right something like that which i don't even think is a data center but if yeah i mean i'm not going to say that you wouldn't have them in the same data center because your use case might warrant it, maybe. But it's definitely easy for us to think if we take a global stance at this from an application that is

like a Google.com or something where maybe it needs to be spread out geographically as well as having, having that spread out for redundancy, then, uh, the multi-leader might make more sense. Yeah. They did say that typically you wouldn't do it in the same data center only because the complications may outweigh the benefits at that point. Not saying that somebody doesn't for any good reason. There probably is, but at least across data centers. The reason why I was thinking that maybe you would is if you were in an environment where you have heavy, heavy traffic to it, even for

internal use, then maybe you would. Nothing that's coming to mind, though, because know, maybe you would, but I, nothing that's coming to this coming to mind though. But, um, cause everything that I'm thinking of is more like just bandwidth, not necessarily transactions. Yeah. It's,

it's hard to come up with those. I can't think of a multi-liter replication database, uh, where, uh, like I can think of ones like elastic or a Kafka that kind of have something like that, but there's a little kind of nuance that keeps them from kind of fitting like Elastic or Kafka. They kind of have something like that, but there's a little nuance that keeps them

from fitting this textbook definition. But remember, we're talking about cases where the replicas are 100% of the data. No partitioning involved. And I just can't think of a single database system that's got a notion of having multiple leaders that is also 100% of the data. I thought some of the

NoSQL type document type databases were like this. Maybe a Mongo or a Cassandra or something. Cassandra is meant to be horizontally scalable on the data, not they contain the full set on a node. That's the only way

I know that's leaderless. I tried looking at MongoDynamo. I looked at both of those on dbengines.com and they have a section on replication. Both of them I was like, yes! DynamoDB is the book's canonical reference for a leaderless

database. In fact, they refer to the whole leader... leaderless database. And in fact, they refer to, um, you know, the whole leader, they, they were, we're skipping way ahead,

but the whole idea of leaderless, uh, Amazon kind of like brought back that, you know, resurgence of the idea of a leaderless database with Dynamo. And so now databases that are Dynamo or that are, uh, leaderless like that, they're referred to as like dynamo like. Interesting. Yeah. I mean, again, we're more talking about when you have a Postgres database that is fully replicated across multiple nodes and you can write to any one of them at any given time. Right. So, yeah. So one of the things that they said is they have this interesting approach for a data center to where they have leaders and

followers similar, similar to the single leader, right? So within the data center, you have a leader, then you have followers. And then across multiple data centers, you then have a leader in that data center with followers in that one. But that leader is a follower to the other data centers, right? So it kind of spiders out is basically what happens. And that's usually across multiple data centers. And they drew a picture in the book that is pretty helpful to look at. So if you have it, or if you want it, well, we can describe it, go take a look at that. But it's basically like a chained single leader setup. We can describe it. Hold on.

Give us a minute. Draw some triangles and pyramids and stuff. Yeah. You know, it was basically like it was, no, if I remember the drawing that you're referring to, that was basically like a document gets written to Alan.

Alan tells Michael, Michael tells Joe, right? That was there across multiple data centers. Yeah. But that's the chaining that you're referring to. Yeah. So the chaining is data center.

One has leader one and then followers, right? Data center two has leader and then followers. So the leader from data center two is following the leader from data center one. So it's weird. Your leaders are kind of following other leaders and then the followers are all following their leaders. So it kind of spiders out.

That's why I was trying to use the telephone game example as a way of just describing it from the server point of view and not care about the the users yeah a lot of the i will say that is one thing that the book did a good job of here well i mean most everything but they do draw out the pictures to where it's easier to visually see how the how the data is moving across the topology and I wanted to say, too, I think I talked about it the wrong way. I kind of made a distinction of databases being either single-leader or multi-leader replication, and that's kind of not really the case. Basically, the deal is that a lot of the databases that support

single-leader replication also support multi-leader. They just have a mile-long list of caveats and things you have to do so like postgres supports multiple leaders uh you'll frequently see it listed as master master so if you actually google master master replication databases i found a pretty big list and so uh oracle's on there microsoft sql my sql is on there postgres but with usually with external tools so that's what they were calling out yeah it. It's not built into it. Yeah. Yep.

In a lot of cases across or whatever. So it's kind of like one of those things where like, yeah, it really is just like a kind of a complication where they can figure out how to make it work, but they'd rather not. Right. Yeah. That's kind of the thing. So they also then go into compare the single liter versus multi-liter replication. And, and like some of the things that they were looking at were performance, right? So performance is actually faster multi-leader, which sort of makes sense because you can have writes occurring in each data center independently of the other data centers, right?

So that makes sense. You're not hitting latency for you trying to write something from, you know, the West Coast over to the East Coast data center. Right. And the synchronization of that data we talked about earlier, that can happen asynchronously. So behind the scenes, after you do your right, that thing gets replicated across. So that's good. Fault tolerance. In single leader, everything is on pause while you're electing the new leader

in the multi leader. You don't have that problem, right? Like, or you only have it in one data center. So, so you're not pausing everything. So that's fast. Um, in the multi leader, the other data centers can continue taking rights while the other one is catching back up after the new leader was elected. Right. So again, that's all pretty amazing things, all in favor of multi-leader replication. Definitely makes it sound like you're making the case to like multi-leader replication is always the better thing. And it's just, it's just so easy, guys. You go into the YAML.

That's what you're going to configure. It's probably not YAML, though. It's just YAML. Yeah, it might just be XML or JSON or whatever it is. But you're going to flip a bit there and turn it on. You're going to go, yeah, of course I want multi-liter replication. And restart the application, and boom, it's done. I mean, here's another one that makes it sound like, yes, what you just said is absolutely

true. Even if there's network problems, if there's network problems in a single data center, your other data centers are unaffected. They're still taking rights. They're still doing things. So your multi-leader setup is still more fault tolerant, more capable of continuing. So yeah, it sounds like, well, I mean, that's it. Yeah. Let's just go set this up. Let's be honest though. Like what happens when that network does

come back? What happens when like, you know, let's pretend, let's go back to the example that I gave, I think in the last episode where it was like, you know, the, the country gets split in half internet wise, you know, the East coast can't talk to the West coast, but there's a replica in each place, you know, so there's still reads and writes that might be happening in each. And now it comes time, like when that, uh, connection gets restored and you got to like sync that data back up and pray that there's no collisions, uh, no conflicts, you know, I mean, it makes you think like, like how would you key your data for that type of scenario? Right? Like, so this goes back to what my comment that I made earlier is that knowing what your, your data strategy is going to be, the storage engine, okay, well, if I'm going to key it in this way,

I need to make sure that my keys can, you know, Matt can, can handle this type of scenario. Right. And you just hit on the one thing, like we just listed off a ton of reasons why multi-leaders amazing. You just hit on the one thing that makes it almost all completely fall apart, which is if, if they can't communicate for some period of time and now it's got to reconcile everything when it all comes back up, like that's not small, that's, that's not, that's not insignificant, right? Well, I mean, you hope it's insignificant because it was only like a small blip when, when, you know, for the outage, you hope it wasn't a day's outage, right? Well, I mean, you hope it's insignificant because it was only like a small blip

for the outage. You hope it wasn't a day's outage, right? Yeah. Yeah. So, yeah, I mean, just that one thing is a massive, massive thing that you have to consider if you're going to go the multi-replication, the multi-liter replication route, because

that is a big deal. That's why my blog now, I've got nine total SQL instances. I've got three regions, one leader, two replicas in each. And yeah, it's really expensive to host a blog, but if I ever wrote any posts, then the inserts would be pretty nice. And I don't have to worry too much about network partitions. When Fastly went down, it didn't matter because first of all i had no traffic but also i'm built for that sort of stuff oh man so here's the other thing we mentioned that that postgres and these there's usually

like third-party tools that that allow for this multi-liter replication. So one of the problems that they say is it's typically bolted on after the fact, right? Like this isn't something that was built into those database engines to be able to handle. So a lot of these things are handled in a way that maybe isn't perfect for data that's stored all in one place. Dealing with auto-incrementing keys, that's a big deal. Triggers, constraints, right? Like if you have triggers that go funnel through and update a bunch of different tables when something happens,

how do you handle that when you're trying to sync things back up? And if you have constraints, check constraints, anything like that as well on your tables, that's all business logic that's got to be reconciled at some point when this data comes back together. So they said that these alone, those things that we just said are reasons to avoid multi-leader replication. Yeah, I like that. So you start off like, okay, so multi-leader is better than single leader in every way, except for the real world problems that arise that actually make it much worse. Don't do it unless you have to. It's truly funny. It's set up to help you avoid the problems of things like networks that go down and all that. But then the very thing that it's set up to help you with is the one thing

that will crush you when it all comes back up. Yep. This is distributed systems in a nutshell right there. It really, it really is. Yep. Yeah. This is why you just got to like go,

uh, you know, you gotta, to like go, uh, you know, you gotta, you gotta just stay active, find some kind of other activity to get rid of all the, all of the frustration. Cause like dealing with this kind of, this kind of stuff is just,

you know, you need like a punching bag to, to take that out on or, or yoga or, you know, cycling or whatever. Okay. You know,

I mean, I was waiting for a punchline that none, none came. You were being serious.. You know, I mean. I was waiting for a punchline. None came. You were being serious. No. Well, I mean, I think I did make a yoga joke, but it was a bit of a stretch. Here we go. I knew it was coming.

I'm not that predictable, Alan. Sorry. I've known you for a while. Fine. Do you really want to joke? That one was Graham, by the way. How does the moon cut its hair? Eclipse it. Oh, I even thought about this.

I just kept thinking about a moon with a bowl cut and how silly that would look. I had a bowl cut. Didn't we all at some point? In the 90s. If you're in the 90s, it's a bowl cut. I had the vanilla ice steps. Oh, no. Yes, sir. I would have said that.

Oh, dude. I had the flat top and everything. It's not getting cut out. Oh, God. Yes, sir, Matt. I would have said that. Oh, dude. The flat top and everything. It's not getting cut out. Oh, God. I want this picture. I want this picture. If you want to see this picture, leave it with you. I know your wife. I'm going to find a picture of this.

This is going to happen. Oh, that just might be the hero image for this episode. Stubbly valid. Maybe we'll get a picture up here for the next episode. That'd be so fantastic it'll be the first non-pet picture since like the 90s i don't know you put those nasty slugs on a couple episodes ago my gosh man those are snails Ew. This episode is sponsored by Educative.io. Educative.io offers hands-on courses with live developer environments, all within a browser-based environment with no setup required.

With Educative.io, you can learn faster using their text-based courses instead of videos. Focus on the parts you're interested in and skim through the parts you're not. I mentioned last episode that I was working on through the Grocking the Advanced System Design interview, which is a follow-on from the massively popular course from Educative, Grocking the Advanced Systems rather, Grocking the System Design. I can't speak right. Easy for you to say, Joe. What I was trying to get at

is that they had a section on Kafka and the design Kafka, which is really nice. And so I worked through that, got 100% on the quiz at the end there. And then I searched Kafka on the website

just to see what they got and found my next course, which is a microservice architecture practical implementation, which has got 112 lessons, 71 quizzes, 10 playgrounds, and 92 illustrations. Oh man, that sounds amazing. So I was also on there looking around because of all these enterprise-y things that I've been dealing with, right? And they've got an amazing course on Spring. So if you are looking to up your enterprise Java development game, go check out

their Spring course, which is just excellent. They got a ton of good information in there. And maybe that's not your thing. So guess what? They've got some brand new courses like Coding Career Handbook. They've got the DevOps for Developers Learning Path. They've got Decoding the Coding the coding interview, basically whatever your, uh, path is that you would like to learn on. I guarantee you're going to find a bunch of great courses there or even learning paths.

So it would be collections of a whole bunch of things like they've got you covered. And so, Oh yeah. Sorry. I got so excited to go ahead. I know. It's so exciting to talk about it.

I was just going to say, be sure to check out their best-selling Grokking the Interview Prep Series. They have the courses like Grokking the System Design Interview that Joe couldn't say earlier, but they also have Grokking the Coding Interview as well. Well, I want to tell you about the newest edition, Grokking the Machine Learning Interview, which actually focuses on the system design side of machine learning by helping you design real ML systems such as an ad prediction system. It's the only course that was kind of on the Internet.

Yeah, so go ahead and visit educative.io slash coding blocks to get an additional 10% off an Educative Unlimited annual subscription. You'll have unlimited access to their entire course catalog. But hurry, they don't keep these deals that often. So that's educative.io slash coding box to start your subscription today. Okay, so you've heard us say this before, and especially if you have listened since the beginning, you know, since episode one, like Anonymous has, then you've probably heard us say this once or twice where, you know, hey, if you haven't left us a review, we would greatly appreciate it if you would. They really do put a

smile on our face and mean a lot to us when we read those. So you can find some helpful links at www.codingblocks.net slash review. And with that, we head into my favorite portion of the show. Survey says, all right. So a few episodes back, we asked, how do you prefer to be interviewed? And your choices were behavioral interview. Tell me all about your mother. That sounds wrong. We actually said that is like the thing. I didn't make these up. That can't be right.

I mean, that's clearly an outlawism. That don't blame me. Okay. Well, let's move on. Your next choice was take home project how many hours do you really spend or whiteboarding like surfing but worse or multi-person interrogation room like firing squad but better or informal dinner because that's not creepy. Or weird esoteric. How many angels can dance on how many ping pong balls on a plane with three potential destinations?

You know, I actually wrote these. That's why you hate them. That's why. That explains so much. Okay, so this is 161. So odd number. Alan goes first. Thanks to Tutko's trademarked algorithm for who goes first.

Yeah, I mean, I got to say, too, these are all amazing answers and equally terrible. Emphasis on the equally terrible part i don't know that anybody loves these processes so um man i think i think people are gonna say whiteboarding like surfing but worse it's pretty bad if that's the best. Yeah. I mean, out of all these, this is, that seems like the one where you're at least trying to take some of your nervous energy and place it somewhere else.

Right. So I'm going to go with that. And I, man, there's, there's several here. I'm going to go with 30%. Okay. I'm going to say multi-person interrogation room like a fighting squad but

better and I'm going to say 15% to win okay Alan what was that that's a percent right I forget yeah I think you're close you're somewhere in the mathematical realm of something.

There were numbers in it, so I'm going with it. Yeah, totally. Alan goes with whiteboarding for 30% of the vote. 30, yep. And Joe goes with a multi-person interrogation room with 15%. Correct? Yep. And the answer is take-home project for 30%.

Wow. Okay. I guess the heat's not on when you do that, right? Like you can cheat sort of. Oh, I'd be anxious for days. I don't know about you get time to cheat. I mean, I've heard about some of the take-homes that I've heard about. You're like up all night trying to solve it.

Yeah, that's the problem. I mean, I guess I say when I say cheat, you're not having a hand code in front of somebody, right? Like you can actually go do something, Google, Stack Overflow, the stuff that you do on a regular day. Yeah, except it's a ridiculously hard problem that in the workplace might take like a whole sprint know, a whole sprint to do. And instead they're like, Hey, get this done overnight. Right. And then we're going to ship it. We're going to, we're going to see how well you do if you do it overnight. Like they're giving you a hard problem that they don't expect. I don't, they don't expect to be fully done in the night, but to see how far along you get and you know, what your, your take is on the problem. Interesting.

What was number two? Multi-person. Multi-person, really? 25%. I like multi-person better than one person. There's something creepy about being in a room with one person. Because you can focus on somebody that maybe doesn't scare you so bad in a group of people, I guess.

I don't know. Yeah, one person seems very confrontational. But with six, it's like, okay, I got this. Interesting. Maybe. But, I mean, if you were coming at it from that point of view, you would think they'd be like, oh, my gosh, there's six of them and only one of me. But if it was just one person, then it's just one-on-one. If it was one, it's like, oh, crap, I've got to carry this conversation.

Am I talking too much? Am i not talking up with six people it's like shoo you just hang on however you're done whatever hang on maybe you got a baby didn't oh okay uh speaking of hanging on do you would you like another joke? Yeah. How about this one then? Tell me why hasn't Marvel put advertisements on the Hulk? I mean, his clothes just rip off of him. Something smash. I don't know. I mean, I don't know why they haven't because he's

essentially a giant banner. Oh, geez. Yeah. Yeah. Awesome. I, of course, like, you know, like I credit people. So that one's from Lars. But, you know, I ruined the way they originally give them to me.

So, you know, whatever. It's awesome. I appreciate it. And I've got more of these awful jokes, too you know just wait for it just wait for it i mean at this rate i'm gonna need a new job you know what i might get a job cleaning mirrors i could see myself doing that it's a job i can see myself doing hey back to back to ripping clothes off. Can we get a survey here? He's still struggling. The mirrors.

Did you get it, Joe? Of course. Do you lie? What was the joke? Yeah, exactly. I might get a job cleaning mirrors. It's a job I can see myself doing. Oh, I got it.

So, it consisted with the Hulk joke. Oh, that one was from Mike RG, of course. Yeah. Well done. The mirror joke, that was. Consistent with our Hulk joke here, then. Like, you know, because he has to get dressed. So, how do you put your shoes on?

Right? So, this came up here in my house. I'm curious. I'm really curious to know how this goes. But here's your choices. Sock, sock, shoe, shoe. Just in case I decide I only want socks on. Or sock, shoe, sock, shoe.

You can't have a foot partially dressed. Or shoe, sock, shoe sock shoe sock or wait uh or socks you don't wear socks with boat shoes or shoes it's a flip-flop life for me i actually like the survey a whole lot yeah because i had to think about it after i read it i was like oh there's one definite answer here yeah i agree i mean this yeah like this is right up there like do you put one leg you know do you put the left leg in first or the right leg in or do you put both legs in first when you put your pants on you know like obviously it's put them both on right in there right so i mean this and this answer is kind of like that too like you just gotta get it arranged

properly you gotta set things up that means you gotta sit down no no no it's still hanging on the hanger it's already perfect that way and you just jump into it and when you do it it comes off the hanger because they're just like clipped on yep i don I don't know how you, how you hang your clothes out. Yes. Kill me. That does it for clothing blocks. Hey, Hey,

one last joke. Well, I don't know. Do you want, do you want one less? Cause I'm kind of torn. Like it's a, do you want to hear a COVID joke?

Cause I don't know if it's too soon. Do you want a COVID joke? I because i don't know if it's too soon do you want a covid joke i mean whatever just do it i'm on the record as nope yes never mind i don't want to spread it another one from mike rg thank you yes good god you got me. All right. So let's talk about clients with offline operations. So this is kind of going back to this world that I mentioned of like the calendaring app, right? And like complications that might arise from there. Yeah, there's lots of ways to store data. So we mentioned relational a lot tonight.

Of course, NoSQL. But also, sometimes you just have a full-on copy of the data on different clients. Yeah, and Outlaw actually talked about this with the calendars, right? Although you said something that was a little bit different than how they put it in that you said that there would be copies or replicas on everything. What they actually said is when you do these calendar type things on all

the mobile devices, your, your phone, your, your tablet, whatever, each one of them acts as a leader, which it makes sense because you might write a calendar entry onto your phone while you're offline. Right. And then when the thing comes back online, then it now needs to sync up with all the other leaders out there. Cause you could have done it to on your tablet. You could have done it on your phone. You could have done it on a laptop. You could have done it everywhere. And then they've, they've all got to reconcile at the end.

I'm going to sworn that was given as a leaderless example though. No, this was actually, this was in the same section with the multiliter replication. And here was an interesting thing that I did not know. So CouchDB was designed specifically for this type of application, which is the multiliter offline need to resync when it comes online thing. So anybody out there that is planning on doing some sort of task management thing or or calendar type application you might want to take a look at couch db

it was like handled emerging for you like i guess it's got like client versions of it they can kind of sync up i i don't know i don't know we'll take on it yeah they called it out. It just reminds me, though, you hear... I remember a few years back, we were at a JavaScript meetup or something like that. Remember when we used to go to those? I do remember those. Things like CouchDB would get just a bad name.

There'd be somebody there that would just hate like, oh, we used it for X, Y, and Z, or we used Mongo for this problem, and it was just an awful ball, and we had all these problems and and and like the thing from this book that just it uh like just kind of beats into your head though is it's like okay well you got to make sure you're using the right technology for the right use case so you know if you had a rough go of it with couch db there's a possibility that you know you were just using the wrong technology at the time for that problem.

And did you not like the database? Did you not like the problems that you were trying to solve with that database, which are hard? Yeah, that's a good question. Yeah, so the next section that they jumped into, and this is kind of interesting because we're using one of them right now, is collaborative editing. They were talking about things like

Google Docs. They called out Etherpad. I'd never heard of this one. I mean, there's tons of them out there, right? Like OneNote's probably maybe OneNote. I was thinking that would be Etherpad. Might be Etherpad. Could be Ether, Ether. I don't know. VS Code

has the live sharing and JetBrains now has it with code with me. Oh, that's right. So those are all really cool. And they said basically these act like multiliter things as well because your changes are saved to your local version. So let's say you're in a Google Doc. Your local version is probably the browser cache or something or, or browser storage. And then it syncs that up to some other replica that then gets pushed out to other users, browsers that are doing it. Right. So, um, this is another form

of that multi-liter type setup. Yeah. I'm just struggling now because the whole realization that the calendar example was under the multi-leader and not the leaderless, even with the Google Doc example, I'm like, man, why would that not be a leaderless example? I'm trying to think of

what would a leaderless example be then? I haven't read that part. Cassandra. Would you say? Cassandra is like the, it's the example I always see given for leaderless. And actually it's been a long time since I've, I've seen highlights in the book for a leaderless, but I don't remember and I haven't rewrote it yet.

Yeah. I've got to read it. My guess is, well, I guess, I don't remember and i haven't rewrote it yet yeah i've got to read it my guess is well i actually i don't guess i don't mean i guess i don't mean technology like the the the database behind it i was just thinking more of like a use case yeah because google docs and calendar applications those are use cases not databases right Yeah. I don't know. I'm going to have to read it. I can't say anything on it. So we threw this in here too. We talked about the whole problem when, when in a multiliter setup, you know, the network went down and it comes back up later. Conflict resolution is actually a bit of a big deal. And this is like one of the major thorns in the side of the multi-liter replicas. So you think about this, like there's all kinds of situations where this becomes a

problem. You have the networks are down in between two data centers and you write the same columns on the same record in both places, which one wins. It's not necessarily that it can be a time-based thing, right? Because it doesn't matter that I modified the record first or Joe did or Outlaw did. If all three of us modified the same record, how do you pick which one is the winner, right? Like, you know, if I change the description on a product, what makes yours more right than mine or mine more right than yours? Like, it could be time. Maybe it has nothing to do with time. Like, those are the types of problems that you end up running into. that depending on what your use case is and the technology behind it,

you know, it could be either the conflict. You try to resolve the conflict on the right, in which case the server has to, like, make a decision. And so maybe it could try to do a merge strategy and Git came up as part of that. Or maybe it could, you know, just pick one, you know, based on some kind of value like time or whatever, like, you know, whatever it's in, whatever its reason is, it picks it picks one. Or the other strategy was that it would do, it would force it on read. And so the application developer could then be given back a data structure that contains both changes in full. And then you would

say like, which one you want to be, uh, the one to, to win. And then it would write back that change based on whatever the, um, uh, the user said. And, and you've kind of seen examples like that in, um, I think Google docs does that occasionally where they can be like, Hey, there was a conflict, which one of these is, is correct. And confluence, um, Alassian confluence, I think does that same thing in the case of conflicts. It can say, hey, which version is the one? And also an add-on to Confluence, which is what's the drawing package? Gliffy? No. Gliffy. Yeah, Gliffy. I think it does the same thing, which may be just

building on top of what Atlassian already has in there for Confluence. And the important part of what you just said there is, this is ways trying to automate it. When you try to automate it, there are sometimes some features built in, like you said, maybe timestamp type things for simple. But a lot of times what these tools have built in for the multileader replication is they allow you to plug in your own custom logic, right? So a lot of what you were just saying is their custom logic applications or algorithms that were written on top of it to say, hey, we don't know how to merge or resolve this conflict.

So our custom code we wrote is going to send it back to the application to let the user make the decision in the application. So that's why it's complicated, right? It's not just an easy button on any of these things, which is why a lot of times people will choose not to do the multileader thing because you've got to come up with some time to write this code to make this stuff work. That was the on-write example, was that there was a database or whatever its storage is that I'd never heard of, but Bucardo. But it would provide you a handler that you could basically like a callback, right? Like it would say, Oh,

there's a, there's a conflict here and it would call your method and you could make the decision as to like how to do the merge. And it would write back. And it was specifically a Pearl, a Pearl script. You could write a little bit of Pearl that you would add back to it. Yeah.

And CouchDB was the example that they gave for the on-read conflict where it had both sets of, both changes that the application developer could receive both and decide how they wanted to present that to the user to make the decision. And here's what stinks, right? You can take it a step further. I mean, just with the three of us, if three of us wrote three different documents and then somebody else gets it, it's no longer just merging two conflicts. There's three.

And you might have to handle a number of conflicts to, you know, however you plan on doing that. So again, it just, there's some complexity that comes along for the ride with being able to have these multi right leaders you know go ahead one last thing so you brought up the amazon thing earlier and this is where they were talked about in this particular section and what they said is amazon had done a good job on on like if add items to a cart, then that would get replicated across multiple leaders pretty well. But where they kind of had a bad setup was when you removed items from the cart, it didn't replicate that stuff the same way. So somebody would reload their page and all of a sudden their items would be back in their cart that they thought they had removed. And so they, I guess their algorithms for handling

these multi-liter rights, um, weren't perfect for all situations in your shopping cart. So when you get an Amazon owner order and you're like, wait a minute, I didn't think I ordered that. That's why. And that's also why they have so much money. Exactly. You ordered more than you thought you did because you tried to remove some stuff. Yeah, totally. Makes sense. So I was just thinking of an example of multiliter.

Git is a pretty good example where, you know, Git is not, or maybe it is a database, but it's kind of a situation where you go and you clone a replica of some data that you've got locally, you make your changes, you commit locally, and then eventually you say, I want to push this back to the leader. Well, when you've been working locally,

you've been treating that as the, basically as the leader, you've been working slowly with that, and then it's time to kind of synchronize your changes. And that's where you can get into conflict, you know, like merge conflicts, all that sort of thing thing so it's kind of like a really slow motion example of a multi-leader system where you're committing locally and also synchronizing those changes occasionally with a more prominent uh you know leader a leader that's got more authority than yours but you know even so you've kind of got a multi-leader setup

which is pretty cool that That is. And the conflict resolution is on the person doing that merge. It's interesting. So I did look to see if anyone has built a database on top of Git, and there are. Several people have done it. It's like GitRose is the most prominent one on GitHub I found.

That's pretty cool. But yeah, it literally just checks stuff into GitHub for you and saves it as files, and you can have history and all the benefits of Git. Conflict-trayed solution. It's pretty cool. That's awesome. So, multi-leader replication topologies is something we kind of hinted at a little bit.

But basically, the topology is just a description of how the replications communicate. So topology, you can kind of see it as being a map. Like this replica talks to that replica, this leader talks to that leader, and you can draw some arrows. We won't try to describe any pictures here, kind of maybe.

Oh man, come on. I just read briefly. On all seriousness though, this was pretty key to the conversation. The three types of topologies that they talked about, which they did draw. But I mean, you could picture it really easily in your head that they just talk about four replicas and the different strategies. One was circular. So

in my mind, I immediately thought of Token Ring. I don't know if you guys thought of that. Yeah I don't know if you guys thought about that. Oh, totally. IBM. Yeah. With the way that the network would work there. So like in the circular example, you know, one replica sends all of its changes to the next replica. That replica then takes all of the ones it received from the previous plus anything it has and then sends it to a third one.

And it just keeps going around the around the circle like that and so it works great for even or odd doesn't matter um yeah as long as there's more than one you can do it you kind of read from the right and right to the left yeah then there was the the star uh pattern which you could also refer to as like i've heard more often heard it referred to as like a hub and spoke, but you know, there's one replica in the center of the three of the four, sorry. And, uh, the other three read and write through it. Um, so it's, it's in charge of distributing it and the one that's at the hub. Yeah. And you could kind of envision that that thing would be like of, of the different strategies that are here. Like that one might be the most brittle because if that hub goes down, then you have this whole like, okay, which one is, you know, okay.

I realized that we don't have like a single leader quote, but we kind of do. So one of us need to decide who's the new hub. So, so you have like your same, uh, uh, leader election kind of strategies to solve in that hub and spoke or, or star topology. And then the other one was, um, all to all. So in these four, um, replicas draw read and write arrows between all four to all four, right? And then

that way, that one actually seemed like the, at least maybe the most, maybe hard, but also the least problematic in terms of like, it was of like, how about I word it this way? It seems the most fault tolerant of all of them because everything is talking to everything. And so if any one of those went down, nothing changes. Like everybody's still talking to everybody but in the case of the circular one if you lost you know if you picture you have one at the 12 o'clock one at three o'clock six o'clock and nine o'clock and you you lose that one at nine o'clock well when six o'clock goes to communicate to nine he has to know like oh no he's down i need to go straight to 12 so there's like there's like a little bit of latency there that he has to figure it out before the circle continues again kind of situation

but i'm with you. The all to all, while it might be the most fault tolerant, it is also the most rife with the, the conflict resolution that's going to have to happen all over the place. Yeah. It's definitely a chatty, chatty Kathy,

right? Like, you know, so you need to have the bandwidth to handle it. So depending on what type of data that you're trying to synchronize, it might not be, you know, the most performant.

Cause if you have like, you know, super, super large bits of data, like if this was binary data that was in your database, you know, maybe that's not the best one then, because those,

those might be very expensive reads and writes that are just for replication not to actually go to your users there's another problem too with uh what they call the overrun with all to all whereas if you've got like a circle and one of your nodes has a much faster network or is just much faster than the others then it doesn't really matter too much that one will just read fast and it'll pass on its stuff a little bit faster than the others. But overall things still kind of even out when you're going all to all and you've got one that's significantly,

or, you know, some small number of them that are specific that are faster than everything else, then it can kind of start getting ahead. And then you start having more and more positive causal relationships that we mentioned out. And that's something that you don't really get with the ring, which is

kind of cool. And also, you hit the problem with the, you nail the problem with the stock technologies, you lose that middle one, you got the problem. And then the same with the ring, too, is like you lose one, you break the flow. It's not as bad because

you're not missing that kind of one that suddenly nothing can talk, but it still has to be manually repaired. So, yeah, it's not as bad because you're not missing that kind of one suddenly nothing can talk but it still has to be manually repaired so yeah it's all just rough and they mentioned one kind of strategy for dealing with the kind of overrun problem was just keeping those version vectors which is kind of like the logical clock we talked about

which kind of helps you from going backwards so it's like a built in throttling almost yep yep and then they also So it's like a built-in throttling almost. Yep. Yep. And then they also mentioned that these are three common patterns, but you can actually, like people will do common,

kind of manually set their own topology sometimes. So if you know that you've got an extra fast node, maybe you'll connect it to less or maybe to more or whatever. Based on your geography, you might have some sort of other kind of pattern that's maybe a mix of star and all to all yeah that's what i was gonna say is depending on what your your use case is you could actually see you could easily envision like a world where uh you know maybe maybe across the open internet, you're going in one particular pattern, like a circular or, or a star pattern, for example. But then once you get inside the data center, you might do something like an all to all within that given data center. So like, if you were to picture like the U S for example, maybe there's like, uh, uh, a data center in Texas,

another data center in Atlanta and another data center in California. And, you know, maybe there's like a data center in Texas, another data center in Atlanta, and another data center in California. And maybe because Texas is centrally located, maybe it gets treated as the hub in that star topology type of scenario. But once you get inside the California data center, then you might have 100 servers in there that are all replicating amongst each other. You know? So, I mean, there could be a combination as the point. Yep. Well, with that, we will have some links to resources that we like. Obviously the book's going to be one of them. And well, I mean, we, I don't, we can't, I, we can't rave enough about this book i actually i was thinking about this i would have to guess that this would probably be would it be fair to say that collectively this might be our each of our favorite book like all three of us would we each say that

i think so yeah yeah this is what i learned the most about. And it's even it kind of like shines a light on things that I've kind of thought about or kind of half new and like help to explain why they were, you know, just kind of filled in some gaps for me. Yeah, I mean, it's definitely there are a lot of other books that have a lot of great things that we've read in it. But for some reason, this one just makes me think about problems that I wouldn't have otherwise taken the time to think about. And for that reason, I just love this book.

So definitely, if you don't have this book, if you haven't read the book and you would like to have a copy of the book, head to this episode's show notes, codingblocksnet slash episode one, six, one, and leave a comment there for your chance to win a copy of the book. And with that, we head into Alan's favorite portion of the show. It's the tip of the week. They may get a flat top picture too. If you, uh, Oh man, that exists. I'm sure it does somewhere uh photoshop in your days are never buddy all right uh so typically i have one this week this is the jz og tip of the week right here

which i stole from uh co-worker chris hey chris thank you uh i have never heard of dot http files it's uh just a file extension dot http but what's special about it is that intellij and with a plug-in vs code knows how to interpret these files and what a http file is is basically a collection of uh essentially curl commands basically these are uh you know gets posts deletes puts whatever you want to do except it's laid out in a really convenient way that is easy to read and easy to run and it's you kind of got to see it we'll have a link in the show notes here but the deal is you have a list and it says you'll say maybe post and then the url and then you've got a couple lines where you can add headers or authorization wherever you you want to, if you've got them.

And if you've got a JSON body, you just paste that JSON in there. You don't have to set up all these arguments. If you've ever seen a curl command, it's pretty hard to read. You know, it gets long, especially if you've got JSON. And if you need to pass a JSON file because it's too long, then you need to figure out how to get that binary file up there. It's one of those things where I'm always Googling for the arguments. I'll fix that link. It's got a parenthesis on the end. Sorry about that.

Hopefully I just fixed it. The deal is it's much easier to write than curl, which is nice. But the real magic is the IDE support. I can open up a file dot http extension and i have you know say 20 order effectively curl commands but instead of a shell script where i have to run the whole thing or have to comment out the parts i don't want i can just go hit the little

play button next to each request in order to run it and see a history of the uh the commands that have run i can also go up of the commands that have run. I can also go up to the top, click run all. I can do things like set an environment file that will change things like the URL or pop in passwords. I can have variables, looping constructs, and all the things that kind of make shell scripts nice and you can do, except I can run it piecemeal. I can go to line 137 and click play and issue just that command.

So it's kind of like having Postman, but all in one file. That is awesome. That is awesome. I mean, because like, oh, sorry. No, go. Well, I was going to say, like, how many of us have used, like, Postman for your Elasticsearch queries? Right. Right.

And so, like, immediately I was thinking, like, oh, well, this is a game changer. Yep. So the project I've been working on now at work, I have Elastic.http file that has like a thing that I can click here to delete index, click here to create one, click here to whatever. And I can set variables in that. And unlike shell script, I don't have to worry about running the first three lines to get my variables. And then if I need to run the 11th call on the file, I can just run in various pieces. It's all set up for it. But also the beauty of this too, though, is that I can commit it.

Unlike postman uses, I can commit this in my, you know, live aside my code. Hey, and what I was going to call out here, when was the last time you saw any extension that had a perfect five star rating? This thing has, it's 4.96 out of five stars and it's got 269 ratings with 1.6 million installs of visual studio code. Like that doesn't happen.

Like everybody loves this. It finds out about it. As soon as you see it, you're like, I know that this works. It's up on the, if you click on the link to the Visual Studio Marketplace, where you see this underneath REST client on the page, you'll actually see in green there all those things. Oh, I see it now in green. Okay, because I was looking at the stars, and I only saw five out of five.

I'm like, where's he getting this extra granularity? Yeah, man. Somebody there was a poo-poo butthead. It was like, nope, I can't possibly get this thing a five star. It's a four star. Nothing's perfect. That's right.

I only kind of hit on the top of the iceberg or the tip of the iceberg. There's all sorts of stuff. There's UI support for adding requests and stuff. So you don't have to worry about like, oh, let me go find an example of a post. No, just click it. Also, if you've got a curl command, you can convert it. You can just paste it in there and it'll go ahead and convert it into the format. But really, it's one of those things like once you see your first request, you're like, oh, I know how to do this forever.

Yeah, I'm never going back. Right. They've got making GraphQL requests in here too. So if that's your cup of tea, bring it. As much as I like Postman, like when they went to the standalone product and they tried to create profiles and folders and save everything, it just become, I'm not a huge fan anymore.

Yeah, log in and do this. And like, no, I just want the stupid little thing that it used to be. Why is it so complicated? For real. Okay. Alright, so for my 35 tips of the week. Okay, so starting with, I thought we would start with iTerm 2. Because I don't know that we've ever discussed I term two. Have we? Oh, it's totally been on our like yearly developer

tool things. Okay. Okay. So for those that don't know, I term two is a terminal for Mac OS. Now here's why you're like, well, wait, Mac already has terminal. Why would I want I term two? Well, to me, I don't know what your justifications are and I'll let you speak to that. But for me, the big selling point was that like, I can open up multiple pains inside of the same tab within it. Whereas like if I'm using terminal and I want to open up, I can open up a new tab easy enough, but I'm going to have to like flip between all of them in order to see it. And I might want to have like multiple different panes open at one time.

And so that's the beauty of iTerm too. You can easily create these new panes. So if you want to, you open up that first one and now you want to open up a second one to split vertically, you can just command D and boom, you get an, it'll, it'll evenly split whatever the window size is. It'll evenly split it vertically. And you have two side-by-side paints that you could execute commands. And so like maybe one of them is like, uh, you're using, uh, Jay-Z's favorite tool canines, you know, you might have that open in one so that you can kind of like monitor your Kubernetes cluster while in the other one you might be executing commands and, you know, scaffolding up an environment or whatever. Or you can split horizontally if that's your thing because you're a monster.

Then you can command shift D. So just add the shift to it and it'll split a window horizontally, which, yeah, I mean, I'm just joking. There's valid use cases for that. And then if you do have multiple tabs open, that's great, too, because you can command and then the number for the tab, and it'll show you the number at the top of each tab, like which, which one it is. But what it doesn't show you is that if you were split the, uh, a single pane into multiple, then you're like, well, how do I navigate around? Well, it's pretty simple command option. And then whatever your appropriate arrow keys were would be. So command option left or right or up or down, depending on how you ended up splitting it. So, um, pretty cool.

Uh, if you've never tried it, then, you know, give I term to a try and you'll fall in love. All right. So then, uh, my, that was, that was like 33 of them. Um, so then for number 34, I guess I fell short on 35 math, whatever, uh, is a hack that I had where like, we've talked about my love for my, uh, ridiculously oversized. Um, it's not even a mouse pad. It's a mouse mat because it's so glorious and awesome in its size and you like Marvel and its beauty. But, um, you know, I was using my laptop as, you know, not connected to anything else, just, you know, laptop, but like specifically on the, you know, some of the max, right? Like there's, if you ever noticed like where it tapers off on the bottom, like there are like, you know, vent holes for it to, to bring in fresh air for cooling and whatnot. But I kind of wanted to like bring it up off the desk a little bit since it was

on this like fabric instead of on like a smooth desk. If it was on a smooth desk, maybe it would have like, you know, there'd be less friction so that the air would like flow and good enough. And I was in, I wouldn't care. But in this case, because it was fabric, I don't know, something in my mind was just like, Oh, well that's going to slow down the airflow. So I need to improve the airflow. So I came up with this silly hack because I was like, you know, what would be cool for, uh, laptops in general. And I'm surprised that this isn't a thing like a, a, a widely selling thing. Like, you know, I mean, there are some examples of what I'm going to say, but it's not as widely popular as what I would have expected and no, like one winner from what I researched, but just a simple, very short wedge kind of thing to where it, you

know, just, just picture like literally, like if you were to make this thing out of wood, right. You know, just a simple wedge that you would put on your desk, and it would be short too, no taller than an inch. We're talking about really short. And then the laptop sits on top of that. And then that way, the laptop is kind of at an angle, but not enough of an angle to where uh typing on the laptop's keyboard would be bad for your wrists right but just enough so that it would kind of like raise it up off the surface and it could breathe a little bit better and get some cooling and everything so my my little hack for this was i decided to use an architect scale ruler

have you ever seen an architect scale ruler. Have you ever seen an architectural scale ruler? Do you know what I'm talking about? Is that the three-way one? Joe for the win. If you've ever seen the, you know, if you were looking at it from a profile, it's like a, you know, one of those throwing star triangle things.

But, you know, yeah, this is the three-sided, uh, ruler. And so I caught an architectural scale ruler, but that just happens to be about the size. Cause these things are just over, uh, 12 inches in a little bit of extra, uh, to where you can set your laptop on it in the feet that like the rubber feet that are already on it you can put it to where the rubber feet catch that and then it it it's just enough off off the table to where it can get a little bit of airflow and you know my uh my my cdo is okay you know it's awesome you say that because I was just showing, I have a piece of wood trim that I have on my desk sitting underneath my laptop

for the same exact reason. Yep. So you put it at the front or the back? I put mine in the back because that's where the vents are. Okay. Yeah. Well, the vents are on the sides. Well, not on my Windows one.

Oh, okay. Yeah. So I have my Mac up on a stand that is next to my monitor, and then I have my Windows laptop that is down below it on the desk that I have propped up on the trim to give it some airflow. But, yeah, I mean, same type thing. It's a simple thing. You don't need anything super expensive. Just raise it a little. But yeah, I mean, same type thing. It's a simple thing. You don't need anything super expensive.

Just raise it a little. But yet it's so surprising. Because when I had this idea, I was like, surely there's got to be something. And this idea came to mind, and I'm like, no, there's got to be something better. Because there really is an opportunity. Somebody from Shark Tank is going to make this. Because ideally, you would want the top of that thing to be uh a little bit more grabby you know so because especially like not all laptops are going to have the same feet as the laptop that i have so that you know it might not work as well

but if it was a little bit more grabby than who would care right right but the reason why you want that that ruler you want the that kind of width is because if it was too narrow, then the laptop would kind of pivot back and forth. You know, as you like have both hands on it and then you take one hand off, then it's going to like pivot down on that hand that's still on it. Right. And you don't want that. You want it to, to be sturdy. Totally. So silly little life hack. I like it. So I was going to share this with you guys earlier because, all right, so first let me set the backdrop.

I've been working on a very enterprise-y application of late. And by very enterprise-y, I'm talking heavy dependency injection, heavy configuration files everywhere, heavy use of factories and things that you just can't follow. Like you can't navigate to anything because it's all abstracted by 50 interfaces, right? So me and this other guy, Sean, are working on this thing. And he's like, man, I found something you're going to love. And there is on GitHub a Fizzbuzz Enterprise Edition that is springified with all the glorious, hey, you know, this is supposed to be a bit of a joke, right? Like down at the very bottom, they say under the contributing, hey, although this project is intended as satire, because FizzBuzz, we've talked about it in the past, it's super simple, right? If the number's divisible by three, write Fizz. If it's divisible by five, write Buzz.. If it's divisible by five, write buzz.

And if it's divisible by three and five, then write fizz buzz. That's the application. And print the number. Oh, and print the number. Else print the number otherwise, yeah. So if you go into the source, man, it's so great. If you go source, and then they've got main, Java, column, serious company, business, Java, fizzbuzz, package naming, package, right?

So you've got 5,000 directories nested. Then you go into the impl directory because, of course, every good Java app needs impls. Then click on the factories folder, man. It's hilarious. You've got the buzz strategy factory, buzz string printer factory the buzz string right returner factory like no joke would you get into enterprise software this is the kind of stuff you'll run into because it's like we can't have any hard new ups of of classes anywhere it's all going to be done via auto wiring and di and spring. And it's mind numbing.

Like it really is. So if you want to see what a real enterprise application looks like, it's funny that they did it here, but this isn't far off guys. My eyeballs are puking. It's, it's, it's awesome.

So that was funny and truly excellent. So thank you, Sean, for that. And then I was looking around in our amazing Slack channel because we really do have just awesome people and content all over the place there. So if you're not a member of it, hit us up and send us your email, DM it to us. We'll add you. But our buddy, Jamie Taylor, who does the.NET Core show, he also does Tabs and Spaces. He also does the Waffling Taylors podcast.

Like the guy, he's into the podcasting world for sure. He shared something that is really cool. If you're doing local development and you ever need to do like secure socket stuff, like TLS communication or anything, you always need certificates. Somebody wrote a follow. So tile, I don't know, wrote this thing called MK cert, and it's a simple tool for setting up certificates on Mac and windows for developers. So you don't have to do any configuration. You don't have to do anything hard. You just basically say, Hey, I want to create some certs that I can use locally for doing HTTPS type stuff. Right? So, um, have a link in the show notes for this, but it might be a way to ease some pains when you're trying to do some stuff locally.

It basically sets up your local system as a CA and installs a root certificate in your local systems trust store. All for you. And if you've ever done this on your own, that's not an easy set of steps if you've never done it before. So being able to do this with a mk cert dash install is pretty beautiful you could brew install this thing isn't that awesome or app install it depending on your flavor i wonder do they have a chocolatey they probably do have choco install mk cert there you go man like this is like two lines of anything on every single OS. Like that's absolutely beautiful.

So yeah. Um, thanks to Jamie for sticking that, that out there in our tips channel and our coding blocks like, so, um, that's all I got this go around. Well,

let's go play some video games. What do you say? It's early, man. It's only midnight. I'm going to play some Mortal Kombat? Did you know that Mortal Kombat was actually based on a Scandinavian church song?

No. Did you know that? Hold on. I can't think of the song right now. Dun, dun, dun, dun, dun, dun, dun. I don't get it. It's a Finnish hymn.

Oh, geez. That's really good. I'll sing an actual song. Mortal Kombat. It's a finish him. Yeah. Geez. That was from,

uh, Gregory. Wait, I think his name came up before and I think we decided he said it's just pronounced Gregory, right? Yep. Not Greg. Greg.

Greg. Okay. I just call him Greg. Okay. All right. Well, thank you, Greg. Well, the Slack name says Gregory, so I don't know.

Yeah, that's what confused me. But at any rate, I wanted to say thank you anyway. So I'm sure by now he's figured it out and he's like, great, thanks for butchering my name. Right, yeah. Naming is hard.

Alright, well with that, we hope you've enjoyed this show on multiliter replication. And subscribe to us on iTunes, Spotify, Stitcher, in case if you just happen to a friend hand you a link or a device to listen to, you could subscribe to us if you haven't already. And as I mentioned earlier, we greatly appreciate all reviews that we get. So if you haven't already, you could leave us a review at www.codingblocks.net slash review. And if you have already left us a review, maybe there's another destination that you haven't left us a review. So you can leave us a second one. And that would be your way of giving us a cup of coffee or something.

Totally. Hey, and while you're up there at our site at CodingBlocks.net, check out our amazing copious show notes, examples, discussions, and more. And send your feedback, questions, and rants to our Slack channel. And as mentioned, because our Slack thing is busted, just DM us with your email and we'll get you added to the channel. And we're also on the Twitter. We're on the Twitterverse.

Yep, that's right. I tweeted earlier today even. It wasn't a very good one, but, you know, some of them are. So you can go to Cooking Vlogs on it and you can find all our tweeters and other social links at the top of the page. I gotta find this tweet. What was it? Is it really... I think I was flirting with other podcasts again.

I do that. Oh, is that what it was? Okay, moonlighting. I see. Yep, yep. So yeah, stay tuned. I may have another appearance coming up. So, you know. Just saying. You're such a flirt.