Coding Blocks - Multi-Value, Spatial, and Event Store Databases

Episode Date: March 4, 2024

We are mixing it up on you again, no Outlaw this week, but we can offer you some talk of exotic databases. Also, Joe pronounces everything correctly and Allen leaves you with a riddle. The full show notes are available on the website at https://www.codingblocks.net/episode229 News Multivalue DBMS Spatial DBMS Event Stores Resources we like […]

you're listening to coding blocks episode 229 maybe subscribe to us and listen on itunes sputify and more using your favorite podcast app and leave a review if you can and check out the website codingbox.net where you can find show notes discussion examples and more how about that leave your feedback questions at comments at code or your whatever at comments or whatever at comments at codingblocks.net and uh yeah follow us on x at coding blocks and with that i'm alan underwood i'm jerzak and outlaw will be back probably next episode so again we're having to go back and forth everybody's running through things right now so so yeah in and out in and out but we're trying to keep it coming. So we are continuing. This is our third episode on different types of database storage. And again, this is thanks to Brantley asking the question in Slack

to talk about the various different types and why you'd use them and why you'd use one over another or whatever. So with that, before we jump into into it we always like to mess up some review names so and before before we go too far i i'm going to do the first one only because we've said it like this is the third time now ivan kuchin he has now left us three consecutive reviews so he currently is at the top of the leaderboard so with that jay-z you want to do the rest of these all right we got thanks to yoon doggie uh also we got psycho duck and uh oh uh nihira gold i apologize that is surely not correct. In fairness, they said they were looking forward to however this was pronounced. Pronounced. Wow.

Golly, this is going to be a rough episode here. Pronounced. Yeah. I hope it lived up to your expectations. That's beautiful. And I assume Orlando Code Camp's already gone down now. Yep. Did you hear anything about it?

Did it end up being a massive success? Yep. I heard it was fantastic, and I'm not surprised. All right. Excellent. Well, Jay-Z is now fully moved into Georgia again. He is now right down the road from us other guys. So, yeah.

Welcome back. Yep. I had to figure out how to uh turn off the water outside it's like it was instantly it was just hot weather getting down to 20s we're like what do you what do you do googling like what do you do in cold yeah that's good times man yeah you know it got up to 70 something and now we're back down into the 30s and i think it's supposed to happen again here in a week so yeah i didn't even get that cold like i was being uh we were being overly prepared yeah florida people scared the cold yeah yeah although i will say it's so much less humid here that like a

georgia 40 is like a florida 50 i don't know why that yeah i don't know why the cold the what the what makes such a difference but it's ridiculous i'll't know why the cold the what the wet makes such a difference but it's ridiculous i'll go outside i'll take the weather be like oh it's 50 outside let me get a jacket i'll go outside i'm like nope throw the jacket back in like yeah totally fine here now when you dip down into the 40s it's a little bit rougher especially when the sun goes down it's crisp but yeah oh one other thing so i mentioned it's funny i mentioned that i was planning on redoing my wi-fi and i had mentioned like a few different companies like there was unify omada which is tp links

business one and then also a company called alta labs and it just just to give a quick update on this i started looking into alta labs and it was really promising like their performance was up there tied with the omada stuff and quite a bit better than at least some of the unify equipment but reading through their forums like they seem to have lots of firmware problems like when they update their firmware they end up you know causing iphones not to connect or andro's not to connect or whatever. And I, and that's not unique to a newer company. Like I had the same issue with Orbeez over time, uh, the Netgear Orbeez. But, you know, when you see that many, that many complaints about, you know, Hey, the firmware update screwed up my wifi. Now my whole family's mad at me. I ended up going with the more stable route of the TP link of modest because

it's like, man, they're probably going to put a lot more resources into making sure that my family isn't mad at me all the time. So, yeah. So I haven't installed it all yet because I have lots of walls to go through. Um,

but it's, uh, it's happening. I've bought two of the access points i got the long range access points and and i got a poe switch so we'll see we'll see what happens hopefully that'll be coming up here in the next couple weeks i got some new uh networking hardware too i'll tell you about in the tip of the week section oh nice excellent all right well i guess let's uh well wait what were you about to say something it's not that good don't get too excited we'll do it yeah it's mid

it's mid mid mid tier like medium oh okay okay all right all right let's uh let's do it so the first the first database type we're going to talk about, we only have three left. So this is this is the last episode in this whole database exploration thing. And by the way, one of the reviews said they were surprised that we didn't mention acid compliance type stuff with the relational database things. I guess we could have dove into that a little bit. I mean, I don't know. I guess that like one of the things that we didn't mention is like when you get into, uh, like Cassandra and those types of things or distributed databases, a lot of times you don't have that asset compliance to the guarantee your rights and all that kind of stuff. And that's one of the big selling points of relational databases. But that's ironic too, because like my SQL has options that aren't asset compliant, right? And, and it's one of the reasons why a lot of people have gone with like Postgres in the open source realm. Um, but like SQL server, Oracle, those types of things, those are asset compliant. And it is one of the reasons why a lot of folks will, will lean that way is because they want those guaranteed rights.

So. And as that is kind of a loaded term now too, because the databases are so complicated and configurable and tweakable that you can take a database that's acid compliant and say, allow dirty reads or something. And it's like, now it's not. And so,

and there's other databases that are a little bit more looser, you can tell them to be strict and so the whole like every database now basically has asked every database that claims to be acid compliant has an asterisk by it anyway yeah and we talked about this in one of our designing data intensive applications things where it really was like the industry has hijacked that term over time that made it less clear as to what they're actually trying to portray with it. Right. It was more of a marketing term at some point is I think basically what we ended up saying.

Yeah, totally. Yeah. I remember there was a time when we had to do some compliance at a job site and it was my first experience with like kind of doing one of the heavier duty or heavier duty levels of compliance and i was like okay cool let's get the rules we're gonna make this you know perfect we're gonna get it all checked off it's great and uh what it was at the end of it was basically like here's a list of things you have to do great

it's ambiguous and at the end what we really need to do is sign this so we know who to sue if there's a problem and we inspect and we don't agree with your assessment of yourself. They don't send out special investigators unless you go hire a third-party company. For the most part, it's kind of an honor system, which is really scary. But there's an honor system with liability there. So if something goes wrong, you know, that stinks. So, you know, if a database says we are asked compliant and people find it to be the opposite, it's going to, there's going to be a stink. Yeah. MySQL. Right. And, and not, and that's not to completely dig on MySQL. The thing is they allow you to choose different database engines backing it. Some are asset compliant, you know, quote unquote, some aren't.

Yeah. So, yeah, I mean, it's, it's not a super easy thing to talk about with actually diving deep into each particular database system. So,

you know, um, a good call out in the review, just, uh, wanted to at least share our thoughts on that a little bit. So by the way is a default now to nodb so it defaults to the more

compliant engine oh okay it used to not right like that's been a while back i mean i don't really look at it that much anymore so um yeah all right so first up for this episode is the multi-value database management system so i i thought i'm I think I mixed all this up in my head when I first started looking at it because I was thinking the multi-purpose one like Azure Cosmos DB, right? Like where it does everything. That's not what this is. So the ones that we can call out that they say are the most popular ones are Atibus, which is number 86 on the list. Unidata slash Universe. I guess I could say Universe and Unidata instead of Uni.

That's number 87. And then JBase is number 147 on the list. So this one's interesting. They say it's similar to a relational database system which means that they store data in tables so that's important right which i looked and i could not find anything that says that it was like loose schemas like schema on reads versus schema on writes but the fact that they say that it's in tables makes me believe that this is some sort of schema on right type thing. Yeah. Makes sense.

So here's, here's the big divergence from a relational database. They can store multiple values in a particular records attribute. And to me, all that means is basically they can store like arrays, right? So if you have an address column in your table, then you could store an array of addresses, right? So if you have an address column in your table, then you could store an array of addresses, right? Like that's basically what they're saying with the multi-value thing.

And they also comment on this, and I'm curious what your thoughts are on this, Jay-Z. So there's several databases out there that could do this type of thing, like Postgres. You can do all kinds of crazy things with Postgres, right?'ve they've extended that thing to do whatever you want and they say that there are relational databases that could do the same thing by storing these multiple records or multi multiple values in a single record but they say that's more of an exception to the rule like you wouldn't typically store an array in some sort of attribute like what are your thoughts on that? Yeah, I definitely agree with that.

And I was just reading a little bit about the, like the wiki page and kind of the history of how these things came to be. And it's got an interesting kind of tie in with some medical records. And it's basically deals with, or kind of came out in part related to just having incomplete data and just wanting to capture whatever you've got, whenever you've got it. Imagine when you go to a doctor's appointment or you get some blood tests, like sometimes you run this test, sometimes you run that one. Sometimes the tests that you do for certain things might change over the time. So the,

the, you know, the values and the meanings of the, those numbers might change. And so, uh, you just want to be able to kind of collect the the made it in the metadata as you need it and then um you're like it doesn't really make sense in like an olap type situation where you really want to do like hard introspection and counting of numbers and aggregations but it's really nice for like pulling up information on like a person a patient you know that's pretty interesting So with what you're reading there, I mean, it sounds like, and that makes sense for the medical industry. It sounds like their whole purpose was, Hey, we know we're going to have some data in here and we, we want to just

make sure we cram everything into the single record so that, so that it's all in one place. I guess my question is being that it seems or appears to be like a schema i'm right type thing are they saying that the schema for these multi-valued attributes are more loosey-goosey like you can sort of cram in there what you want or or i don't know if it's it like like i said i couldn't find anything that like really defined it super well they say it aligns well with jason xml so i'm guessing that it's kind of a little bit more loosey-goosey about what you can put in there. Okay. That makes sense.

That's what I was assuming. But when I looked at the code examples that they had for some of this stuff, it was not clear exactly how it was going down. I got something good here. So in the Wikipedia page, and remember, this is generalized, but just kind of talking about multi-value databases. We'll have a link in here they say generally speaking in a multi-value multi-value database system a database or schema would be called an account a table or collection would be called a file so one account could have multiple files a column or field is then called an attribute, which can have multiple value attributes or sub value attributes. And then a row or document is called a record or an item.

Yeah, I did notice that in the code, like everything was referred to as files and stuff. And it was like, OK, that's a little interesting. But but being that you said that it sort of came out of the medical industry, that sort of makes sense, right? They keep a file on, on a particular customer or patient. Yeah, this is pretty interesting. Yeah. All right. So, so here's what's interesting is they say, you know, you typically wouldn't do this in a relational database system unless you just needed to right like you had some sort of reason to divert from

the the table column type thing in a multi-value database system that's how you should do it they say that you know that's the whole point of it is you have that one thing you can create multiple multiple values into those and that's that's how it's expected to be done. And they said the primary reason for this, and this is coming from the db engines.com summary. The primary reason they do it this way is because these multi-value databases are not optimized for joins. Unlike a relational database where you keep things in normalized form and then you join to get that data. They expect that data to be on that record, that file, that whatever it is, right? Yeah, it's really interesting. I'm kind of comparing this to like a Mongo type situation where, you know, you've got these documents.

And so you can imagine having a situation where you've got documents that models model files and each file can be radically different from another. But in this case, it's it makes sense where you're like really dealing with like one person at a time so you're like you're seeing a patient you say let me pull up your account okay here's all your info let me look at your lab results from yesterday and compare to lab results from a year ago and even though you know they're we're comparing these two things like one has different tests that were on the others but some there are some things in common you can do all that stuff in other databases but it just kind of seems to really fit really well with this kind of notion of you're kind of taking on one big subject or one account at a time and then spidering off to there and and really kind of working with like unstructured data with a kind of a person there

it's cool so it's interesting because you mentioned mongo and that's sort of where my head floats to with this is it seems like a mixture of hard schemas but then the new the loosey-goosey uh embedded nature of a mongo right like because in mongo it's the same type thing like that you're not really supposed to go do joins on object-oriented databases type or that's not what they're called what were they called object oriented databases were different whatever mongo fit into yeah whatever that one was it reminds me of that except with mongo like you can create a collection you put whatever you want in it right like there's there's you could put a car in there and then you could put a person in that same collection. In this, it sounds like more of, hey, this is what a patient is. And then any of the details you want to fill in on that, just drop it in that same, in one of the attributes, right? So

that seems to be sort of the defining difference between those types of databases. You know, it kind of reminds me of like a old school filing cabinet where you might like have a file for your tax return and you put all the papers that you got in for that year's tax stuff and you might have another one for appliance warranties where you put expensive appliance warranty

information all in one folder or you might have something else and the data in there is mixed but a human can go in there and kind of quickly figure out which folder to pull and then kind of manually sift through the documents to find the thing that you need. That's a good example. And one of the other things that they said on the site is they said the primary reason that these or the primary selling points of these things like the Atabas and the Unidata is the rapid application development and the ease of learning. They said

that it's really quick to get up to speed on these things. So it's weird, like over time, I'd never really heard of these nor considered using any one of these because there's just so many, I guess, standard choices out there, at least in the business world of going like, relational database or Mongo or something like that. So it's interesting. I still don't know that, you know, if I would pick it up to use it because it's, it's very niche looking, especially when their highest ranked one is number 86 on the list, you know? So I don't know, it's probably worth looking at. Um, and i'm sure there are like the the medical thing there probably are very good reasons to use this but i i personally have a

hard time seeing that this wouldn't fit into something like a mongo if you wanted to go that route yeah so anyways they do say it's very performant because the data is the way you expect it and it's easy to maintain which is a positive because not necessarily are you going to get that with some of the other larger database systems right like yeah for sure if if you're a dba or something you've you've seen that sometimes these things are a pain in the butt to keep alive and working perfectly yeah yeah for sure it looks like Adabas is actually developed primarily for using on mainframes. Like it's old. It's real old. So yeah, you definitely don't want anything complicated when you're doing mainframes. Next up, this one, this is what I thought the vector database stuff was going to be like, and I was totally wrong. We went over that in depth last episode, spatial, excuse me, Edward didn't want to come out spatial database management systems.

This one's kind of interesting. So first let's start with the popular ones. So number 29 on the list is post-g gist which i think is funny because it's actually not a standalone thing it's a plug-in for postgres and then the next one is number 59 it's aerospike and number 136 is spatial light and i'll let you start a little bit on this one sure uh so i think it's gis by the way it is yeah post gis not post gs yeah yeah i was like sometimes like i know a guy who worked in the industry that's the only reason i would even uh think that but uh he worked for basically a company that uh tracked like um cargo uh like across those moved across uh you know states and countries and uh deal with these kind of databases

is that they provide the ability to efficiently store modify and query spatial data so it's data that appears in geometrical space so like a map or a polygon or something and they generally have custom types data types for storing spatial spatial data i was just doing a little bit of googling on this um i used to work way back in the day in uh mortgage research and a big part of that was finding if there were any liens or any lawsuits or just any sort of problems with the property that someone was going to buy because a mortgage lender didn't want to lend you money to buy a house that you didn't own or had some sort of big problem with it or was, you know know had a lien on the property meaning someone

owed money on it and so what i would do is like go through these maps and figure out like where the properties were based on legal descriptions and then actually like do searches for these legal documents based on the property location so what that meant is we would have these big county maps and i would figure out the plot if it was like in a neighborhood or if it was attractive land, it would kind of run out the little legal description like 29 degrees west, whatever. And we'd figure out what we called ARBs, which were arbitrary grids. So it would be like, imagine like a big piece of paper and you divide it up into fourths. And you have one, two, three, four. You take each of those fours, do it again.

Do it again, do it again and you do it again do it again and then you can get to a really small tract of land by saying uh this is in uh arb four four three four of similar county and that meant like the southwest corner of the southwest corner southeast corner i forget what numbers i said now but you know what i mean so uh you can get really precise with that and that's kind of similar to like the the underlying data structure that's used for a lot of these uh with the quad trees except that this also supports 3d space which is like a whole nother kind of twist on that so much more complicated and i wouldn't have been wanting to do that stuff by hand and y'all talked last time about how three like 3d from 2d is a big jump up and then going to 4d and beyond is just forget

about it but But these databases are basically built around the ideas that power that stuff. So it makes it easier to find things that are physically or spatially related to each other. So mapping data

and stuff like that. And you can imagine topography matters in there. Even as I'm talking about, it makes it sound like you might think, data and stuff like that and you can imagine like topography matters in there like um even like as i'm talking about like it makes it sound like you might think like oh this city in that city or this coordinate in the back order but we're talking about like mapping data and like directions and roads and stuff like that you're talking about a whole many many many more points of data and so if you want to be able to find the fastest route between two places, or if you want to find the most efficient route between 40 different places and, you know, adding fuel and maintenance and all sorts of other kind of considerations that you need to have in there to make sure that your logistics network is going to be able to kind of function. And you can imagine how that gets really complicated.

And that's something where you'd probably want to get one of these heavier databases in here so that's all super cool and you actually hit on a few points that that talking about the technical aspects of it is they talk about the fact that when you're storing things they typically store them in rectangles just like what you said right like you would subdivide your piece of paper into quadrants and then subdivide those into quadrants or whatever. And that's how they do it so that they can find overlapping points within it. So what you were doing manually back in the day, like creating that stuff in software, that's exactly what this stuff is doing. And then we mentioned that they have their own data types for storing this stuff. Really all it is is just shapes. So the data types that they store are points, lines, polygons, basically just any type of shape. Because when you plot those things, you can,

you can plot them inside those rectangles, right? Like you could say, Hey, this particular point falls within these two bounding boxes that I'm looking at. And that's how you can do the important part of this is that's how they can do quick finds on where these things exist. So instead of doing like and they actually talk about this in depth instead of doing a B-tree index, like what most relational database systems do. These are using what they called archery and quad tree implementations so that they can quickly locate points and the sectors or, or rectangles that they fall into. So it's, it's, I would imagine you have more experience with this. I would imagine it's complicated in terms of how it's implemented, implemented, but the use of it's probably not that bad, right? Like as long as you have a grid and these things are plotted in the right places,

it's fairly easy to use, right? Oh yeah. And anytime you do a Google map search and get directions from somewhere, you're basically using a database. It's something like this, is organized like this. And as a user, it's fantastic. I just type in two addresses and then I rely on the heavy work of the mapping structure behind. But just like when you do a direction search on your phone or whatever, it's not doing that ASTAR algorithm, you know, looking at every possible path between two things, including walking, bicycling, public transportation, buses, all that stuff. No, it's already figured out. The software, the service that you're using has already roughly figured out the most efficient paths for just about everywhere to just about everywhere. I say just about. It's like there's some rounding and some stuff. They take all sorts of little shortcuts.

So what that means is they basically have stored information about efficient routes already. And so that when you say I need to go from point A to point B, it can leverage what it already knows about how things connect in order to figure that out much quicker. And then it just kind of sprinkles on. I say sprinkle as if it's easy, but like traffic data and stuff like that in order to say like, oh, this one's not as great. So we're going to update those weights and kind of recalculate. But I think the whole subject is just fascinating. It's really cool it's a really cool space uh the whole mapping thing is really cool if you even just google maps and google earth in general uh and i know there's other services now but i mean it's just such a technical marvel on so many different levels and

it's so usable and so helpful and just the technology behind it and the user experience that everything about it is just like a masterwork yeah it's it's incredible and it's funny like when you said it's sort of easy to just break along some other things like i think about that it's crazy right like there's the way that the crow flies like you know for the software to be able to point the the quickest the quickest route between point a and point b in terms of distance. But then when you start factoring in, if you're, if you're looking at an application like Waze, that's factoring in traffic patterns and then speed limits along those routes and all, like it's insane, the amount of calculations and whatnot. So that goes above and beyond just the 2D space that's there, you know,

going along these roads and then having to add in all kinds of other stuff like it's it's insanely cool what they've done with all this stuff and this is all possible because of these these geospatial databases which are basically just a subset of these these spatial database systems so um really really neat things oh look at what database uh google uses for maps oh that would be cool so there's there's another type so i just mentioned the geospatial that's like you know places on earth or whatever there's another one called spatio temporal data this one's kind of interesting this is spatial data that has timestamps. Now I'm not exactly sure what that would be used for, but that sounds pretty cool. So that's, that's one that exists out

there. And then the other thing that Jay-Z mentioned is, you know, we were just talking about 2d space a second ago, right? Like you look on Google maps and you know, Hey, I want to go here to here. Okay, cool. Awesome. Well, if you're in Colorado, you might be going up and down mountains, right? And that's, that's a little bit different and they support that kind of stuff. here to here okay cool awesome well if you're in colorado you might be going up and down mountains right and that's that's a little bit different and they support that kind of stuff like they have support for weather and elevation and and all this so like post gis has all kinds of stuff built on top of it and i'm sure these other systems like aerospike and spatial light also do but you know it's it's super powerful with these

things present and give you the ability to do if you're working with spatial data yeah it's interesting uh it looks like aerospike i thought i'd heard of it before but and i would have been surprised if uh you know if it was just for gis stuff because because I just don't mess with that stuff. But it's actually used for other different type things too, like real-time database, kind of NoSQL type stuff. So I'm interested to see what kind of features they have in here specifically for this space. So that's pretty cool. They compared themselves to Redis.

Interesting. So an in-memory type thing. Yeah, that's interesting. Yeah, I want to i want to know more about aerospike that's that's why we do this stuff so a couple other things to hit on real quick like if you want to learn about this like if this is neat to you or if you have a need for it as a matter of fact somebody in our slack channel i don't remember who it was mentioned that they use postgres or they use a relational database when we were talking about the first set of things because it could do so much and that's that's one of the reasons why postgres is so popular is like this plug-in for it gives it spatial data right and you can do most

of the things that you need with your data using postgres even though things like search like an elastic search might be better for but you can fit most of those things into your Postgres. So that might be a reason to lean that way, right? But what I was going to say is they have a workshop page for PostGIS. We'll have a link in the show notes for it. But if you're interested, it's a massive workshop. It walks you through a ton of steps. So they've got, uh, the workshop modules is split up into 41.

Now the last three or four are appendixes, but if you really want to learn this and you really want to go through some stuff, like there's a great resource here. If you want to step through that. Yeah. So it's a super cool space. And I don't even really know what companies kind of hire and do stuff in this space but it's just cool yeah i agree it's really neat oh and it was also worth noting you've i'm sure you've probably heard of redfin seeing as how you've been going through you know looking for a house and all that kind of stuff so redfin was

sort of a a competitor to realtor.com and they used postgres and post gis for their stuff at least that's they have it listed on the post gis page so you know kind of cool like one of the major you know real estate companies out there doing things is using open source post g technology. So pretty neat. I see OpenGIS, you got a link here, which looks like it's got a lot of information that's available to you. Because you kind of imagine like if you're starting a business and you need to do GIS type stuff,

you pick your database. You also need a lot of data. This is kind of like an interesting situation where it's like you really expect your database to come either with a lot of data this is kind of a like an interesting situation where it's like you really expect your database uh to come either with a lot of data populated or you expect to buy a lot of data and kind of shove it in there to use it that's interesting i didn't even think about that when i worked at ups back in the day they bought data to fill in the systems right so when they were trying to map out routes for for their drivers

or whatever yeah when you think of a spatial database it's like oh yeah it should have everything about the u.s in here or whatever right no it doesn't you gotta you gotta get that somehow yeah you think about like redfin what they're doing it's like they you pull up a house on there of course you're gonna be able to want to see it on the map and stuff but then also like what schools are nearby what's your closest hospital uh who's your utility companies you know just like little dumb stuff like that it's all based on like physical location what counter you and what are the tax rates like and how you get to that the meta data all that stuff is really important when you're considering buying a house yeah and it's really cool too like

if you go into realtor.com like the app and you search for that stuff like what you just said hey show me the schools nearby schools or whatever it'll it'll draw them out on the map right like it'll draw the bounding lines and everything around it like it's it's amazing what they can do with these database systems so you know again if you're in that space like you should be looking at this type of thing and if you are in that space you probably already have looked at this so yeah you know walkability score have you seen that i have yeah yeah i think it's a zero where we look yeah i was gonna say like pretty like 75 of the u.s is just a zero so yeah it's not that impressive i guess yeah unless you live in a downtown area you're basically you got a less

than three probably everywhere all right so uh we we do this every time and and we want to ask again like hey we don't ask for much but if you wouldn't mind and you find yourself with a few minutes leave us a review say hi say thanks and you know leave a smile on our face and help other people decide whether or not they want to spend their time listening to one and two hour episodes of software talk so you know we we really do appreciate it ivan thank you for leading the leaderboard there that's that's awesome and thank you everybody that has taken the time over the years to to drop a line and say hi and say thank you we we really appreciate it love it thank you yes uh i guess we're not gonna do mental blocks it's hard to do with two

people yeah yeah stuff we gotta skip it um i think somebody given a suggestion for something else but i forgot it already because it's been a busy couple of weeks so um yeah we'll be back to our regularly scheduled um interruption in a future episode. So, all right. So I think you're probably pretty familiar with this one too, right? Yeah, a little bit. You know, like I'm definitely familiar with the space of, of, oh my gosh, I can't remember the word. It's like event stores is what we have in the, in the notes,

but I'm thinking event sourcing is like where my mind instantly jumps to. And that's a big use case for these kinds of databases. And you've got the most popular databases in here. We've got event store DB at, is that one 78? Yeah. One 78. It's not high up on the list.

We'll talk about why that's so far down, even though it's something that a lot of people have heard about three 36 for a DB two event store and three 38 for an event store. And remember, remember there's only like 400 total. about $336 for DB2 event store and $338 for an event store. And remember, there's only like 400 total. So these things are like bottom third of the list. Yep. And the idea here is that these are used for implementing the concept of event sourcing,

which we've talked about before. It's kind of like in your bank account where you have a list of transactions and you can total them up to see how much you currently have in the database but it's really important to have those transactions too so you can explain to a customer or if you're the customer you can see like where did my money go what happened here and you know that nobody's like robbing you or kind of sneaking anything anything bad is happening and yeah it's kind of we we talk about the notion of like replaying these transactions in time in order to see how you got to where you're at now. And when we talk about where you're at now, we often refer to that as materializing and saying, we because the data is immutable, right? Like once it's written, you can't change it. It's written in stone, right?

Like that's what it is. Yeah. So if I had to make a database, it might be this type just because I'm lazy and I don't want to deal with updates. And even querying is kind of limited because in a lot of ways, it only really makes sense to just kind of loop through from the beginning to the end of these things yeah it's it's pretty interesting i mean they they call out the main contrast like most people think in terms of relational databases like that's you know that's probably the standard that most everybody starts with and the big difference is a relational database stores the current state of a record right like that's the whole purpose of it if you know if jz has a record and i have a record it's going to

tell our latest address in there because that's what it is right and if you want to store historical data you can and there's all kinds of ways to do it like for instance like sql server has temporal tables right where you can basically do a few commands and you can hook up this, this sort of historical tracking behind the scenes. So every time you update a record, it throws a history record into this temporal table. You can do it manually, right? Database systems that don't technically support, you just create a history table or an audit table. That's basically a complete mirror of the main table. And then you could do it, but that's you sort of creating your own solution. You're having to maintain your own inserts, updates, triggers, all kinds of things, right? Like that's what you're doing. This,

this is how it works out of the box, right? Like you, like Jay-Z said, if you have a bank account and you deposit something and then somehow money leaves your bank account and you didn't do it, nobody's going to be able to go back in and update that and say, no, that didn't happen. What they'd have to do is they'd have to add another record to move the thing back in. So you'd actually see the flow of exactly how everything happened, right? So it is truly just a living audit history of every change that happens to an entity in your system you imagine like uh you've probably like you can either imagine or you worked at a place where you had a relational database and someone said they needed history so

you made an underscore history table or you turned that feature on the database and then someone said oh we also need this table now you've got two tables like underscore history you've got those features turned on too and next thing you know it's like you're up to five or six tables and then somebody says like hey what did this person do around this time and you're doing this weird query where you're unioning stuff and doing aware on this and unioning the results of this and like looking at all these tables like that's the kind of stuff that's like really just natural to these kind of databases and you can imagine for like a security perspective on how that's really important to say like hey um are was there an abnormal amount of withdrawals within the last 24 hours like that's something that's you know kind of easy to do here like is

this person having a flurry of activity more than usual you know like fraud type situations or accounting uh so it just it just works well it seems really great. But why don't we see, well, I guess we'll get into, but why don't we see more databases that are kind of designed around this notion? I think it goes back to what we said. I think it's the reason why everybody sort of falls back to relational databases because it's not a join system right yeah it's it's here's here's my thing and then follow this thing throughout time not hey i need this thing to join to a person into a location into a whatever like it's just it's not made for that right it's made for tracking the state of objects over time yeah so it's like by definition you need another database next to it and you can

also just do this in that database too you just have to be a little bit more careful about your use cases but for the most part it's kind of defined by what you can't do like you can't you know you can't query it uh it doesn't make sense to really query it too much independently and you can't change stuff like i can i can implement that done right right yeah i just have to not do stuff that That's great. It's super important. Like what, what are you saying? You can't change stuff. You technically, you are not allowed to update an event that was there and you cannot delete an event that was there. So it's, it's sort of simple in terms of what it provides you, right? It inserts new events and it allows you to query those events over time. That's, that's kind of the whole gist of this. Now this, this is where we've always had conversations, like, especially with the

designing data intensive applications is, you know, when do you decide that you're going to add another piece of tech to the stack? Right? Yeah. I can, I can add a bunch of underscore history tables to my Postgres or my SQL server or whatever, but that gets nasty over time. Just like Jay-Z said, right? Like now you're going to union your main table with the other table and you're going to have all these things, all these query criteria on these joins of these unions to make things work. And it's like, well, if you just had this thing over here, that's just purpose built for exactly what you're trying to do, then that's cool. But I'm going to save some of that

because there are some really cool reasons why you would use this over something other than a relational just as a as an augmenting type technology yeah totally um now one one thing that's super cool here is in your mind i don't know if your mind goes here. Mine definitely did. If you have, if you have a bank account that has just, you know, tens of thousands of transactions over time, it seems like it's horribly inefficient, right? Like, Hey, I'm going to look at Jay-Z's bank account, but I've got to replay these, these 10,000 events to get to what the current statement or the balance is. Right. Like that seems wrong. Yeah, totally. Like 10 years, like if you if you maybe bank for 10 years imagining like running through every little transaction every little time you gained one penny in interest like all that stuff like no that's ridiculous so if we

could have this database kind of maintain the current total and be able to like truly rely on it so we can do an instant check at any point in time like what is my score what is my number now and that's great and they have that they they have this thing where they can't not all of them but i'd say probably a vast majority of the popular ones have this ability to store these snapshots over over time that materializes events up to a certain point in time right so so i don't know maybe say it's daily they do it um you know if you go to look at yesterday's thing it's got this materialized state so it doesn't know maybe say it's daily they do it um you know if you go to look at yesterday's thing it's got this materialized state so it doesn't have to replay all 10 000

events to get there so so it's been thought about right like it's not like everybody's like oh i'm going to take this hit every single time so that's good yep i see you got the event store db uh linked here you take a look at that i did take a little bit of that and they were talking about some of the information in there so like i did on on several of these databases when i was taking notes i would find usually i would go for the most popular one and if their documentation was good then i go with it and in some cases the most popular one had horrible documentation and i would find the one that had better documentation in this case event store db we did a pretty good job and so i grabbed some notes off their stuff right like one of the things that they mentioned is they sort of define theirs

as an immutable log which is perfect right like it and it's funny because it reminds me of kafka when i hear that but this is a lot more purpose built and i want golly i think it was jim hummelson had mentioned that kafka is not the same thing as an event log right like it's yeah and he had an entire write-up in our slack channel that was absolutely fantastic so don't mix two, but it is an immutable log. Yeah, there's a lot of stuff you can do with Kafka, but you are kind of swimming upstream. You're going to have to do a lot of configurations. It's easy to get wrong.

And then there's just some kind of weird stuff. It's just not a database ultimately at the end of it. But looking at EventStore right here, I checked out their API documentation, like right in the notes and documentation, like these are pretty much the three calls that you're're the only three calls you're ever going to have to make with this database one is appending the second one is reading which is reading the stream and you retrieve all recorded events and then the third one is subscribe so you can get like

new events coming in and you know everything else is kind of like you can but that's not really what we're here for it's pretty cool isn't it like it's it's the the definition of simplicity right like you get what you need and that's it now the the subscribe thing is actually pretty cool because they took it a step further right so the other ones are what we talked about you get your audit log you add an event and you can query the event. That's, that's, that's basically it. But this subscribe thing means that you can now have reactive applications, right? You have applications that respond to, Hey, there was an, there was an update event or a new event added to, to Jay-Z's bank account. I i can have this my application respond to that like does it

need to go do something because this event happened and we call the hollywood pattern something don't call us we'll call you oh yeah yeah golly that's been a while yeah it seems to be what we do in streaming technology nowadays like that's really what it is yeah polling stinks like let someone else lower level deal with that yeah for sure um let's see so yeah on the event store db like some of the other things that they call out is they have these guaranteed rights they have a concurrency model and they have these stream apis like what jay-z just said which are fantastic like give you tons of ability within your applications it's it's hard not to compare it to kafka because that's what kafka gives you right like kafka gives you the ability to write

these streaming applications that just respond to events that come in yeah totally and it's just uh it could be so many other more things like you're gonna you have no matter what you're doing with kafka you are gonna be tweaking configs and they're really confusing and weird, and they all matter, and you're probably going to get them wrong for a while. There's so many. There's so many. You can't get it all right the first time or the 20th. Yeah.

So the EventStoreDB, another couple things they call out. So it's available for.NET, Java, Go, Node, Rust, Python. I'm sure they have more. And they say it runs on all OSs, which is beautiful. Windows, Mac, Linux. Not everybody runs on Windows. It's funny. I started out as a Windows developer, and everything was Windows.

And then as you get especially more into the open source world, nothing runs on Windows. Or it's jumping through hoops to or it's or it's jumping through hoops to do it so it's nice to see that they actually do it for all yeah and it's open source too oh right yeah uh this runs on a cluster so it can be highly available which is beautiful they have these optimistic concurrency checks beautiful this all, so this is the part that to me is so super cool. So we've talked about the audit history. Beautiful.

We've talked about the fact that it's simple. Beautiful. This is what takes it to another level for me. And this, this right here, if you were deciding whether or not you wanted to write some, some audit temporal tables in your relational database versus using something like event store DB, this would be the defining factor in my mind, whether or not it was worth adding this to your, to your tech stack. So they allow projections based off new events that come through. So they call it interesting occurrences in your data right and i'm just going to read this example verbatim because it will explain it better than i can and

then we can talk about it afterwards so uh let's say that as a medical research doctor you want to find people diagnosed with pancreatic cancer within the last year during the treatment a patient should not have had any proxies for a heart condition such as taking aspirin every morning within three weeks of their diagnosis they should have been put on treatment x within one month after starting the treatment they should have failed with a lab result that looked like lab one within another six weeks, they should have been put on treatment. Why? And when, within four weeks failed that treatment with a lab result, that looks like lab two, that's a crazy amount of criteria that just happened right there. But what they're saying is with event store DB, it allows you to look at the events that happen over time

and use these kinds of criteria to come up with new things that you maybe want to publish to another event somewhere. Um, another simpler example they had is you're looking for how many Twitter users said happy within five minutes of the word food, coffee shop, and within two minutes of saying London. So you can sort of stitch together these complex streams of looking at data and saying, hey, if I see these things happen, then that means that this needs to happen and it can fire off an event and add, add something to another, um, uh,

object. I don't know, event stream, whatever, but that's amazing. Like you're talking about analytical type stuff that's built in just based off the event streams that are coming through. Yeah,

that's really cool. I was like, when I first read that, I was thinking much more simply. It's just like, if there's transactions greater than a thousand dollars and send it to you know potential fraud whatever or uh if there's more transactions and you know x amount of seconds or like if the account goes negative send it to the billing department or whatever

but yeah these are much more complicated and interesting yeah it's i mean i was thinking so we do work like this right like we do work where where you're investigating and trying to tie together events that happen over time and across systems and and all kinds of things and we use different technologies for it right like we use apache flank and kafka and we're doing streaming stuff. And it's really incredible what you can do with it. But it's also not super easy. When I say it's not super easy, you're having to store a ton of state. So, for instance, if we back up here. In order to do this, the reason this can do it is because it's storing those incremental

pieces of state over time, right? Like, Hey, this person, you know, got treated for, for this. And then they took this medicine and their lab looked like this, right? That can all happen because it's storing each piece of that. And so it can look at it and say, Oh yeah, I saw this, this pattern happen. And it's all in the data. say, Oh yeah, I saw this, this pattern happen. And it's all in the data. It's in that, that history data, when you're doing this, something, this type thing with Apache flank and Kafka and all that, you got to store that data somewhere. Right. And by its nature, Apache Kafka is a, it's also an immutable transaction log. But the thing is usually the way that you're using Kafka is you're sending so much data to it. You can't keep that data for a long period of time, right? Like you might keep it for a week, maybe a month, but you're,

you're going to eat it and storage costs because you got a lot coming in there and a lot of people are using it. So you're having to store that state now, not in Kafka, because it's going to go away at some point. So you're having to store that in state within the Flink application, right? And I think behind the scenes, it uses like RocksDB or something. Another thing that we've talked about over time, but now you've got to start caring about each piece of state that's going to matter to you, right? So if you are trying to do something like this lab, this doctor thing where they're trying to find cancer, you have to know all those data points that you actually want to store in state.

And that's going to change over time, right? Like at some point, you're going to be like, oh, man, I needed to store this lab type result. And so now you have to modify your application and store that state. So you're not going to have any history for that, right? Whereas if you're using something like this event store DB, it's all there because it's all been there. So now you can just say, hey, you know, I found this new string of events or this pattern of events. You have all the data. And so you could probably add these projections in there and, and start looking at it as opposed to always constantly having to build new state and something like a flank application. And there's,

there's other streaming apps out there. But this seems like it's purpose built for that, that type of scenario that we're looking at. Yeah, it's good stuff. And it's highly performant. They mentioned here on the page, 15,000 writes and 50,000 reads per second, which is really, really big. It's not quite like New York Stock Exchange numbers, but I mean, it's up there.

Yeah. I mean, that's quite a bit. Yeah, it's pretty good. 15K writes per second. And I think they also also this is pretty simple to maintain as well so you know you're getting a lot of bang for your buck for it so to me again the the immutable log all that kind of stuff is really cool if you need if you need like audit history on a lot of things that you do this seems seems like a no brainer. But if you,

if there's that thing, that's the tipping point and this ability to analyze data as it comes in with these fairly complex or even simple type patterns that you want to see, man, that seems like a no brainer to me. Um, especially if it's really not that difficult to maintain. Now I've never run it, so I don't know. I think that's marketing speak, right? Oh, yeah, you can just set it up and it runs. Okay. Maybe it's like Kafka where, yeah, sure, it runs,

but you got to configure it 25,000 times to get it to be perfect. Who knows? Yeah. I just Googled what databases the stock market uses. Of of course the question is like kind of nonsensical, but I saw quite a few people mentioning KDBX or KDB, which you talked about last episode. It's interesting, but they, they, they mentioned it specifically for using for event sourcing. And that's kind of like kind of what I said before too, is like, you can implement this in other databases if it makes more sense for you.

Oh, that's interesting it's i mean is there anything we found that you cannot do in a relational database well uh the graph stuff is the only thing i can really think of you know and keep in mind but i don't i don't really think so but you know the funny part thing about the graph thing is like sql server bolted on graph database stuff in that by setting up unique, interesting type relationships. It's still not great, but, but yeah, I think wrapping up the, the three episodes that we said, and this is, this is what we've been, I guess, sort of preaching for a couple of years now, since we started the designing database intensive applications or data intensive applications is you can make a particular database your hammer. You can, you absolutely can. Every problem that comes in just looks like a nail that needs to get pounded down. There are going to be times that you look at it and you say,

wow, this event sourcing thing really seems like that makes a lot of sense for these use cases or this graph database, right? I need to see 20 levels separated of relationships. Man, you can't do that easy in a relational database. You could do it, but it's not going to be easy. So keep in mind that, that if you find yourself really struggling with keeping something performant or, or trying to do things and it's way harder than it seems, you might be trying to push the wrong thing and with your hammer. And that's, it's really hard. It's really hard to make that decision before you get into something right like i mean we've talked about trying to make a database a search

engine in the past and man you find out relatively quick once you get in there that it sucks yeah this is tough yeah so there's it's not an easy answer of, well, should I use this or should I use that or whatever? Right. Like, do you have a team that can support it? Like, do you have an SRE group that can, that can keep the thing running? Like there's, there's a lot of questions to be asked, but just know that there are just about now for any type of data that you're planning on using, even the vector database type stuff, right? Machine learning. I would have never considered it. I had never even thought about it

because I don't live in that space, but they have databases for storing that stuff, right? So you need to be aware of your use cases and whether or not you think that the added complexity is worth it, I guess. And you know what you always try out. We live in the future. You can spin up new databases now with Docker or

Kubernetes and give it a shot. It's so much easier than it used to be. Just installing SQL Server, you remember how many disks that used to come on? CD-ROMs. That's not something you would normally be doing. Just installing the tools to work with it was such a pain. Now you can just download all that stuff and run a container and then ditch it and it's just gone off your

system without changing anything messing with anything it's great oh that's so beautiful docker run it all right docker run it and if you like it use it make it part of your stack if you don't docker rm you're done it's beautiful that pretty cool. You can put a crappy movie on or a movie you've seen before and mess around with any database pretty much almost any database in the world at this point. That's pretty cool. And I think that's what you do most of the time, right?

You're kind of sitting there watching TV doing this stuff most of the time. Oh yeah, I've kind of taken a little break from that because it's not very family friendly. But I do enjoy it anyway. taking a little break from that because uh it's not very family friendly friendly but uh i do enjoy it anyway when she wants to talk about what she just watched and you're like oh uh yeah i didn't yeah catch it i'm like leave me alone i'm working uh hey never mind what my bad that's amazing yeah i've been there a few times so all right now we get on to my favorite

portion of the show seeing as though we missed outlaws it is the tip of the week yep so here's uh what i promised earlier uh i just got a little piece of heart uh networking hardware but it's uh it's not super fun so uh i am my new house i have a great internet connection but really bad cell phone service and i've looked at where the towers are and all this stuff and i just can't do anything about it like it sucks like the your phones drop data and it just doesn't work very well so we uh did a little bit of googling and found this device that uh plugs into your network or you can do it white like wi-fi too and uh you get yourself a phone number uh you can browse by area and so we did that and then we just plugged our old like landline phones uh into it we had like a wireless

set and uh it's like 90 the i think the most we got was like 120 or something, $110. And that's it. There's no monthly fee or anything that unless you want to go, you know, get fancy features that I don't care about. And so it was just kind of like a one and done. And it lets us use old school landline home phones just like, you know, you used to without paying any sort of like crazy fee to AT&T or whatever. It's basically an IP phone. You can do that kind of stuff on your cell phone too. But talking on your cell phone kind of stinks.

Cell phones are really not designed for talking on. And so they're not very comfortable even to hold to your ear at this point. They're designed for your pocket or your purse because that's where they stay most of the time. And so it's kind of this funny case where like cell phones really aren't designed for talking anymore. And it's so nice to like deal with the customer service issue for example like by talking on the landline and looking at my phone for whatever information they need like receipts or warranty info or whatever uh it's so it's been really nice and this device was just perfect i set it up in like

you know 15 minutes or something most of which was waiting for it to update and then i just plugged the phone in and it's just worked great. That's amazing. So fun fact for you. Anytime that my phone has rang at the house, an outlaw would be like, hey, the 80s called. They want the phone back. That's what I have.

I have an UMA device that sits here, does the same thing. And I was able to port my phone number from whenever I had a landline back in the day like years and years and years ago you're able to port it over to uma so if you don't have a landline then you just get a new number but if you did have one you know then i think it costs nothing per month like what jay-z said except you have to pay the the federal regulation garbage fee that's like four bucks a month so that's that's that's what it ends up costing but yeah it's a fantastic device yeah yeah let's say we got a little ask to start this got a little asterisk on there because like they don't charge anything they still have to pay somebody yeah it's really good and i think also another another benefit of at least the new uma

devices i think you can bluetooth um headsets and stuff to it so if you don't want to carry around the headset hand phone or the the phone like what you have in the past i think you can bluetooth connect something so you can have headsets like if you just have a bluetooth thing like i don't know if the apple um earbuds or something work i'm not sure but i believe they have that feature yeah they have like three tiers of the actual hardware device. And I think the third tier is the one that supports that. I got the middle one. So it does wifi. So I didn't have to plug in anything.

Okay. Yeah. Mine's mine's laying connected. I didn't have the wifi at the time. One other thing worth noting though, and I brought the, and so your use case is what reminded me of this. So my sister lives out in the middle of nowhere down in florida like nowhere and they had a similar problem where they one bar on their cell phone which is horrible

right like you you can't carry on a conversation because it breaks up all the time they actually have a booster that's installed in their house and i I can't remember exactly which one I went, I went to Best Buy and looked at it. And I want to say it's something like this. We boost and, and I'll put a link to it here in the show notes, but it actually works pretty well. Like I can talk to her, she's at home and it never cuts out. Everything works well. Now these things are not cheap. I want to say that they're, they're like anywhere between 500 and like $800 depending on, on the boost thing you get, but it absolutely does work. It takes that one bar cell phone signal at their house and turns it

into, you know, four in the house. So, you know, if, if you happen to live out in the middle of nowhere and you need something like that, which the irony is Jay-Z does not live in the house so you know if you happen to live out in the middle of nowhere and you need something like that which the irony is jay-z does not live in the middle of nowhere but there are for whatever weird reason where we live like it's a fairly populated area there are several dead spots on popular services which makes no sense i have a verizon tower one mile from me and a T-Mobile tower 0.4 miles. And I have T-Mobile and it's awful. It's awful. People at AT&T have been over here and they say it's awful too.

I have yet to find anybody running Verizon, which I thought was the most popular one, but who knows? Yeah, man, it's absolutely insane. And I don't know if it's because of the trees, because it's not all that hilly, I don't think, where you live. Yeah, a little bit. know if it's because of the trees because it's not all that hilly i don't think where you live yeah a little bit yeah it's it's it's insane i mean there's there's dead spots in my neighborhood like as you're driving through it's like really yeah so yeah all right so mine and this is i guess in the spirit of outlaw, I saw this on, on Twitter or X or whatever the other day, and it was pretty interesting. I, Oh, did I not? I will paste the link to the, to the Twitter thing,

which apparently I left out, but here's the gist of it. They said, don't use get reset dash dash hard. Now I found this interesting because we've definitely said use get reset dash dash hard in the past, right? Like if you have changes that you've made and you just want to undo them, do this, it wipes it out, takes you back to the previous state, the previous commit that you were working on, right? They said instead use get stash dash you. And the reason they say that is because and we talked about this in

our git episodes where we were going over in depth the git stuff git stash adds the objects to the git history right so if for some reason you did a git stash or git reset dash dash hard and you were like oh man I just messed up. I need that back. You're not getting it back. There's no history. It didn't write any objects. There's nothing in that, in that get tree there. It's gone. If you do a get stash dash you, it actually writes it to history and then cleans you back up just like the git reset would but if for some reason you're like oh man i forgot that i had those three files that i did there you can now get back to it because git has the history in there it built the tree it built the objects so found this really interesting hopefully that helps

somebody out and uh i will have a link to the twitter thing where the guy mentioned it but yeah i thought that was a fantastic thing yeah it's like we talked about get reset hard is like the one i think maybe was even like the one thing that destructively affects your files yes yeah it's like the only thing you can't recover from like git is really good about giving you guardrails but that one's like you're just like no i don't care i don't want guardrails yeah bye yeah so yeah i think uh i think that's it all right yeah i think we're good you're gonna do the real outro or yeah why not i mean we started with sort of the the real one so let's let's end with the sort of real one all right well subscribe to us on itunes spotify and more using your favorite podcast app and be sure to give us reviews by

visiting www.codingblocks.net slash review hey and while you're up there at codingblocks.net make sure you check out the show notes which will be a lot on this one uh examples discussions and more and send your feedback questions and rants to our Slack channel. If you have not been to codingblocks.net slash Slack, you should join because there's a lot of amazing people out there. And we still, in my opinion, have one of the greatest tech communities out there because people are just

generally nice to each other, which is refreshing in this day and age. Yeah. And also, check us out on Twitter, even though we don't post that much up there. But we do read and find some things. X. Yeah, yeah. And also check us out on Twitter, even though we don't post that much up there, but we do read and find some things like X. Yeah. X.

They're not tweets. They renamed them. What are they now?