Coding Blocks - Designing Data-Intensive Applications – Data Models: Query Languages

Episode Date: February 3, 2020

We dive into declarative vs imperative query languages as we continue to dive into Designing Data-Intensive Applications while Allen is gallivanting around London, Michael had a bullish opinion, and Joe might not know about The Witcher.

you're listening to coding blocks episode 125 subscribe to us and leave us a review on itunes spotify stitcher and more using your favorite podcast app and check us out at codingbox.net where you can find shows examples discussion and a whole lot more and you can send your feedback questions and rants to comments at codingblocks.net and follow us on twitter at codingblocks or head to www.codingblocks.net where you can find all our other social links at the top of the page. And with that, I'm Alan Underwood. I'm Joe Zach.

And I'm Michael Outlaw. I think I got it. Nailed it. This was really good. I was like, where is he? This episode is sponsored by Datadog, the monitoring platform for cloud-scale infrastructure and applications allowing you to see inside any stack, any app, at any scale, anywhere. And Educative.io. Level up your coding skills quickly and efficiently

whether you're just starting, preparing for an interview, or just looking to grow your skill set. And Clubhouse is the fast and enjoyable project management platform that breaks down silos and brings teams together to ship value, not just features. All right, now we're continuing on with the last segment of Chapter 2 of Designing Data-Intensive Apps, talking about query languages. But first, we've got a little bit of news. Yep, so as always, well, first of all, I mean, we've got to talk about the elephant in the room. So, you know, it's just Joe and I this episode. Alan is off at NDC London and left us here on our own devices. So here we are talking.

Yep. Not bitter. Not bitter at all. Not bitter. Yeah. So yeah, cause he's, he has to meet all our friends that we've met over the years on Slack.

You know, they're on the other side of the pond. So all right. Well, as always, we'd like to say thank you to those that left, left us a review. So we'll start with iTunes and we have three uh three deviant ryan in ryan's world 23 all right over song scissor we got thomas vc

through s witcher for uh dave the shirt and yarpskin dia i've never heard of a witcher though that's weird anyway what is it like somebody who makes witches the witcher yeah the witcher are you trolling me come on yeah i'm not i'm the only one on the planet who hasn't watched the show yet and of course i'm aware of the video game but yeah the witcher keeps coming up over and over again. And so I was delighted to see it as a stitcher of you. And that is just another reminder that I need to go watch the show right now. That makes so much more sense. But you are not alone.

And I will go so far as to say you are in good company because I too haven't seen it. All right. I haven't taken the time to watch the show yet, but I have heard nothing but good things about it. All right. Well, I'll tell you what. Go leave us a spoiler in the comments. No! No!

Now I can't ever read the comments. Yeah. Don't spoil The Witcher for us in the comments. And you'll have a chance at winning the book that we'll be giving away to go along with this episode. So drop that comment over there on CodingBox.net slash 125. Yep. Is that right? Is it slash CodingBox.net slash 125. Yep. Is that right? Is it slash episode 125?

Slash episode 125, yeah. Okay. What did you say the first time? Isn't that what you said the first time? No, you just said slash 125. I don't want to say it wrong again. Yeah, you're right. Don't say it wrong a second time.

All right. Well, we've got some conferences coming up. So obviously, you know, we already mentioned Alan being gone for this one. But. Yep. Software DevCon is coming up. I'm going to be talking about streaming architecture. So I'm going to be doing GraphQL through Hasura and a little bit of Kafka and showing what it looks like on the back end and front end. And if you remember, we talked about all of that in the episodes we did about the three-factor app.

So definitely going to be kind of leaning heavily on that research there. And I think it's going to be awesome. Yep. And all three of us will be at the Orlando Code Camp this March 28th. So stop by, try to kick us in the shins, grab some swag. Somebody pointed out, though, that it was you specifically, Joe, that liked to be kicked in the shins so i guess maybe it's true yeah okay well i always joke about that like you know anytime we're going to be somewhere if you look at the show notes there's always like hey kick alan in the shins he'll be in london or you know hey we're all being orlando kick us all in the shins yeah so alan does not like that joke he doesn't i did not know that alan's afraid to get kicked in the shins i did not know that okay that makes it even better

definitely he's definitely scared of it so make sure to mention that to him if you would if you want to do them over in ndc all right well um all right so let's get into it. So as Joe said, we are continuing to talk about designing data intensive applications and we are into query languages. So specifically, let's talk a little bit about declarative versus imperative. And, you know, I thought about, I don't know, have we ever really discussed the two into any kind of detail in the past? Because I don't think that we have. Am I wrong? Yeah, we once, in one of the episodes on deliberate practice, actually, we talked about the difference between imperative, explicit, and implicit declarative. It was more about like basically building memory, muscle memory. So you can kind of do things automatically by thinking at a higher level abstraction. And that's the only time we've ever really touched on that. We've never really talked about it in the context of programming languages, which is the only way I've ever really heard it before that.

Hmm. Well, okay. So, well, this is going to be our opportunity then to dig into it a little bit more. So, um, at least from the book, they say that the, you know, we've talked about the different models that came along, right? So there was the relational model document models. So they said that the relational model introduced the, a new declarative language that we know and love as SQL, right? I think that's a fair statement to say that we all know and love it, Especially the love part-ish. No, no. No, come on. You have no bad things to say about SQL.

That's not true. Just try to name one thing off the top of your head. You can't do it. Sorting. Yeah, you can't. No, dynamic sorting? No. What?

No. Yeah, just sorting and dynamic group bias. Come on. Yeah, so here's the interesting thing, though, is that, like, prior to relational models and with it the advent of the declarative query language that we know as SQL, prior models used imperative code, right?

So if you had to think about, like, okay, let's back up what's imperative, what's declarative. Right. So an imperative language, you perform certain operations in a certain order. So basically think of like most programs that you've ever written where it's like, hey, do this, then do that. Or if this then do that, you know, that would be an imperative language, like the actual code, right, to do something. Versus a declarative language, you specify the pattern of data that you want and the conditions that must be met, you know, for, so for example, any sorting or grouping. So, you know, when you think about the typical select, I'm sorry, your typical SQL syntax, you know, that is a declarative language.

So select column one from table, you know, from my table where column two equals four, right? And so, you know, you're not saying how it has to be done. You're just describing, hey, what I want is what's in the select statement. And I'm telling you a pattern where I want it, which is where column two equaled four. And maybe I might add some additional, like, grouping or sorting onto that, right? Which is Joe's favorite thing about SQL. Grr. Like, can you imagine if it did actually do those things in that order?

Like you say, select first name, last name. It's like, all right, I'll be back in a minute. I'm grabbing all the first names and last names. No, wait, wait. Here's the table I want it from. It's like, oh, crap. Okay, let me filter it down. And here's the where clause where user ID equals one.

It's like, oh, okay, you only need one of these. I fetched all of them. Oh, okay, give me a minute. Yeah, and especially now imagine if that's like on network traffic flying over the network and there's like a billion uh records in there like that would be awful right right doing it declaratively which kind of has a basis in like mathematical uh like algebraic set notation um it's kind of where the the worlds collided there and that's how we kind of ended up with some of the terms like select and intersection union, things like that are kind of common things that you see in those kind of mathy type situations.

But it gives – by having a declarative language, we can kind of turn that stuff over to the query optimizer that we talked about last time and let it figure out the best way. So not only is it doing like the froms and the wheres, but it's also able to do them in a smart order based on information statistics that it's stored about your data. Yes, that is the key point to make here that we can't, we will say it a second time because it shouldn't be understated, is that with the declarative language, because we are not taking the time to specify the how, we leave that up to the optimizer to figure out. And so, you know, there's figure out. And so, uh, you know, there's some other advantages to that, but you know, that was kind of the secret sauce that we talked about before in, you know, like what really differentiates an Oracle from a SQL, uh, or a Postgres, right. Was that, that query optimizer, but, um, you know, there's other advantages and, uh, I, I might have

this later in the notes, but, you know, cause one thing that comes to mind is that by leaving it up to the SQL optimizer, you can take advantage of future upgrades to that optimizer, right. Without having to make a lot of change to your code. So, um, yeah, I feel like I, I feel like I'm skipping ahead because I feel like I have that somewhere else, But so let's skip that. We'll move along. We'll come back to it if it's not in there. But I will say that declarative languages can be or are sometimes considered more attractive because they can be shorter and easier to work with.

But I think that's really like a sometimes thing. Like, I don't know. I don't know your opinion on that, Joe. I was just over here trying to think which declarative language I was most attracted to. And so it's definitely not SQL. But, you know, HTML looks pretty good. And, you know, there's CSS is another one I'm kind of familiar with. So those are two, you know, I guess.

I don't know. HTML5 is pretty hot, I guess. Yeah, I was thinking like some UI frameworks can be rather tedious sometimes where like, you know, they have magic things happening. And because you're just doing it declaratively, you might not realize all the things that are happening in the background and not know or be aware of some of the bad things that might be coming out as a result. So sometimes it can be more attractive and sometimes it can be really tedious, but that's definitely going to be a case by case. I would say in large part, though, I would agree with the more attractive, right? Yeah. I've seen some weird stuff too, like XAML is the one that kind of pops into mind where it's declarative and that it's kind of like xmle based language you

define you know things and it's up to rendering and engine to kind of do that but then it also has like constructs for things like looping which is kind of it's a weird paradigm to kind of fit into that and you know you basically end up doing it via some sort of binding or something where you say like hey here's a list of items and here's the data that populates it but it's just kind of a i don't know it's weird kind of mix where you've got some logic behind it that ultimately needs to tie to it so you're mixing declarative and imperative there and it just gets a little weird yeah and and that's actually a great example because uh you know i wanted to mention here too like when we talk about some of these declarative languages you know ui frameworks specifically like and one of the ideas that came to mind as a way to you know um maybe

familiarize people with the idea here is that you know you could there you mentioned xaml but there's many ui frameworks out there where you can like describe what you need what you want the ui to look like without actually having to write the button. You know, like for example, with HTML, for example, right, you can just describe kind of like the layout of the content, but you don't actually have to write the code that actually draws the specific button in the specific size with a specific label. You know, like you don't, you don't do that part. You get to take advantage of the framework that's actually going to run it. Yeah, and when I first started learning to program outside of grade school experiments,

I kind of learned HTML early on because that was becoming big and I had a job opportunity. And so that's kind of where I came from. And then when I started going to school for computer science around the same around the same time uh or maybe like a year or two later and i started messing around with consoles i was like okay i'm gonna make some cool console games or i'm gonna try and do some graphics type stuff it was like set pen red set fill one set position zero comma zero and you know i'm just trying to draw a red line right right that's kind of how you got started and that was to do that imperatively was just a nightmare you know it's like that turtle like the logo kind of left right whatever it was just a nightmare way of rendering and of course

if you look spending time looking at game frameworks or anything like advanced drawing libraries uh you don't do very much of that stuff it's more about like defining models and the camera and then kind of letting the graphics engine render that. Yeah. I can remember some of my first, um, programs like C doing similar kind of, uh, you know, game drawing, you know, where game kind of apps just for the learning purposes. But, you know, to your point, like I'm specifically saying like, draw this bitmap here and erase everything else, like erase everything and draw the bitmap here now. And like, yeah, it was awful. I wouldn't want to go back to it again, but also had like, maybe, huh, this is probably an unpopular opinion. And, and I definitely

don't think I would feel the same way about it, but I used to have this like kind of bullish opinion years ago that like I could probably say that I was probably jerkish about it maybe if I look back on it. But, you know, or like somebody would say like, you know, that they were like an HTML programmer. And I'm like, I don't know if I consider that a programming language. Like, I don't know yeah i don't know i don't i don't know but would you say that a sql programmer isn't a programmer yeah i know that's what i'm saying like that's what i'm saying like my my views are kind of like different now right but back then it was like think we'll think

about like how simple html used to be oh yeah, yeah. And then for somebody to be like, I'm an HTML programmer. And it's like, are you? Yeah. Really? After you get the frame set up, it's easy going. It's a table. I don't know. But again, I get a lot of hate coming my way.

You can send that all to – you can hit me up on slack at alan have you seen all the like css animations and stuff people are doing people are always being like here's the breath of the wild intro done in only uh css that's what i'm saying like i don't know that i i don't know that i would say that anymore though right because especially like things that you can do in html5 now i'm like, man, it's insane compared to where it used to be, right? And yeah, to your point, there have been entire crazy simulations or games or environments that are created just in HTML and CSS, which is all declarative, but yet it works amazingly. Yeah, it used to be just more advanced Word. It's like you kind of decided where to put the peas, and you needed a table you put it on a table there wasn't a lot of thought to it but now you

see the stuff that people are doing with like i don't know like the grids and stuff like css grids or just kind of um like the flows making things responsive and like now it's a it's pretty advanced and it does take a lot of uh creativity creativity visually but also uh even to get a design and to make it so it kind of composes down well and works on a phone or a website or anything like it takes a good amount of skill. Well, I think to your point, I think that's why I kind of had that bullish opinion that I used to have, which was that, do you remember, like, I think it was the old versions of like WordPerfect, where, you know, it was kind of similar, you know, the same type, it wasn't the exact same syntax, but the same similar kind of concepts, like, if you wanted something to be bold,

as you were writing it, like you actually wrote the code in, like, start the bold here in the bold there. Right? You remember that? So that's why when HTML came along, it's like, well, okay, I mean, I didn't consider when I was writing my documents for assignments, that was programming. So why would I consider this programming? So, you know, that was my old opinion, but yeah. All right. So again, all your hate mail can be sent to me on Slack at Alan. All right. So additionally, I think we kind of already hit on this, but the declarative languages hide those implementation details. So we kind of talked about this in regards to the query optimizer, right?

Like you write that declarative select statement and you don't say anything about how or how to actually perform the query. And you just get to take advantage of the optimizer, figuring that out for you. And then as future optimizers come along, right, you get that benefit too. So you don't, and it's because you don't focus on those implementation details, and you're unaware of those implementation details that you can take advantage of future updates to the optimizer. You know, we didn't really kind of mention it in this context, but declarative is like super in right now.

You can think of React as being a very declarative kind of focused framework because you basically write what the state should be and the browser is responsible and React is responsible for kind of getting you to that state. Stuff like Kubernetes too. You define your environment in the state that you want it and it's responsible for figuring out how to start, stop, or whatever based on the parameters in the state that you want it and it's responsible for figuring out how to start stop or whatever based on the parameters in order to get you to

that state um like we've i don't know if we've ever talked about like things like um salt or chef or puppet too which are kind of declarative in that you say like this is you know my environment these are the services that should be up these are the the versions they should be running and it's responsible for making that happen and so we keep seeing more and more of this kind of functional influence from language and mathematics or whatever just kind of taking over the world more and more and uh we keep spending more and more of our time uh editing yaml nowadays yeah yeah that's our world yeah that's the downside uh it's such a great way to think about it. Yeah. Yeah, so, like I said, you get to take advantage of

the updates to the underlying engine, but also because with declarative code, this also means that you can take advantage of performance enhancements with little to no change to your declarative code as those updates are made. And also, not always, but sometimes depending on like what we're talking about here, often at times, because it is declarative, it's actually more portable as well. So, you know, a select statement that I write in SQL server is going to be very similar to a SQL statement that I might write in Postgres or DB2, for example. Now, obviously, you can get you can start to get advanced with like, you know, if you wanted to do a select top 10 versus a select limit 10, you know, so there are going to be some differences in it.

But you know what I'm saying? They're like. Like HTML, for example,ml is a great example html as it runs in firefox is very similar going to produce a very similar output to the html that would run inside of chrome or safari or you know edge which is really chrome it's getting harder nowadays with better product and pen but used to be able to see how the um and better hardware and better software uh used to be able to see how pages would kind of pop in and rearrange redraws more information kind of uh was available now things are better they call it jitter when uh when websites do that but uh yeah back in back in the day when you know on dial-up or, you could really notice when the style sheet was finally downloaded and applied and stuff like that.

And so you got to see a little bit more of that, which is cool. But I mean, even the rules on that kind of changed. So you used to optimize by putting maybe your CSS at the top because you wanted the page to look good earlier. Maybe put it at the bottom so you could load the content faster or whatever. You can kind of strategize a little bit about that, but that stuff's not as important anymore. Now, here's an interesting one that before reading the book, I hadn't really considered, but because declarative languages only need to specify the result instead of how to get the result, they tend to be more likely to be able to take advantage of parallel execution.

Okay, that means you don't have to be smart about deciding when things can be paralyzed, but it can. Yeah. In the HTML example, I guess you could think the rendering and the loading of the data from your browser should be going on at the same time, right? One's not necessarily related to the other. Yeah. And think about it because they say conversely, like imperative code is more difficult to paralyze because things need to happen in a specific order. So think about any parallel code you've ever written before in your life, right? Anytime you've ever had to manage

threads on your own, it was always more complicated. Oh, yeah. But then, to your point, take your React example. That might actually be an even better example. Think of all those different React components, right? And how they could be drawn on, because it's JavaScript. I hate to say it happening concurrently, but you get the idea. Yep. Yeah. Because it's JavaScript. I hate to say it happened concurrently, but you get the idea. Yep.

Yeah. Yeah, and React's working on that suspense feature, which I don't really understand a whole lot about, but basically allows some things to kind of happen async, and you don't have to make any changes necessarily to your code in order for some of those things to take effect. So, yeah, it's cool. And the Shadow DOM is even interesting, too. Like kind of you make your changes based on your state and it's possible for those changes to result in no rendering needed but you don't have to make that decision and make that call you basically just do your thing and it's going to update a virtual model and if that virtual model is different than

the physical physical that the model the browser has then it's going to apply that change otherwise it's not going to bother and you don't have to be a part of any of that which is really cool yeah i mean it always seems like the less that you the more that you can do with less then that's just fewer bugs that you're gonna have to worry about right yeah and i could write plenty of bugs without very much code at all. So the less I have to write, the better. Now I say that, but I could write plenty of bugs in SQL. So just give me a chance. I can make it happen. Yeah. I know what you mean. It's the subtleties are tough, but speaking of tough, have you ever done much work with MapReduce? So it's been a minute, like several years now since I did anything with it. And it kind of like for many years, it kind of, how would you say, like tainted my view of Hadoop.

That sounds bad, but it definitely like made me think of Hadoop. That sounds bad, but it definitely like made me think of Hadoop in a certain light. Like every time I would think of Hadoop, I would always think of it in a map producer mindset. So you, if you, if you ever started to talk to me about Hadoop fossils, I'm like,

what, what? No, what? And so, yeah, you ever get like, you know,

you start to associate one thing with you know something with one thing and then like forever in your mind it's hard to get out of that mindset so yeah linked yeah i've done a little bit um but uh yeah i've done like the examples on websites which are awesome and great and it works out really well and it makes sense like when you do like the little getting started with uh you know mongo or map reduce or something and then you try to do like a real world example and you start running into all these little things that you didn't realize that,

especially coming from a SQL background, it's like I'm just used to kind of be a take for granted. Now all of a sudden I'm like having to very specifically think about like say my, any aggregations I'm doing or whatever and how that applies to both these two functions I'm writing, the map and the reduce. Yeah, so let's take a moment back to step back here. So obviously we're talking about map reduce, which was made popular by Google and it's a, a programming

model meant for processing large amounts of data in bulk that can be horizontally, uh, the horizontally distributed. So, um, that that's this real big advantage here is the fact that you could do all these things and distribute it horizontally. So some NoSQL databases, such as MongoDB and CouchDB, support MapReduce in a limited form as a way to perform some read-only queries across many documents, which is kind of cool. I think that they would even provide that kind of functionality. Also, maybe it's scary. I don't know. Yeah. So, so map reduce though, does it fit into, oh wait, don't look ahead.

I shouldn't, I shouldn't have put that one in there, but don't look ahead, Joe. Let's see if you know it off the top of your head. So it isn't a declarative query language, but it's also not entirely an imperative query API either. So what pattern would you say it follows? Oh, well, it's got to be template or Hollywood as we like to talk about. There it is. Yeah, this is because, you know, to implement MapReduce, you would be implementing the template pattern, or as Joe referred to it, the Hollywood principle, i.e., I'll call you, don't call me. And we've talked about the template pattern before. It's actually come up multiple

times, but we first talked about it way back in episode 16. So yeah, what was that, like 109 episodes ago? Yeah, it was a while. That's crazy. It's been a minute, is what Alan would say. It's beautiful. Oh, what was it? Somebody had a joke that they were betting Alan couldn't go an episode without saying it. And I'll say it later. Oh, I don't remember.

I do. I do. So I'll make sure to throw that joke in later. We get to make all these jokes because Alan's not here. So I feel fine. It's fine. It's probably okay. It's acceptable.

So at the time I first started playing with MapReduce, it was mainly tutorial-based and a little bit for some work stuff. But JavaScript didn't have a map or reduce function. C Sharp Link was still kind of new. And so these kind of functional concepts didn't really jive very well. And so it was very different paradigm for me. But now I think pretty much after working with link and C sharp and JavaScript map and reduce functions that I use pretty often, Kotlin's gotten map and reduce, you know, particularly map I use all the time. And so that concept of just even saying like, first you map or that's not even true.

Just you have a mapper and a reducer and they work together to provide query results. Like I have a much better understanding of that just based on the mapping and reducing I do in these other languages because it's kind of similar concepts. Yeah. So with MapReduce, you're going to implement two methods, a map method and a reduce method. So the book gives an example where they use the map method to reshape data into a particular pattern. And then they use the reduce method to perform an aggregate summation on the values for each key that gets passed to it. Right. Which is kind of along the lines of what you were describing.

Yep. Um, so, you know, but you think about that and you're like, okay, so I'm going to get some kind of pattern. I'm going to perform an aggregate summation on that.

And I was like, wait a minute. That doesn't, that kind of sound like aggregate functions that you can already do a SQL. Huh? Uh, yeah. Yeah.

Yeah. And, and rightly so, because the book actually later warns that they were describing how specifically MongoDB's actually find itself actually accidentally reinventing SQL in disguise by doing exactly things like this. Right. So I thought that was kind of like interesting, you know, not necessarily a jab at MongoDB per se, but just, you know, it's like, it's always like something great comes along. You ever noticed this? You ever noticed that technologies are like this, Joe?

Um, you have, you have some like all purpose, just engine that just cranks out stuff like it. But, but that engine is like, you know, uh, just a burly diesel engine. It's loud. It creates a lot of smoke and it's just nasty to listen to, but it gets the job done, right? Like it just gets the job done. But then somebody else comes along with this lighter, smaller, you know, four cylinder that's quieter and it's got turbo on it and it's like super fast. It'll run circles around the other one. It doesn't have as much torque about it, you know, it's just it's quick off the line whatever you know um but you know it doesn't

have nearly the amount of horsepower right and so it's like it's it seems like all technologies are like that like something's leaner will come along and it does like one thing just super well but then it starts to fall apart like some of those other things it's like yeah okay fine it doesn't do that but that's why it's able to do this other thing so fast is because it only focused on that one problem and then and then eventually that thing will like evolve and become popular over time and they'll start adding in other functionality to it where it's like hey wait a minute now they're like they're really closer to parity with each other and so it kind of feels like oh well maybe over time that's how mongo db came to where it is where it's like oh hey we're going to do the same kind of aggregate

functions with sql except we're going to do it with a map reduce functionality instead yep and it keeps growing bigger and bigger um and that's how you like flasks the flasks of the world end up evolving into the jangos of the world yeah yeah everything starts off with simple and then the more popular it is it's tempting to just keep growing and growing and growing and that especially happens with databases which definitely at some point i i don't know i i can't even think of a database right now that doesn't support some sort of sql-like syntax oh man that would actually be well in a relational database oh sorry i meant like any data store it's like elastic search supports the sequel you know it's not mongo has a sequel like language yeah

so they may call it eql or mql or whatever like jira has a sequel like language you know right yeah so uh these mapper map and reduce functions, these are pure functions. So that means that they can only use the data that's passed into them. They can't, as it relates to like a MongoDB, they can't perform additional queries, and they must not have side effects. Yeah, so we didn't really give an example, but I was thinking like an example might be something like if you've got a big data set, you might say like, count me all the users who logged in yesterday. And so you might have a map function that is responsible for essentially it's like the visitor pattern where it gets a node or not a node, a person that says, did you log in yesterday? And it returns either one or zero. And that will be the map because it takes in a person and it spits returns either one or zero and that will be the map because it takes

in a person and it spits out a one or zero and you're gonna have a reduce function that just sums up all those ones or zeros and then comes up with a count so you've got these two functions one that returns one or zero and the other that sums and that's your map reduce and no side effects you can't affect any other records you can't backwards. You can't do any funky stuff because... Yeah. Well... Pure functions. Yeah, because...

Okay, sorry. Because they're pure functions. So that kind of sounds like functional programming, though, doesn't it? Yeah, exactly. That's what... I was trying to lead there, but then I said it wrong.

Yeah. And that's because it is, though. I mean, pure functions are a concept used in functional programming. So, yeah, that's why it sounds like functional programming. Yeah, and the world every year seems to be going more and more functional-based, and SQL databases in general have kind of been there for a long time anyway. So, then maybe all SQL developers have been years ahead of the rest of us,

and we're just catching up to their expertise. Don't tell them we said that. There are a lot of problems you can solve just in SQL. Don't tell them we said that, though. Right. Never. Of course. We'll never admit to that.

You never heard it. We will never record us saying that. Hold on. Wait. Too late. Don't forget to cut this part out. cut this part yes yes all right so from a usability perspective though it does require writing two functions that are somewhat tied together right which can which can be more effort than just writing the single query. That's strange.

Yeah, I mean, because you can see how like, think about just writing a map reduce function and you would only like that reduce function is kind of tied to what it expects that it's going to get from the map function.

So it's going to be looking for a particular shape of data coming in. And that map function is probably the only thing that's going to like be producing that really. Yep. So I think when I work in JavaScript too, I don't typically just have one map and one reduce. I have like map,

map, filter, map, reduce. Map, map, filter. Yeah. Right. And then if you've got to just break that down to two uh functions

no cheating no you know none of the kind of normal uh weird sidesteppy stuff i do in javascript all the time uh illegally against the law uh you can't do that sort of stuff it's just not available it's like it only does this one thing. It only maps, it only reduces. Yeah, so following along with the usability perspectives, though, a purely declarative SQL query, again, would better be able to take advantage of the optimizer, which your quasi-imperative map-reReduce code isn't, right? And so for this reason, MongoDB added a declarative query language that they call the aggregation pipeline,

which is kind of wrapping around the MapReduce functionality. So its expressiveness is similar to a subset of SQL, but it's in a JSON syntax instead of the mathy language of SQL. That's a great point, too, to just remind. We've mentioned this a couple times before, too, but everything's blended. There's nothing quite so hot and dry. We're talking about general trends, but I'm sure that Mongo has some sort of query optimizer. I'm sure they can do some fancy stuff with its map and reduce that. Kind of bend the normal rules a little bit to let you do stuff. And that's just the world we live in.

Like things are always kind of creep out the edges and overlap. And so, you know, nothing's quite so cut and dry. But the general trends are still there. They're still strong and they're still good. But you can imagine, like, if we only had a pure map reduce world and we wanted to look at all the users who logged in yesterday, we'd have to look at every single item and every single person and say, look at their last login time, which would be absurd coming from a SQL background where you've got indexes. And so maybe if you've got that thing indexed by login date, it can only look at a very small fraction of the data, which is going to be much more fast, much more faster to do. And that's not something that you can really do in a pure map function, reduce function only. Yeah. Yeah. And I want to call out too that, you know, we're not necessarily picking on MongoDB. I know, you know, we've mentioned it a few times in this, but that's because the examples

in this portion of the book, the book, you know, kind of just transitioned to like, Hey, let's focus on just MongoDB, for example, just to try to like illustrate the point, not necessarily like to, to, you know, pick on it or, you know, you know, say it has any faults or anything like that. Yeah. It's, it's the most popular by far, I think. So it kind of gets a lot of the things that people are frustrated with about NoSQL databases that tends to get kind of... Well, let's come back to that.

Put a pin in that. Ping! Today's episode is sponsored by Datadog, a modern full-stack monitoring platform for cloud-scale applications and log metrics all in one place. From the recent report on serverless adoption and trends, Datadog found that half of their customers are using EC2s and have now adopted AWS Lambda. And they've examined real-world serverless usage by thousands of companies running millions of distinct serverless functions and found half of Lambda invocations run for less than 800 milliseconds. You can easily monitor all of your serverless functions in one place and generate serverless metrics straight from Datadog.

Check it out yourself by signing up for a free 14-day trial and get a free t-shirt by going to codingblocks.net slash datadog. That's codingblocks.net slash datadog to sign up today. All right, it's that time again. Take a little break here and ask you to leave us a review because it's like the most important thing that you could do for us.

It helps us a lot. We love it. It means a lot to us and it really helps us out. So if you just take a minute and go to codingbox.net slash review,

we try to make it easy for you, give you a couple links and you don't have to install anything if you don't want to. We try to just lay it all out there to make it easy for you and we love it.

So please consider it. Thank you. All right. So, since Alan isn't here, I didn't think it would be too fun to play Survey Says, just the two of us. But I did think of a survey, though, that we could go ahead and add to this one.

Oh, boy. Yeah. So I see you're already looking ahead, huh? Yeah, this is going to get controversial. I don't know if we're ready to go here. Yeah, no, we definitely are. And Kafka will be next on the list. But this one will be, how do you pronounce data?

Do you pronounce it data, where that first A is a long A? Or do you pronounce it data, where that first A is a short A? Which one do you pronounce it data, where that first A is a short A? Which one do you say? Data, after the Android. Okay. Data. So, long A. Yeah.

All right. All right. Well, I'd be curious to see, like, which one comes back as the more popular option. Yeah, you know, I think uh to use them differently in context like there's an album that i like data learn the language i think i've always pronounced it mentally as data there but most you know i default to data well i know that i'm pretty sure there was like an episode a couple episodes back where one of us and i don't remember if it was you or Alan, used it interchangeably, like in the same breath, in one sentence. At the start of the sentence, it was pronounced one way,

and at the end of the sentence, he pronounced it a different way. But yeah, in the same sentence. So that was pretty great. Yeah, that's goofy. And yeah, I think that I do say it consistently for those contexts, which doesn't make any sense. So I don't know. Enlighten us in the comments and make sure to vote for data.

All right. So I thought we'd have a little bit of fun here though with, you know, you mentioned about you, you were suspecting that MongoDB might be the most popular. So I was going to ask, we never did talk about the last Stack Overflow developer survey. And so I thought, well, it might be interesting to bring up some of it as it relates specifically to the topic of these big data applications that we've been talking about. So I guess if I were to ask you what you think the most popular DocumentDB among Stack Overflow respondents, I guess you're going to tell me MongoDB,

or were you going to pick something else? Well, it depends. Okay. If you need to phrase it like what's the most popular DocumentDB, then I instantly want to answer Document DB because it's in the question. But I know that we're actually talking about document space DBs, so I'm going to say Mongo. Well, I had it written down as document space.

Yeah, I know. I just got to have to be a jerk sometimes. I can't resist. Okay, well, what if I were to rephrase that as like a NoSQL? What's your favorite? What do you think the most popular NoSQL database would be among stack overflow respondents is that is that better you know it's it's then when you say it like that oh the change of my answer again no i i think it's still probably going to be mongo but i think elastic is going to be in there okay six search so so but you're

going with but mongo is your official answer. Yeah. Okay. Well, you would be right in picking that. Which one? But with picking Mongo. Okay. Yes. So it is in the top five, but barely.

Oh. Yeah. It is number five on the. Wow. Most popular databases of all respondents on stack overflow so oh it's like redis in there and stuff it is but it's not in the top five well that was number six you want to try again for no sql, just database technology, period. Oh. Mongo was number five, made the list as the number five spot.

So it was the most popular of the NoSQL databases because it was, you know. Okay. Oh, I see what you're saying. Okay. Yeah. So yeah, one through four are going to be like Oracle, Postgres,acle postgres sql server and and elastic okay no uh you had a couple you had you you named some big names there so you got some of them right so i will tell you this so my sequel is number one number two is postgres uh three was sql server and number four was sq. Wow. No Oracle in the top five. No, not in the top five.

Oracle would have been, uh, let's see, eight, number eight on the list and elastic was number nine. All right. Yeah. Which is crazy to me to think of it as a, to think of elastic as a database technology, but I guess technically, it's just another index, right? like that's all database technologies are but it's about creating indexes to be able to search for the data fast right yeah you know so it's just a purpose-built index um yeah i don't think i until i saw it and then if you had asked me before outside of the context of this conversation i don't know that i would have

lumped elastic search into a database though um but yeah i can see that and you know something else that never dawned on me too um because technically i kind of want to give you credit for picking oracle when you said oracle postgres sql server because i never realized that oracle owns my sql oh yeah i remember i remember when that happened because i was working in my sql because I never realized that Oracle owns MySQL. Oh, yeah. I remember when that happened. I was working on MySQL at the time. I was like, uh-oh. Yeah.

Yeah, because it was bought by Sun, who was then later bought by Oracle, and you just never think about that unless that's the world you live in. Oracle also bought Java, which is weird to me. Yes. They bought Sun, who had Java.ava yeah it's just very strange and ibm now owns red hat which is also strange i was like one of the biggest tech acquisitions i hear that it was like 35 billion 35 instagrams i don't remember how much it was but that that is a weird but the to me the owning the language is the weirder one to me yeah owning java seems weird oh yeah dell uh buying vmware was one of the other biggest tech acquisitions of all

time oh but but you know to to just back up on that uh you know owning the language being weird though i say that and it doesn't bother me at all to say that C Sharp is a Microsoft thing. What about, say, Postgres is a Microsoft thing? That would be weird. That is weird, yeah. That would be really weird. Is it? No. Yeah. What? Microsoft owns Postgres? No, they don't.

Postgres is open source. It's a company that maintains, that are the maintainers of the official Postgres repository. Oh repository oh really i didn't realize that yeah they bought them and uh situs around the same time huh yeah coincidentally i doubt it that is curious i did not know that learn something new every day um yeah and and the mongo being number five on the Stack Overflow survey was also consistent with where it ranked on the DB engines.

We've talked about before db-engines.com. MongoDB was number five there. And in fact, the list is very similar, except that instead of Oracle being in the number six position, it's in the number one position on DB engines. Yeah, I really like DB engines.com. It seems I don't know where exactly they get their data from. It seems like it's really good and on point.

They have documentation on how they calculate their methods. So in case if this turns out to be like a Tyobi index thing again, where you guys want to start bashing on me. But let's see. it's based on the user variety we measure the popularity of system by using some of the following parameters the number of mentions on system of the system on websites general interest in the system frequency of technical discussions about it number of job offers a number of profiles and professional networks and relevance in social networks. And mention AOL groups.

Oh, sorry. I thought we were talking about Tyobi again. Oh, my God. Probably. Whatever. Whatever. All right. So what do you think the most dreaded database?

NoSQL or? Let's stay on the No sequel focus for a moment what's the what's the most dreaded most dreaded no sequel database is i'm tempted to kind of say one of the relational databases that shoehorn the feature in but i'm gonna go ahead and say mongo uh no no okay uh Congo? Uh, no, no. Okay. Uh, wait, I'm on the wrong page only because it's also the most popular. No. Uh, couch base really is the most dreaded according to that stack overflow developer survey.

Okay. I don't really know how it's different. I don't know enough about either to tell you that either, but, but what do you think the most dreaded relational is oracle yes i like how you didn't think about it you're just like oracle and you were so matter of fact about it it's just yes of course it's oracle what are you talking about it's oracle yeah and i you know i have a bias against oracle just because i've worked so much in sql server and it's similar in so many ways to SQL Server.

And the little differences are so aggravating. Yeah, but you can say that about a lot of them though, right? Yeah, but especially Oracle I say it about. I used to complain about Java before I loved it. And I used to complain about JavaScript before I loved it too. Maybe I'll grow to love Oracle one day. All right. Well, one more game before we get back on topic. We haven't played Google Feud in a while. Do you remember this? Yes. All right. So, I'm going to type into Google, Big Data Is, and then you try to get me uh just real quick name three things that you think

would fill in the blanks for big data is that would show up in the list uh oh gosh um um big data is cloud okay big data is not that big. Okay. And big data is, oh gosh, dumb. I don't know. Okay. I'm just trying to think of what would be the craziest thing I would say on Google. Okay.

So, yeah, there were some crazy ones here i'll i'll give you a couple of my favorites so the the top one which wasn't necessarily my favorite which is the top one is big data is better data which i'm like what how okay i guess maybe you could you're saying i guess maybe the the result the idea there is that you know if you have more data then you can make better decisions. Maybe. I don't know. There were some weird ones, but some of my other favorite ones, though, is big data is watching you. Yeah, I saw that.

That's awesome. And then I was so surprised you didn't say one of these. Big data is dead, is bad, or is good. Gosh. I should have guessed that. Surely you're going to get one of those, right? I'm sorry. I'm off my game tonight.

Yeah. All right. Well, let's get back into this. But first, this episode is sponsored by Educative.io. Every developer knows that being a developer means constantly learning. There's new frameworks, languages, patterns, practices. There's so many resources out there that where should you go? Meet Educative.io.

Educative.io is a browser-based learning environment allowing you to jump right in and learn as quickly as possible without needing to set up and configure a local environment. And the courses are full of interactive exercises and playgrounds that are not only super visual, but most importantly, they're engaging. And the text-based courses allow you to easily skim the course back and forth like a book. So there's no need to scrub through hours of video to get to the parts that you really want to focus in on. And amazingly, all of their courses offer a free trial and a 30-day return policy, so there's no risk to you.

You can try anything you want and always feel comfortable that, you know what, you've got that 30-day return policy if you needed it. But hey, even better, they just recently introduced subscriptions. So now you have even more comfort because you could just get the subscription and basically you get to see every course they have in their entire catalog. And our listeners can get 10% off of any course or subscription by going to educative.io slash codingbox. As you know, we've been talking about distributed systems a lot lately, and so I keep going back to reference the course that I love so much called Grokking the System Design Interview.

And I wanted to particularly mention the section they have on Twitter, which is not one of the ones that you can view freely, but you can still go to that course and check out one of the other sections. But I wanted to mention Twitter because they had two interesting use cases that kind of tied in with some of the polyglot type stuff with persistence that we're talking about tonight, where Twitter has some kind of very different use cases where most people have few followers and they're able to make use of the technique that they describe in the course called fan out. But a small number of Twitter users have billions of followers, like say a Kanye West or a Taylor Swift or whatever. And so it's such a dramatically different use case that is different from 99.9% of the users, but can just crush the way that the algorithm was originally set up. And so the course is really great about kind of describing these two use cases and how Twitter deals with these two use cases by kind of taking two different approaches. And I thought that was just a really cool section to kind of tie in with some things

we're talking about tonight. Yeah, absolutely. Fits in with the whole book series that we've been doing lately about designing data intensive applications. So listen, go ahead, get yourself a subscription today. You can start learning today by going to educative.io slash coding blocks. That's E-D-U-C-A-T-I-V-E dot I-O slash coding blocks and get 10% off any course or off of a subscription today. All right.

Now we're going to talk about graph-like data models. And we're not going to go too deep in this just because I haven't done a lot of work with graph databases. I don't know about you, Outlaw. But it's just not really something I've spent a lot of time in. And so I don't want to kind of mislead anybody on that. So we're going to skip a couple of things about some of the specific languages for dealing with graph databases, just because that's kind of hard to describe anyway, and not spending any

time with it, just kind of rough. But I want to mention, like we talked about a lot last episode, we talked about when to decide to use a document database versus a relational database. And we didn't talk about graph-like data models at the time, and talking about them now, it comes up because as your relationships grow and things tend towards complexity and you get more and more many, many relationships, the relational model starts to break down a little bit because you have to start relating more and more things

and it gets complicated to query. So there's this set of graph databases. That operate just like the graph algorithms. That we talked about in episode 163. No. No. 85. That's what I was looking for.

Yeah. Remember that we talked about like graph algorithms, acyclic graphs, ASTAR algorithms, things like that, Trevink Salesman problems. So there's a set of databases that are built around those concepts where you kind of start with a node and you traverse the graph doing your filtering and finding and inserting and things like that and uh so the book talks about bringing these type of databases in once you start having too many many many relationships and it becomes painful but they didn't really define like a set number it's not like 10 mini minis is too many or five is too many too many so i don't really know how to decide where that line is to know like this is too painful it's time to swap too many okay so if you have too many mini to many relationships and that's when you need to switch to a graph database so already we have too many

minis in that statement yeah that's the too many many minis mini to minis that's like kind of the distinction in the book is like basically if you have too many many to minis you should probably switch to a graph like data model there's no real guidance on when that is i know that i've worked with graph databases a little bit just kind of of exploring, messing around with Docker and tutorials, whatever. I've done a couple of proof of concepts just to see how something that was hard to do in SQL would work out in graph databases. And my experience with them have been really good so far because the things that I've had a really tough time expressing in a SQL-like language with a relational database have been easy in the graph database. And that's what I wanted. But in the experience that I did,

and remember, I'm a total noob here, so maybe I was missing some basic common things like indexing or whatever that would have helped, but the performance for the size of data I was trying to achieve just wasn't really adding up. So I didn't end up going forward with any of the POCs I kind of messed with, but it wasn't because of developer experience, because that was the thing that was most amazing to me.

I wanted to find these relationships, and I was able to kind of step it out and write a query that was easy to read. And that was very much the opposite of its equivalent in SQL. So now when I kind of think about trying to decide if GraphD a graph if graph database is worthy of talking about bringing into an architecture the distinguishing factor that kind of triggers me to even consider it is how miserable it is to query my data okay not that makes sense so how do you know when you've got too many mini maze relational database is when it gets to be miserable to query them. To write the query or to run the query? To write the query.

To write the query. As the human, when your queries get too painful to write or too painful to generate. I would expect it to be like the execution of it would be the problem. Hmm. Yeah, so that's kind of, that's my own

distinction and personal definition is like when it gets too painful for me to want to write queries. Now we should call out though like just to avoid any confusion here because we talked about before how everything had a

query language and they might just put like their letter in front of QL so you might have like an EQL or Jira's JQL or, you know, Elastic's KSQL. But this GraphQL does not equal Graph Database query language. Right. Yeah, there's a distinction there the graphql is just a kind of specification but it turns out it does map well to graphy type concepts so there's one graph database dgraph in particular uh that's that's kind of popular now that uses graphql as its query language and that's because you can kind of do these like infinite net levels of nesting so if we're talking about like a social media example like i can do something like give me the friends of my friends who support this political party. And so in GraphQL, that's easy to express declaratively because it's basically like, you know, person, you know,

I don't know what the language ended up being, but it's something that's kind of easy to express and sounds, looks just like I said it. But that's something that can be really tough to do in a relational database where you say like, find me the but that's something that can be really tough to do in a relational database where you say like, find me the friends of my friends that like this thing because it's joining a table onto itself and it just gets kind of weird. So I came up with a kind of another social media example that, uh,

it's kind of contrived, but it's the kind of thing that the way I'm going to say this is going to be very similar to the way I would write, uh, like the, the graph query. I've done a little bit with Neo4j. It's the main one I've used.

And so I'm blanking on the language that they use now, the main one. But the way you would write it almost looks like the sentence. Fetch the top 10 people that are friends with my friends, but not friends with me and have liked pages that I like. And let's sort on our common interests. Okay. So think about that from a relational perspective. It's miserable. It's like table people joining on, you know, friends, which is basically a people to people, many to many. And then we got to join uh back to that to make sure that it's not related to me and then we've got to do an exist there to make sure that they're not also friends with me we have to do another exist to make sure that they're friends

with the other friends and so it just ends up being this big like kind of self-referencing nightmare to write in sequel and you're going to end up going easily over like 40 lines of like very complicated, like tricky SQL rather than like something that almost looks like an English sentence. Right. Even, even as you say it though, it's kind of like, okay, I could see the value in something like, like you, you know, you're coming at this from the point of view of like a social platform kind of example. You're like, yeah, I can see that totally being a value, right? If you wanted to add a feature to your application that's like, if you were writing a social app, right? You're a young Mark Zuckerberg sitting in your dorm room writing this thing. You're like, hey, here's other people that you

might like to be friends with because they're already friends with your friends and they like the same things that you like. So so yeah you should get to know these people yeah but that seems like a that seems like a valid kind of use case right yeah absolutely and it's one of the three use cases that they gave so um like social graphs basically uh something like a linkedin or a facebook where you got friends of friends and you think about facebook like you've got entities like people or media or articles and people can like media. They can be friends with other people. You can be married to someone else on Facebook. You can have these different complicated relationships. You can

work for a company. You can define all these kinds of different connections. And what's really cool about that is, you know, one that the entities and the relationships don't have to be homogenous. So I can kind of like come up with a new entity tomorrow for Facebook and say car. So now user Joe drives this kind of car. The relationship is drives the entities involved. They are a person in a car. And so that's something I can add to that model without adjusting any existing

schemas without adding new tables. You know, I basically can kind of just throw these things in there to my existing graph database, and it doesn't affect anything else I've got going on. So it's a very flexible way to represent these kind of real-world situations. And then it's immediately usable in those queries. We can say something like, show me friends of my friends that haven't bought a new car in 10 years, assuming you've got the data for that kind of thing.

And maybe you can kind of show them our highlight posts where I post about my new car or something like that to maybe encourage them to go shopping for a new car. It's all scary stuff. So let's take a step back here, though, and just kind of describe some concepts around the graph database. So document DBs have documents. Relational databases have tables and rows and columns. And we should say that inside of the document DB documents, there's properties within that document, right? And property values and values in those documents.

But graph databases have, they have vertices, which are the nodes in the graph so like joe might be one node i might be another node and then there's edges which would define the relationship between those nodes so there might be an a line you know from my node connecting to joe's node that would say like hey these guys are friends right yep you can have additional metadata on there too like maybe they became friends on this day or you know other stuff that's a contrived example but um it's hard for me to say vertices i never i always want to say nodes yeah i've got a bad habit of kind of saying that and they're interchangeable in this uh same with edges sometimes you'll have edges referred to as relationships or vertexes and i really hate talking about vertices and vertexes. Yes.

I would agree with that. Yeah, I kind of like the nodes and edges better, but now going back to your car example, though, now you can see why, like, hey, adding that other car with just another line, drawing Joe, too, is like, hey, Joe drives, you know,

I don't know, Ferrari. Okay, yeah. I was just a wild guess there Ferrari. Because... Remember him. Oh, okay. Yeah. I was just a wild guess there. I guessed it right. Wow. That was a long shot. I'm definitely into trucks so much, dude.

I love my Ferrari truck. I didn't know Ferrari made a truck, but okay. Oh, my bad. Oh, yeah. I thought you were talking about Ford. Oh, yeah. Yeah. So, you can see how, like, you know, adding those additional relationships,

you kind of have the flexibility, which is what you were getting at with the, the Facebook example where like, if you wanted to add in, you know, the capability for like, you know, Hey, these people drive these cars or, you know, these are the articles that these people like, right. Like, you know, making that additional node and then drawing relationships to it is a little bit more flexible. Yep. And so a lot of times you'll hear like Facebook associated with graphs, same with LinkedIn.

Another kind of common functionality that are two more that you may not necessarily think of as being like graph oriented is actually one is kind of search. So like a web graph a graph of the internet where you might have your nodes or vertices are pages and the edges are links and so the initial early versions of google was built all on this kind of concept of page rank which basically had these nodes as pages and it counted the links between them to figure out how to rank

stuff so that things that were linked to more often showed up more prominently and they kind of thought of those as being more authoritative and that's basically a graph type algorithm and something that would be really hard to do in sql because you have to do almost like a recursive kind of like you know if you had two tables basically a web page and then you had a breakout table with like a web page id and a link and another web page id to try and query that to count to see what's the most popular uh is is just complicated because things can kind of uh link to each other it's a second graphs they can go in circles and so it's just kind of like a this weird recursive kind of relationship that is hard to express in a

relational database definitely in a document but is just kind of express in a relational database, definitely in a document, but is just kind of natural in a graph. Yeah, and another example that you have here is for maps, right? Yep. So you could think about any address, like how many different ways could you possibly get there, right? So if that address is the the the node then all the roads would be the edges to get to it yeah and the data you could have on those edges might

be the speed limit or whether it has tolls or not or whether it's traffic conditions yeah yeah yeah all sorts of stuff so all that stuff can be factored into this thing so maps are a great case where you might want to have um you know a graph database backing that and something that doesn't really fit very well in relationship. But still, I have a hard time really kind of putting my finger on the one thing to say, like, this is when you should switch to a graph. But, you know, we're seeing some like some cyclic stuff and some, you know, just recursive relationships. But it's still kind of hard to really define that or to know it when you see it in the wild. Well, I mean, I think the more we talk about it, though, the more I tend to agree with your, I tend to like your mini, too many mini to minis, right? Like when that, when that, when you start to get a lot of those or it starts to feel a

little heavy handed or like your usage, uh, you know, know that that seems to be the types of querying that you want to do more often than not then it sounds like you know you don't have to necessarily use the graph database for everything but at least for those those instances it sounds like you hit the nail on the head the first time yeah that's like i have to think about it too it's like when you spend when you're spending like three hours in your sql editor trying to get this query down it's weird and doesn't fit right and you want to change the the way you've represented your data like five different times like you may want like that's kind of a sign that you may be dealing with a different type of data that would work better in a different database it may still not be worth bringing a new technology for if it's a small use

case or something but uh at least you can kind of recognize like oh hey this is a graph problem that i'm trying to solve with a you know essentially a tree of sql of relations i feel like a bill ingvall uh meme should fit here like here's your sign remember that here's your sign comedian oh yeah yeah yeah yeah like how do you know when you need to use a graph database? Here's your sign. Right. Right? Yeah. Oh, man.

Yeah, we talked about him recently. And it was not the other guy that I thought it was. Jeff Foxworthy. Oh. Yep. That is not the sign guy. Jeff Foxworthy is not the sign guy. You are correct in that.

Yes. All right. So, yeah, go ahead. i was just cleaning up the notes a little bit i was going to say that um another example um kind of like a similar to the maps there and similar to the way that graph databases are kind of flexible we talked about the the ability to add new things in is uh some things are really hard to represent in a strictly normalized way and the example that we saw in the book is basically um america has states and states have counties right and other countries like france for example has uh departments and something else that i forgot uh departments and regions and these things have slightly different rules um and even in america too like you know you have a city that may span multiple states or multiple counties and things get kind of gross to try and and do that and uh so in a graph database i can associate france and

then say france you know this country has these departments and these departments have these regions. And then I can go over to the United States node and say, well, it doesn't have departments, but it does have states and states have counties. And so I can kind of keep these two different similar but different models in the same database. And I don't have to try and normalize them. Because when you think about how you would represent that same kind of situation in a SQL database, what you'd have is a country table with country ID and your country name. Now to link the departments, you'd need a breakout table called maybe departments. It's got like a country ID in it and also the name of the department. And then for the US, you'd have a table for states with a country ID linking back to the country. So that means whenever you're querying to display that data on the page, you have to join that country table to both the departments and to the states.

Even though you know it's only going to ever have one. And then you have to do that hop again for the departments in the counties so am i making sense there yeah i was trying to think of it like um how we might be able to like make it more relatable to those of us in america that uh if you were to just stay with like consider dc right because it's not a, but it's also not a city. Right. So it's its own kind of little entity that's unique than to every other part of the United States, right? It doesn't really fit in your typical

relational model of how you would define even just the United States. Although I guarantee you that most people would just put DC in the city of any relational model. Yeah. It's tough. I've seen, um, taxes done in systems where it's like,

um, okay, here's the state tax that we charge on any sort of item, state sales tax. And then you start selling internationally. Oh, okay. Well now I guess we'll have an international tax rate that can apply

in the country but we'll have the states override it if we have a state and if they don't have a state then we uh we just we'll make this left join now and it just starts to get more complicated that's one you know like simple hop there but like we start talking about tax districts and maybe like departments do have tax rates but regions don't but rates, but regions don't, but states do, but counties don't. Or maybe one supersedes or maybe others are cumulative. Or some cities have their own tax or they might have tax on specific items. Yep. And to do that relationally, it just gets kind of gross because you start doing these things where you break out things into separate tables but they uh and they'll link back to another table but there's

certain rules that are not expressed in the data that like say for example that you don't you can't have a country that has both states and regions because that's mutually exclusive but the the database doesn't guarantee that constraint so the flexibility then of the graph databases also make for great evolvability then because you might not know you know what you need today for today's purpose and then tomorrow comes along and it's like oh yeah you also need her to add in france it's like oh no worries i can i can easily make their model for how their uh you know hierarchy of of how the regions are broken down, you know, into the different nomenclature, I can break that apart into our database just easily, right?

Yep. Yeah. And also that flexibility can be a double-edged sword, though, because, of course, now if you want to implement a common feature, you might have to go kind of explore that twice with those two different kinds of, or you have to have some domain knowledge to know like, okay, well, it could be this or it could be that, and these concepts are similar. So you have to know more and make those decisions yourself rather than just kind of getting it for free by adding a column to a table. All right. So just when you were selling

me on graph databases, you went and said that. So now I guess I'll go back to document database, all the things. Yeah. Well, I do think it seems like we're living ever more and more. If you're dealing with a lot of data, if you're building a data intensive application, you're probably going to have more than one sort of data store. Maybe a search engine, document database. I didn't know where you're going with that. But yeah, I would totally agree with that though like it used to you know 20 years ago you'd be fine with just a relational database of some kind right whatever it was whether it be at oracle a sql server or a db2 whatever you you were fine right but yeah you're right now it's like you know at a minimum you're

going to want something like a a search engine to add on top of it. So like an Elastic, for example. You know, you might have some other document DB that you use just for like, you know, these use cases all go against that database. But, you know, occasionally you might go against this, you know, search index or maybe query back to a relational database. Or maybe the relational database is just kind of like the data warehouse that's sitting underneath all of these other ones, feeding all the other ones.

Yeah, I don't know. And it's common to export to a data warehouse just for reporting. So I kind of dump that stuff into basically some sort of OLAP solution that's specific to that, and then sales can run the reports or whatever on it. Yeah. And what about caching servers, things likeis or um whatever whatever the other one is you know those things are common and those are kind of like they're data stores for sure and they're kind of like a database in

some ways key value yeah well i mean how far'd be a very simple one right like one table with one one table that has two columns with one primary key yeah that's true uh yeah i've heard someone recently referred to environment variables kind of like a database is it kind of viewed as like a key value store that's resident on your you know your environment it seems weird to me i've heard a little bit about etcd i don't know what that is but i think it's kind of like uh let me see here yeah distributed reliable key value store for a distributed system yeah but when i think about it see i still think about like file system so maybe i'm just mixing that up there.

Yeah, when you say Etsy, you're saying like E-T-C, Etsy, right? Yeah, Etsy D. It looks like it has nothing to do with the file structure, but I can't help but see it when I see the name. You can't unsee it. Right, yeah, I get it. So, you have this question here, and I kind of want to steal your thunder but i don't have an answer for it but but how how well do graph databases perform in scale yeah and now that's something that particularly when i was doing my like kind of poc phase whatever for some stuff i really wanted to know and of course the answer was it depends and you could find some use cases and people are talking a lot about how well we've got terabytes of data

and it works great for us but then you look and you know dive deeper into what they're talking about and maybe they don't have very complex relationships and at the same time you could see someone that says oh this scales you know terribly but you look at what they're doing you know you could see why it would scale poorly and so it's really hard to say.

And then, of course, there's a lot of information put out by the various database vendors. And there are some for-profit databases that are out there, graph database like TigerDB comes to mind, Aragato, and that's on it, the A1, crap, I forget the name, that, of course, you know, have all sorts of articles talking about how fast they are and have some, you know, interesting performance numbers where articles talking about how fast they are and have some, you know, interesting performance numbers where they show how much faster they can query

than a relational database in this very specific use case. And so it's really hard to, to figure that out. And so I don't have a general rule of thumb where I can say like, Oh yeah, they're faster or slower than a relational or document database. But I got to believe that they do pretty well when you consider that somebody like a Facebook is using it.

Yeah. Right. Yeah. I would think so. Yeah. I mean, it's probably pretty good. Yeah.

It's probably good. Yeah. If it's good enough for Facebook, it's probably good enough for you too. Well, well, I mean, if we're being fair though,

what works for a company that has 2 billion users, isn't necessarily the same solution you need for your 20. Yeah, that's true. Yeah, very true. And yeah, it's probably fine. I kind of made the joke about big data not being big data. That's for a long time when the term big data first started coming out and people started talking about Hadoop. It was all the rage. And then you'd talk to people about their big data solutions and find out they people started talking about Hadoop. It was all the rage and then you'd talk to people

about their big data solutions and find out they only had 10 gigs of data ever. You know, it's like, oh, I think when you read those articles and you do the stuff like they're talking about terabytes and so you're trying to go through all these extra steps to deal with this overhead for something that could

be done with a relational database, no problem. But I mean, part of that too, is it just, it's the, it's just like common to all of, all of us though, like as developers, like we always want to play with like,

Hey, what's the new big cool thing. Right. Yep. So, I mean, I get it. Like,

you know, it, it kind of like when we tease Alan about him, like trying to solve, you know, something for a billion concurrent users, right? Like, yeah, I, you know it kind of like when we tease alan about him like trying to solve you know something for a billion concurrent users right like yeah i i you know i get it you might not need it but that's what you want to you wanted to like have an excuse to like play with some code works with whatever that thing is right yeah a ring a ringo is the name i was trying to come up with uh thanks andrew diamond uh but i want to mention, too, that Docker, thanks to us living in a containerized world,

it's made it so easy to spin up like a Neo4j or Dgraph or any of these others really easily. So you can just kind of run a single command and you're running in the database and you can find a getting started guide on any of their websites and be just ingesting data and experimenting with that kind of workflow, I mean, instantly. So prototyping is something you can do in a couple hours to see if it's something you're even worth spending some time in. I will say the query languages that I've spent some time with, Cypher is one I was trying to remember of in Neo4j,

is at first I was horrified because I saw there were like these arrows pointing in these lines and it didn't even look like programming. And so I was kind of initially put off. But when I realized how quickly and easily it mapped to the English expression of the problem, I kind of fell in love with it a little bit. And so I'm excited to go back and play with graph databases one of these days. Yeah, I remember because Cypher was in the book too and and um as soon as you said it i was like oh yeah we're not talking about cypher as in anything to do with cryptography we're talking about cypher as in a reference to the character from the movie the matrix right yep oh yeah i should mention too

that there's uh there's a couple of really cool things going on there's a lot of innovation particularly in the graph database space right now there's um this thing called tinker pop which i don't fully understand but basically the idea is that it kind of um provides you like a common platform for using uh like one language across multiple different vendors because the deal is that it's kind of like sql where all relational databases kind of do the same kind of thing same with graph databases they're all kind of built around the same kind of model so the fact that there's like different programming languages for them uh it's kind of weird and so what the tink prop does is kind of provides a platform you for you to kind of pick the language that you want to use to work with the database that you want to use so

it's kind of this weird division between the language that you work with and the database that you use you normally think of those things being tightly bound, but TinkerPop drives a line between and says, yeah, you want to use the Gremlin language with a Neo4j. Fine. Do that. If you want to use Cypher for a Rango DB, like,

all right, yeah, sure. We'll do it. And so it'll kind of act as like a translation layer to kind of type, uh, or to translate into the language that the database needs based on the language that you want to use. It's like a transpiler for Graph SQL.

That's pretty awesome. Yeah. Oh, or as Alan would say, that's freaking awesome. Yeah. That was the thing that somebody was teasing about. Yeah, okay. That's awesome well i recommend looking at the graph uh the gram the gremlin language in particular and uh it's got

a super cute website and uh it says some really good uh examples of what that language looks like i encourage just going to look at it just because of how funky it looks to if you're used to doing like kind of imperative like c-sharp javascript like languages you go look at a just because of how funky it looks if you're used to doing like kind of imperative like C sharp or JavaScript like languages you go look at a gremlin and look at the way you program that or go look at a cipher for Neo4j and just look at an example query it's like what?

This works? It does, it's pretty cool. Huh. Well you know they gotta be serious when they have like a math formula as part of their logo. Yeah, that's true. Oh, you know, one thing I wanted to mention. So I've done some deal out of the GCP podcast, Google Cloud Platform Podcast.

And I listened to an episode talking about Cloud SQL, which is basically they're kind of bandaged either Postgres or MySQL or SQL Server. And the person on there mentioned dealing with SQL queries that were 50 lines long and how miserable that was in relational databases. And you're like, only 50? I'm like, if you've only seen the horrible, horrible things I've done to relational databases, 50 lines is nothing. Yeah. Like, let me get some unions in there. Let me get some CTEs. Like, oh, man, I could have 50 lines.

It's just variables. Give me a break. Wow, that's pretty cool. I didn't realize there was a GCP podcast, but why would there not be one? So, yeah. It's really good. I'll tell you, though, they pan. So, if they have two authors, they'll have one that's heavy on the left, one heavy on the right.

And I hate that so much. You got a dual mono. Come on. Dual mono. I've tweeted him about it. Really? Yeah. We'll see. Yeah. the left one heavy on the right and i i oh yeah that's so much you got a dual mono come on dual mono i've tweeted him about it really yeah we'll see yeah they uh they'll also google does a kubernetes podcast that's super good but they also do the panning thing i think you have the same editors oh man so you've been warned i need an app that mixes the mono that'd be a great feature this episode is sponsored by clubhouse clubhouse is a fast and enjoyable project management platform that breaks down silos and brings teams together to ship value, not just features.

I mean, let's face it. Slow, confusing UX is so last decade. Clubhouse is lightning fast built for today's software teams with only the features and best practices you need to succeed and nothing more. And here are a few highlights about Clubhouse. They've got flexible workflows, which are easily customizable workflow states for teams or projects of any size.

They've got advanced filtering, which lets you quickly filter by project or team to see how everything's progressing. And you've got sprint planning, so you can set your weekly priorities with iterations and you can let Clubhouse run the schedule for you. And then Clubhouse integrates with the tools that you love. So they tie into things like support your services and workflow.

So you can get notifications, for example, about a story in Slack or update the status of a story with a pull request. So these are really great integrations with the tools that I'm already using, like Figma or whatever. And so you can build your own integration with their APIs as well in case there's not an integration that's publicly available for you already. Yeah, and make one that fits your specific needs. And Clubhouse is an enjoyable collaboration tool, easily drag and drop UI. There's dark mode. I mean, what developer doesn't like dark mode? We've actually had that debate many times, you know, dark mode or light mode, right? Hey, you want to make fun, just have some light with, you know, some light, you know,

communication along with what your pull request is or, you know, what your ticket status is. There's emoji reactions and more. So when you're doing your best work is when your team is clicking and life is good. Now, Clubhouse has recently made all of their core features completely free for teams up to 10 users. I can't like we would have to say like Alan would say that is freaking beautiful. And they're offering Clubhouse. Sorry, they're offering CodingBlocks listeners two additional free months on any paid plan with unlimited users and access to premium features. So give it a try. You can go to clubhouse.io slash CodingBlocks. That's clubhouse.io slash CodingBlocks and find out what makes Clubhouse so awesome.

All right. So with that, I want to mention some resources we like. I will have a link here to the book, Designing Data-Intensive Applications, that we've been talking about for a long time. In particular, tonight, we talked about just different query languages and a little bit about graph databases. And then moving on to somebody's favorite portion of the show.

It's Alan's favorite portion of the show. It's Alan's favorite portion of the show. It's the tip of the week. And yeah, so he isn't here, obviously, as we've mentioned. But in his spirit, since he would normally have like 15 billion tips, we each came up with 20 of our own. You ready to go with your 20 first? Let me grab a drink of water first, but then yeah. Okay.

No, I'm just kidding. There's not 20. Okay. So I have a couple tips here. One was they're both based off of a conversation we had with E. Lickman in Slack. And I think we talked about this before, but I don't know if we ever made it officially a tip of the week. So I thought, okay, fine, you know, this would be the time to do it. And that is, if you've ever wanted the equivalent, if you're used to a Linux environment,

and you're used to being able to do something like a tail minus F on a file and just watch it, you know, like a log file, right? And you can like, as new traffic comes in, you can just watch the next line of the log file go in and you can keep doing that forever, right? But you want the equivalent of that on a Windows environment. It can be done. So it would be a PowerShell commandlet that you would use called get content. So you would say get content, the file name dash weight. And that's how it would work there. But there was another really cool one that was shared with us in that Slack conversation that so I thought I'll share and it's gonna be difficult to follow in, you know, just listening to it.

So just know that you can go to the show notes and you'll be able to like see this. And if you're like whatever your podcast player is, like you could just hit the show notes directly as you're listening to us and follow along and you'll be able to see this example there. But if you wanted to be able to recursively find files that match a particular regular expression pattern and you wanted to see some context around that to like maybe you're looking for something. And in his case, what he was looking for was like different includes. They were happening because in different parts of his application, like different, like imagine you want to look at, hey, let's find all of the using statements, you know, that are using some particular library or

something like that. Imagine that kind of scenario, right? But you, your mind may vary for why you might want to use it. But you could use another set of PowerShell command lists that you start with get child item, then whatever files you're going to look for. So let's just say it was like star dot C sharp or CS, right? So get child item star dot CS dash recurse, because you want to recursively look through all the subdirectories, then a pipe, then select dash string, dash pattern, and then in single quotes, enter your regular expression pattern, and then dash context, and then the number of lines that you want. So, you know, the lines before and the lines after the context of what you're at. So in this example,

if you wanted to see like the first 10 lines above whatever your pattern is, you'd say 10, comma, zero. So again, that's going to be a hard one to lines above whatever your pattern is, you'd say 10 comma zero. So again, I, that's going to be a hard one to follow along with, but know that you can go to the show notes and see the example of what I'm talking about. And, and,

you know, bless PowerShell for having auto-complete. Yes. Terminal. Oh, and the best part that I love about, uh, PowerShell's auto-complete though,

is it's not just on the command, but on the, uh, uh, the, the, PowerShell's autocomplete though is it's not just on the command but on the the optional arguments to the command as well so like even with like dash recurse right you could type in like dash REC tab and it'll finish

out recurse for you it's so awesome yeah so I know you can do the same thing in grep I think it's like dash capital C for context. It's like dash capital B for before maybe, I don't know. Yeah. But you got to like, remember, I remember like years ago, they were, I remember hearing talk about like bash was going to get tab completion on the arguments, but I guess it just never happened. Yeah. There's probably some plugin or something you can do that will kind of

help out with that. But I've still never known anyone to actually use that day to day. Yeah, or if it ever did happen, maybe I'm just not lucky enough to be on a distro that uses it. Yeah, and most of my bashing these days happens inside the context of like Docker containers. So I'm definitely not going to be installing, you know, whatever plugins on those things. Yeah, I'm with you there. So, you know what? We didn't mention that mention that elictman brought these tips to our attention while pulling tips from the website oh yeah yeah yeah because he because he he thought that we had already mentioned uh something similar

to this and that's that's what brought up the whole conversation so yeah i did something similar uh and after episode 55 and i gathered all the the tips up to episode 55 and then i never did it again i didn't maintain it but now he sent us the script so we could start it up again all right so for my tips uh so i don't know if you saw this one yet outlaw uh this was sent to us by MicroG who we talk about all the time and it's called Application Inspector and this is in the Microsoft GitHub account but it's not just C Sharp

what it is is a source code analyzer that surfaces features of interest and other characteristics to answer the question what's in it for code bases and it uses static analysis and a rules engine to answer the question, what's in it for code basis? And it uses static analysis and a rules engine.

And so if you scroll down to the link we've got in the show notes here and take a look at like a screenshot, you can see that you can run this thing on source code and it'll give you like a little highlighted thing. This is basically like, hey, this integrates azure and twitter and postgres and uh you know i don't know environment variables and github and it doesn't integrate or we didn't find any other integrations for any of these other services so have you ever seen that um that website that does this sort of thing where it's basically like what's it built on and people manually kind of maintain that stuff this is a programmatic way of finding that out.

So maybe if you're investigating using some new service or project or something on GitHub, you could download that repo just like you normally would and then run this service on it. And it's going to go out and sniff for known integrations and say, hey, this talks out to Twitter and to, I don't know, Coinbase. So maybe you don't want to run it because this might be trying to steal your Bitcoins or whatever. Huh. That is. That's pretty cool.

That is. It's got OS system changes too. So it can like try to detect if it's going to write to disk or read to disk or whatever. Of course, like all that stuff, you know, it's basically using kind of thumbprint type technology where it's trying to like our fingerprint. So it's trying to figure out like based on common usage pattern. So I'm sure it's not going to be perfect, but still, it's pretty nice just to even be able to see like a high level. Like if someone's like really trying to trick you, then this probably isn't going to help too much because you can obviously create a lot of this stuff. But I mean, just for a general general purpose like what's this code going to

do if i start it like this is a pretty great tool yeah i kind of want to run out on my own stuff now and i do have another one uh and this one has a special place in my heart this is from uh notorious nvg nvg the mad viking god and this is Muzzle, which is Mac only. But what it does is it stops notifications while you're screen sharing. I don't know how they're detecting it. But if you've ever been in a meeting and you're sharing your screen

like Al and I work remote and so this happens all the time. Do you have notifications turned on, say for Slack or something, and someone messages you sometimes it'll pop up and be visible on your screen. And there's definitely been times when I've been in your meetings or I've been watching someone else's screen share. Well, like someone will message them maybe who's in the meetings to kind of say like, you know, hey, that's crazy or, you know, something that is just awkward that you don't necessarily want to be seen. And it stinks that that's public.

It's just maybe if you sent that message, you didn't realize it was going to show up on the big screen for everybody to see. You're just trying to send a private message. And so this is a tool that will silence all of those Mac notifications while you're sharing your screen. Now, how is that different than the built-in Do Not Disturb? Because, like, I forget. It's been there for a couple versions now in mac os right so this will automatically turn on do not disturb when you screen share oh okay so so it's just using the built-in functionality is its real thing is it just

automatically does it during a screen share yeah so it's not doing anything it's like a plug-in for hangouts all these other things and so you start the screen share and it immediately turns it on. So you don't have to remember that at any time. I got you. I got you. Oh, yeah. That is helpful. Yeah.

And if you go to the website, it's actually super cool. So you go to muzzleapp.com. It's got notifications showing up on the right. There are the kinds of notifications that you would not want showing up on uh on the screen share like hey it's your court date reminder tomorrow or uh somebody breaking up with you or your parents kicking out of your house oh some of them are uh not even safe for work yep for sure yeah the names it's yeah so it's it's comical it's worth going to the website just to see. Yeah, it is pretty comical. I'm like, I'm just sitting here. I'm lost just watching it.

It has all of my attention. Yeah, some of these are really good. This is actually, this is really good. So let's just move on with the show and then I'm going to watch this for a little while. Yeah, that is hilarious. Wait till the one from your mom comes in. Yeah, wait for that one from your, your mom comes in. Yeah. Wait,

wait for that one. All right. That's all I'm going to say. Gosh. All right. Well, uh, like Joe mentioned earlier,

be sure to leave us, uh, a comment on this episode, uh, for your chance to win a copy of the book. And, uh, with that,

be sure to, uh, subscribe to us on iTunes, Spotify, Stitcher, or more using your favorite podcast app in case if like a friend gave you a link or they're letting you listen on their device or something like that. You can find us at www.codingblocks.net. And while you're there, you can also find some helpful links at slash review where you could leave us a review on one of those favorite platforms. And while you're at CodingBlocks.net, check out our show notes, examples, discussion, and more. And send your feedback, questions, and rants to Slack.

And you can get there by going to CodingBlocks.net slash Slack and sending yourself an invite. And you can also follow us on Twitter at C blocks or send us a message uh over there or you can go to heading codingbox.net and find all our social links at the top of the page if they're still there i don't know i haven't checked today yeah no they're there all right well another twit is in the can oh wait wrong show