Screaming in the Cloud - Throwing Houlihans at MongoDB with Rick Houlihan

Episode Date: March 24, 2022

About RickI lead the developer relations team for strategic accounts at MongoDB. My responsibilities include defining technical standards for the global strategic accounts team and consulting with the largest customers and opportunities for the business. My role spans technology sectors and as part of my engagements I routinely provide guidance on industry best practices, technology transformation, distributed systems implementation, cloud migration, and more. I led the architecture and design effort at Amazon for migrating thousands of relational workloads from RDBMS to NoSQL and built the center of excellence team responsible for defining the best practices and design patterns used today by thousands of Amazon internal service teams and AWS customers. I currently operate as the technical leader for our global strategic account teams to build the market for MongoDB technology by facilitating center of excellence capabilities within our customer organizations through training, evangelism, and direct design consultation activities.30+ years of software and IT expertise.9 patents in Cloud Virtualization, Complex Event Processing, Root Cause Analysis, Microprocessor Architecture, and NoSQL Database technology.Links:MongoDB: https://www.mongodb.com/Twitter: https://twitter.com/houlihan_rick

Hello and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. The company 0x4447 builds products to increase standardization and security in AWS organizations. And they do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain, and fail-tolerant solutions.

One of those is their VPN product, built on top of the popular OpenVPN project, which has no license restrictions. You're only limited by the network card in the instance. To learn more, visit snark.cloud slash deploy and go. That's snark.cloud slash deploy and go. All one word. This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of hello world demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services in infrastructure, networking, databases, observability, management, and security.

And let me be clear here, it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisk next to the word free. This is actually free. No asterisk. Start now. Visit snark.cloud slash oci-free. That's snark.cloud slash oci-free. Welcome to Screaming in the Cloud.

I'm Corey Quinn. A year or two before the pandemic hit, I went on a magical journey to a mythical place called Australia. I know, I was as shocked as anyone to figure out that this was in fact real. And while I was there, I gave the opening keynote at a conference that was called Latency Conf, which is great because there's a heck of a time zone shift, and I imagine that's what it's talking about. The closing keynote was delivered by someone I hadn't really heard of before. And he started talking about single table design with respect to DynamoDB, which, okay, great. Let's see what he's got to say.

And the talk started off engaging and entertaining and a high level overview and then got deeper and deeper and deeper. And I felt, can I please be excused? My brain is full. That talk was delivered by Rick Houlihan, who now is the Director of Developer Relations for Strategic Accounts over at MongoDB.

And I'm fortunate enough to be able to get him here to more or less break down some of what he was saying back then, catch up with what he's been up to, and more or less suffer my slings and arrows. Rick, thank you for joining me. Great. Thanks, Corey. I really appreciate you.

You brought back some memories and trip down memory lane there. And actually, interestingly enough, that was the world's introduction to see single table design was that that was my dry run rehearsal for reinvent 2018 was where I delivered that talk. And it has become since the most popular two weeks before reinvent, which was just a great thing. I I'd been invited to go. Why not? I figured I'd see a couple of clients I had out in that direction. And I learned things like Australia is a big place.

So doing a one-week trip, including Sydney, Melbourne, and Perth, don't do that. I had no idea that it took so long to fly from one side to the other, right? I mean, that's a long plane. Oh, yeah. And you were working at AWS at the time. So I can only assume that they basically stuffed you into a dog kennel and threw you underneath the seating area, given their travel policy. Well, you know, I have the, actually at the time they just upgraded the policy to allow the intermediate seating, right? So if you wanted to get the IOs. Ooh, expander, expander.

Yes, yes. I could get the little bit of extra leg room so I didn't have to have my knees shoved into somebody's back, but it was good. So let's talk about, I guess, we'll call it the elephant in the room. You were at MongoDB, where you were a big proponent of the whole NoSQL side of the world. Then you went to go work at AWS and you carried the good word of DynamoDB far and wide. It made an impression. I built my entire newsletter pipeline production system

on top of DynamoDB. It has the same data in three different tables because I'm not good at listening or at computers. But right now you're back at Mongo and it's easy to jump to the conclusion of, oh, you're just shilling for whoever it is that happens to assign your paycheck. And at this point, what's the authenticity story? But I've been paying attention to what you've been saying. And I think that's a bad take because you have been saying the same things all along since before you were on the Dynamo side of it.

I do some research for this show. And you've been advocating for outcomes and the right ways to do things. How do you view it? That's basically the story here, right? I've always been a proponent of NoSQL. You know, what I took, the knowledge, it was interesting. The knowledge I took from MongoDB evolved as I went to AWS and I delivered, you know, thousands of applications and deployed workloads that I'd never even imagined I would have my hands on before I went there. I mean,

honestly, what a great place it was to cut your teeth on data modeling at scale, right? I mean, there is no greater scale. That's when you learn where things break. And honestly, a lot of the lessons I took from MongoDB, well, when I applied them at scale at AWS, they worked with varying levels of success, and we had to evolve those into the sets of design patterns, which I started to propose for DynamoDB customers, which had been highly effective. I still believe in all those patterns. I would never tell somebody that they need to drop everything and run to MongoDB. But again, all those patterns apply to MongoDB too, right? A lot, I wouldn't say all of them, but many of them, right? So I'm a proponent of NoSQL. And I think we talked before the call a

little bit about, if I was out there hawking relational technology right now and saying RDBMS is the future, then everybody who criticizes anything I say, I would absolutely have to say that there's some validity there. But I'm not saying anything different I've ever said. MongoDB announced serverless, if you remember, in July. And that was a big turning point for me because the API that we offer is the developer experience for MongoDB is unmatched. And this is what I've talked to people now. The patterns that I've always proposed, I still model data the same way. I don't do it any different.

And I've always said, if you go back to my earliest sessions on NoSQL, it's all the same. It doesn't matter if it's MongoDB, DynamoDB, or any other technology. I've always shown people how to model their data in NoSQL. And I don't care what database you're using. I've actually helped MongoDB customers do their job better over the years as well. Oh, yeah. And looking back at some of your early talks as well, you passed my test for, is this person a shill? Because you wound up in those talks addressing head on, when is a relational model the right thing to do? And then you put the answers up on a slide

and what it didn't distill down to, if you're a fool, because there are use cases where if you don't know your access patterns, if you have certain constraints and requirements, then yeah, you have always been an advocate for doing the right thing for the workload. And in my experience, from my

use cases, when I looked at MongoDB previously, it was not a fit for me. It was very much a you run this on an instance basis, you have to handle all this stuff. Keeping it in triplicate in three different DynamoDB tables, my newsletter production

pipeline now, including backups and the rest, the DynamoDB portion has climbed to the princely sum of $1.30 a month, give or take. Yes, exactly. So there's no answer for that there. Now that Mongo serverless is coming out into the world. Oh, okay. This starts to be a lot more compelling. It starts to be a lot more flexible.

I was just going to say, for your use case there, Corey, you're probably looking at a very similar pricing experience now with the MongoDB serverless, especially when you look at the pricing model. It's very close to the on-demand table model. It actually has discounted tiering above it, which I haven't really broken it down yet against a provisioned capacity model. But, you know, there's a lot of complexity in DynamoDB pricing, and they're working on this. They'll get better at it as well. But right now, you have on-demand, you have provision throughput, you have reserve capacity allocations. And there's a time and place for all of those, but it puts the, again, it's just complexity, right?

This is the problem that I've always had with DynamoDB. I just wish that we'd spent more time on improving the developer experience, right? Enhancing the API, implementing some of these features that help. Let's make single table design a first- class citizen of the DynamoDB API. Right now, I don't want to say redheaded stepchild. I have two redheaded children and my wife is redheaded.

But that's the way it's treated, right? It's treated like a stepchild. You know, it's like, come on, we're fully funding the solutions within our own umbrella that are competing with ourselves. And at the same time, we're letting the DynamoDB API languish while our competitors are moving ahead. And eventually, it just becomes, okay, guys, I want to work with the best tooling on the market. That's really what it came down to. As long as DynamoDB was the king of serverless, yes, absolutely, best tooling on the market. And they still are the leader, right? There's no doubt that DynamoDB was the king of serverless, yes, absolutely. Best tooling on the market. And they still are the leader, right?

There's no doubt that DynamoDB is ahead in the serverless landscape. The MongoDB solution is in its nascency. It's going to be here. It's going to be great. That's part of what I'm here for. And that's, again, getting back to why do you make the move? I want to be part of this. That's really what it comes down to.

One of the things that I know that was my own bias has always been that if I'm looking at something like, that I'm looking at my customer environments to see what's there, I can see DynamoDB because it has its own line item in the bill. MongoDB is generally either buried in marketplace charges or it's running on a bunch of EC2 instances, or it just shows up as data transfer. So it's not as top of mind for the way that I view things through the lens of billing. So that does inform my perception,

but I also know that when I'm talking to large-scale companies about what they're doing, when they're going all in on AWS, a number of them still choose things like Mongo. When I've asked them why that is, sometimes you get the answer of, oh, legacy is what we built on before.

Cool, great. Other times it's a, we're not planning to leave, but if we ever wanted to go somewhere else, it's nice to not have to reimagine the entire data architecture and change the integration points start to finish because migrations are hard enough without that. That's right.

And there is validity to the idea of a strategic exodus being possible, even if it's not something you're actively building for all the time, which I generally advise people not to do. Yeah. There's a couple of things that have occurred over the last couple of years that have changed the enterprise CIO and CTO's assessment of risk, right? Risk is the number one decision factor in a CTO's portfolio and a CIO's decision-making process. What is the risk?

What is the impact of that risk? Do I need to mitigate that risk or do I accept that risk? So right now, what you've seen is with COVID, people have realized that on-prem infrastructure is a risk. It used to be an asset. Now it's a risk. Those personnel that have to run that on-prem infrastructure, hey, what happens when they're not available? The infrastructure is at risk.

So offloading that to cloud providers is the natural solution. So what happens when you offload to a cloud provider and IAD goes down or US East 1 goes down? We call it IAD or we used to call it IAD internally at AWS when I was there because the regions were named by airport codes, but it's US East 1. How many times has US East 1 had problems? Do you want to really be the guy that every time US East 1 goes down, you're in trouble? What happens when people in US East 1 have trouble? Where do they go? Down, generally speaking. Well, if they're well architected, right? If they're well architected, what do they do? They go to USS West 2. How much infrastructure

does USS West 2 have? So if everybody in US East 1 is well architected, then they all go to USS West 2. What happens in USS West 2? And I guarantee you, and I've been warning about this at AWS for years, there's a cascade failure coming. And it's going to be coming because we're well architecting everybody to fail over from our largest region to our smaller regions. And those smaller regions, they cannot take the load. And nobody's doing any of that planning. So, you know, sooner or later, what you're going to see is dominoes fall. Okay.

And it's not just going to be U.S. East 1. It's going to be U.S. East 1 failed and the rollover caused a cascade failure in U.S. West 2, which caused a cascade. Because everyone's failing over during the event the same way. That's right. And also, again, not to dunk on them unnecessarily, but when US East 1 goes down, a lot of the control plane services freeze up. Oh, of course they do.

We're not single point of failure, right? Uh-huh. Exactly. There you go. Route 53. That actually surprised me. You're using DynamoDB instead of Route 53 as your primary database. So I'm actually... I had to move one workload off of Dynamo to Route 53 for this issue number because I have to practice what I preach. That's right.

It made the thing slower and a little bit less caching. Yeah, sure. Okay, I can understand that. But it made the architecture diagram a little bit more head-scratching. And really, that's what it's all about, getting a high score. Right. So if you think about your data, right, I mean, would you rather be running on an infrastructure that's tied to a cloud provider that could experience these kinds of regional failures and cascade failures? Or would you rather have your data infrastructure go across cloud providers so that when one provider has problems, you can just go ahead and switch the light bulb over on the other one and ramp right back up, right?

And honestly, you run in active-active configurations and that kind of deployment design. And you're never going to go down. You're always going to be the one that stays up. The theory is sound. The challenge I've had in production with trying these things is that one, the thing that winds up handling the failover piece is often causes more outage than the underlying stuff itself. Sure.

Two, when you're building something to run a workload to run in multiple cloud providers, you're forced to use a lot of lowest lot of stuff. Yeah, yeah, totally. I hear that all the time. Unless you're actively running it in both places, it's like a DR plan, which doesn't survive the next commit to the code base. I totally buy that. You're talking about the stack stack duplication,

all that kind of that's like, that's an overhead and complexity. I don't worry about the data layer, right? The data layer, we're talking about shipping data layer. Oh, everything you're saying makes perfect sense, right? And honestly, you know, let's put it this way. If this is what you want to do. What do you mean identity management and security handle working differently? Oh, that's a different team's problem. Oh, I miss those days. Yeah, no, totally. Right. It's not my deal. But, you know, I mean, honestly, it's not a deal that somebody wants to manage themselves is moving that data around. The data is the lock-in.

The data is the thing that ties you to- And the cost of moving it around in some cases, too. That's exactly right. So having infrastructure that spans providers, that spans both on-prem and cloud, potentially that can span multiple on-prem locations, man, I mean, that's power. And MongoDB provides that. I mean, DynamoDB can't. And that's really one of the biggest limitations that it will always have. And we talked about, and I still believe in the power of global tables and multi-region deployments and everything. It's all real. But these types of scenarios, I think this is the next generation of failure that cloud providers are not really prepared for. They haven't experienced it.

They don't know what it's even going to look like. And I don't think you want to be tied to a single provider when these things start happening, right? If you have a large amount of infrastructure deployed someplace, it just seems like that's a risk that you're running at these days. And you can mitigate that risk somewhat by going with a MongoDB Atlas. I agree, all those other considerations. But I also heard this is, it's a lot of FUD too, right? There's a lot of FUD in that, right? Because if you think about it, I can deploy technologies in ways on any cloud provider,

they're going to be cloud provider agnostic, right? I can use containerized technologies, Kubernetes. I can use, hell, I'm not even afraid to use Lambda functions and just put a wrapper around that code and deploy it both as a Lambda or a cloud function in GCP, the code's almost the same in many cases, right? What it's doing with the data, you can code this stuff in a way I used to do it all the time. You abstract the data layer, right?

Create a DAO. How about a CAL, a cloud access layer, right? You know? Oh, geez. I wish on some level we could go down some of these paths. Someone asked me once a while back of, well, you seem to have

a lot of opinions on this. Do you think you could build a better cloud than AWS? And my answer took a bit by surprise of absolutely. Step one, I want similar resources to give me $20 billion to spend. Then I'm going to hire the smart people. And not that we're somehow smarter or better or anything else than the people who built AWS originally, but now we have 15 years of experience to fall back on.

Exactly. I wouldn't make that mistake again. Exactly. Don't need to worry about that. Yeah, exactly. You can't just turn off a cloud service and relaunch it with a completely different interface and API and the rest. Yeah, people who criticize, you know, services like DynamoDB and other AWS services. Look, these things are like any kind of retooling of these services. It's like rebuilding the engine on the airplane while it's flying.

Oh, yeah. And you have to do it with a level of service assurance that, I mean, come on, DynamoDB provides four nines out of the box, right? Five nines if you turn on global tables. And they're doing this at the same time as they have pipeline releases dropping regularly, right? So you can imagine what kind of unit testing goes on there, what kind of canary deployments are happening. It's just, it's an amazing infrastructure that they maintain, incredibly complex. in some ways, these are lessons that we need to learn in MongoDB if we're going to be successful operating a shared backplane serverless, you know, processing fabric. We have to look at what DynamoDB does right.

And we need to build our own infrastructure that mirrors those things, right? And in some ways, these things are there. In some ways, they're working on it. In some ways, we've got a long ways to go. But, you know, I mean, this is the exciting part of that journey for me. Now, in my case, I focus on strategic accounts, right? Strategic accounts are big. You know, they're the potential to be our whale customers, right? These are probably not customers who would be all that interested in serverless, right? They're customers that would be more interested in provisioned infrastructure because they're the people that I talked to when I was at DynamoDB.

I would be talking to customers who are interested in like reserved capacity allocations, right? Yeah, I want to ask you about that. Your developer advocacy, which I get, for strategic accounts. And I'm trying to wrap my head around why do you accounts have big ones, potential to spend lots of stuff. Why do they need special developer advocacy? Well, it's funny because one of the reasons why I started talking to Mark Porter about this was the fact that the overlap is really around the engagements that I ran when I was doing the Amazon retail migration. When Amazon retail started to move to NoSQL, we deprecated 3,000 Oracle Server instances. We moved a large percentage of those workloads to NoSQL. The vast majority probably just were lift and shift into RDS and whatnot

because they were too small, too old, not worth upgrading, whatnot. But every single tier, what we call tier one service, every money-making service was redesigned and redeployed on DynamoDB, right? So we're talking about 25,000 developers that we had to ramp. This is back four years ago. Now we have like 75,000. But back then we had 25,000 global developers. We had a technology shift, a fundamental paradigm shift between relational modeling and NoSQL modeling. And the whole entire organization needed to get up to speed. So it was about creating a center of excellence. It was about operating as an office of the CTO within the organization to drive this technology into the DNA of our company. And so that exercise was actually incredibly informative, educational in

that process of executing a technology transformation in a major enterprise. And this is something that we want to reproduce. And it's actually what I did for Dynamo as well, really, more than anything. Yes, I was on Twitter. I was on Twitch. I did a lot of these things that were kind of developer advocate activities. But my primary job at AWS was working with large strategic customers, enabling their teams, teaching them how to model their data in NoSQL, and helping them cross the chasm from relational. And that is advocacy.

The way I do it is I use their workloads. I use the customers, project teams themselves. I break down their models. I break down their access patterns. And when I leave, essentially, the whole day of design reviews, we'll walk through 12 or 15 workloads. And when I leave, these guys have an idea. How would I do it if I wanted to use NoSQL? Right. I give them enough breadcrumbs so that they can actually say, OK, if I want to take it to the next step, I can do it without calling up and say, hey, can we get a professional services team in here? Right. So it's kind of developer advocacy and it's kind of not right. We're kind of recognizing that these are whales. These are customers with internal resources that are so

huge they could suck our developer advocacy team in and chew it up, right? So what we're trying to do is form a focus team that can hit hard and move the needle inside the accounts. That's what I'm doing. Essentially, it's the same work I did for AWS, for DynamoDB. I'm just doing it for, you know,

they traded for a new quarterback. Let's put it that way. This episode is sponsored in part by our friends at Sysdig. Sysdig is the solution for securing DevOps. They have a blog post that went up recently about how an insecure AWS Lambda function could be used as a pivot point to get access into your environment.

They've also gone deep in depth with a bunch of other approaches to how DevOps and security are inextricably linked. To learn more, visit sysdig.com and tell them I sent you. That's S-Y-S-D-I-G dot com. My thanks to them for their continued support of this ridiculous nonsense. So one thing that I find appealing about the approach maps to what I do in the world of cloud economics, where I, like in my own environment, our AWS bill is creeping up again. We have 14 AWS accounts and that's a little over $900 a month now, which is big money, big money.

In the context of running a company that no one notices or cares. And our customers spend hundreds of millions a year pretty commonly. So I see the stuff in the big accounts and I see the stuff in the tiny account here. Honestly, the more interesting stuff is generally in on the smaller side of the scale, just because you're not going to have a misconfiguration costing a third of your bill when a third of your bill is $80 million a year. That's correct. If you do, then that's a real problem, right? Oh, yeah.

It's very much two opposite ends of a very broad spectrum. And advice for folks in one of those situations is often disastrous to folks on the other side of that. That's right. That's right. I mean, at some scale, managing granularity hurts you, right? The overhead of trying to keep your costs, you know, but at the same time, it's just a different measure of cost. There's a different granularity that you're looking at, right? I mean, things below a certain level stop becoming important when the budgets start to hit a certain scale or a certain size, right?

For certain workloads, things that I care about with my dollar a month Dynamo spend, if I were to move that to Mongo serverless, great. My considerations are radically different than a company that is spending millions a month on their database structure. Well, that's right. We don't care about the pennies. We care about, is it going to work? How do we back it up? What's the replication factor? And that, but also it's more than that. It's, you know, for me, from my perspective, it really comes down to that, you know, companies are spending millions of dollars a year in database services. These are companies that are spending 10 times that, five times that in developer expense, right? Building services, maintaining the code that runs that these services run.

The biggest problem I had with MongoDB is the level of code complexity. It's just, you know, it's a cut after cut after cut, right? And the way I kind of described the experience and other people have described it to me, I didn't come up with this analogy. I had a customer tell me this as they were leaving DynamoDB. DynamoDB is death by a thousand cuts. You love it. You start using it. You find a little problem. You start fixing it. You start fixing it. You start fixing it. You come up with the pattern. Talk to Rick. He'll come up with something. He'll tell you how to do that. Okay. And you know, how many customers did I would do this with? You know,

and it's honestly, they're 15 minute phone calls for me, but every single one of those 15 minute phone calls turns in eight hours of developer time, writing the code, debugging it, deploying it over and over again. It's making sure it's Have another 15-minute call with Rick, et cetera, et cetera. Exactly. And it's like, okay, eventually they just get tired of it, right? And I actually had a customer that tell me, a big customer, tell me flat out, yeah, you proved that the DynamoDB can support our workload and it'll probably do it cheaper, but I don't have a half a dozen RICs on my team, right? I don't have any RICs on my team. You know, I can't be getting you in here every single time we have to do a complex data model overhaul, right? And this was, granted,

it was one of the more complex implementations that I've ever done. In order to make it work, I had to overload the freaking table with multiple access patterns on the partition key, something I had never done in my life. I made it work, but it was just, honestly, that was an exercise to me that taught me something. If I have to do this, it's unnatural. Okay. And that's, you know what I mean? And honestly, there's API improvements that we could have done to make that less of a problem. It's not like we haven't known since the last, I don't know, I joined the company that a thousand WCUs per storage partition was pretty small. Okay. We've kind of known that for, I don't know, since DynamoDB was invented. As a matter of fact, from what I know and talking to people who were around back then, that was a huge

bone of contention back in the day, right? A thousand WCUs, 10 gigabytes. There were a lot of the PEs on the team that were going, no way, no way, that's way too small. And then there were other people that were like, nah, nobody's ever going to need more than that. And you know, a lot of this was based on the analysis. Oh, nothing ever survives first contact from the customer, particularly a customer who is not themselves deeply familiar with what's happening under the hood. Like I had this problem back when I was a traveling trainer for Puppet for a while. It was great. Well, Puppet is obviously a piece of crap because everyone I talk to has problems with it. So I was one of the early developers behind SaltStack.

And, ah, this is going to be a thing of beauty and it'll be awesome. And that lasted until I saw what one, the first time I saw what someone had done with it in the wild. It was, oh, okay. Okay, that's how, yeah, I never thought about that, right? Happy Path. We all, we all love the happy path, right? As we're, as we're working with technology, as we figure out how we like to use it, we all use it that way. Of course you can solve any problem you want the way that you'd like to solve it. But assuming someone else takes that clay, they mold a different statue and you go, Oh, I didn't realize it could look like that. Right.

Exactly. So here's one for you that I've been, I still struggle with this from time to time, but why would I, if I'm building something out, well, first off, why on earth would I do that? I have people for that who are good at things. But if I'm building something out and it has a database layer, why would someone pick, choose NoSQL over SQL? And let me be clear here, and I'm coming at this from the perspective of someone who, basically me a few years ago, who has no real understanding of what databases are. So my mental model of a database is Microsoft Excel, where I can fire up a table of these things. Hey, well, then you know what?

Then you should love NoSQL because that's kind of the best analogy of what is NoSQL. It's like a spreadsheet, right? Whereas a relational database is like a bunch of spreadsheets, each with their own types of rows, right? My mind was blown with relational stuff. One other day I was, wait, you could have multiple tables? What do you think relational meant there, buddy?

Yeah, yeah, yeah. My map of NoSQL was always key and value and that was it. And that's all it can be. Sure, for some things, that's what I use, but not everything. That's right. So, you know, the bottom line is

when you think about the relational database, it all goes back to, you know, the first paper ever written on the relational model, Edgar Codd. I can't remember the exact title, but he wrote the distributed model, the data model for distributed systems, something like that. He discussed, you know, the concept of normalization, the power of normalization, why you would want this. And the reason why we wanted this, why he thought this was important, is actually kind of demonstrates how, boy, they used to write killer abstracts to papers, right? It's like the very first sentence, this is why I'm writing this paper. You read the first sentence, you know, future users of modern computer systems must have a way to be able to ask questions of the data without knowing how to write code. I mean, I don't know if those were the words, but that was basically what he said.

That was why he invented the normalized data model model because, you know, with the hierarchical management systems at the time, everyone had to know everything about the data in order to be able to get any answers, right? And he was like, no, I want to be able to just write a question and have the system answer that. Now, at the time, a lot of people felt like that's great. And they agreed with his normalized model. It was elegant, but they all believed that the CPU overhead at the time was way too high, right? To generate these views of data on the fly, no freaking way. Storage is expensive, but it ain't that expensive, right? Well, this little thing called Moore's law, right? Moore's law balanced his checkbook for like 40 years, 50 years. It balanced the relational

database checkbook, okay? So as the CPUs got faster and faster, crunching the data became less and less of a problem, okay? And so we crunched bigger and bigger data sets. We got very, very happy with this up until about 2014. 2014, a really interesting thing happened. If you look at the top 500, which is the supercomputers, the top 500 supercomputing clusters around the world, and you look at their performance increases year to year after 2014, it went off a cliff. No longer meeting Moore's Law ever since. They've been, and per core performance, you know, CPU, you know, instruction executed

per second, everything. It's just flattening. Those curves are flattening. Moore's Law is broken. Now, you'll get people to argue about it, but the reality is if it wasn't broken, the top 500 would still be cruising away. They're not, okay? So what this is telling us is that the relational database is losing its horsepower. Okay. Why is it happening?

Because, you know, gate length has an absolute minimum. It's called zero, right? We can't have a logic gate that's with negative distance, right? So, you know, these things, but storage, storage, hey, it just keeps on getting cheaper and cheaper, right? We're going the other way with storage, right? It's gigabytes, it's terabytes, it's petabytes. You know, with CPU, we're going smaller and smaller and smaller and the fab cost is increasing. There's just, it's going to take a next generation CPU technology

to get back on track with Moore's law. Well, here's the challenge. Everything you're saying makes perfect sense from where your perspective is. I reiterate, you are working with strategic accounts, which means big. When I'm building something out in the evenings because I want to see if something is possible, performance considerations and that sort of characteristic does not factor into it. When I'm at a very small scale, I care about cost to some extent, share whatever. far more expensive aspect of it in the ways that matter have been what is the expensive,

what the big expensive piece is engineering time. That's what we just talked about, right? As a developer, right? Why would I use MongoDB over DynamoDB? Because the developer experience is better. Exactly. Sure. Down the road, there are performance characteristics.

And yeah, at the time I had this super large scaled out complex workload. Yeah. But not most workloads will not get to that. Will not ever get there. Ever get there, right? So they'll run on little tiny instances. How is this going to work when I'm Facebook scale? No, exactly.

Facebook scale is irrelevant here. What I'm talking about is actually a cost ratchet that's going to lever on mid-sized workloads soon, right? Within the next four to five years, you're going to see mid-level workloads start to suffer from significant performance cost deficiencies compared to NoSQL workloads running on the same. Now, you see it right now, but you don't really experience it, like you said, until you get to scale, right? But in mid-size workloads, that's going to start showing up, right? This cost overhead cannot go away. Now, the other thing here that you got to understand is just because it's new technology doesn't make it harder to use.

Just because you don't know how to use something, right, doesn't mean that it's more difficult. And NoSQL databases are not more difficult than the relational database. I can express every single relationship in a NoSQL database that I express in a relational database. If you think about the modern OLTP applications, we've done the analysis ad nauseum. 70% of access patterns are for a single object, a single row of data from a single table. Another 20% are for a row of data, a range of rows from a single table. Okay, that leaves only 10% of your access patterns

involve any kind of complex table traversal or entity traversals, okay? And most of those are simple one-to-many hierarchies. So let's put those into perspective here. 99% of the access patterns in an OLTP application can be modeled without denormalization in a single table. Because single table doesn't require, just because I put all the objects in one place, doesn't mean that it's denormalized. Denormalized requires strong redundancies in the stored set.

Duplication of data. Edgar Codd himself said that the normalized data model does not depend on storage, that they are irrelevant. I could put all the objects in the same document. As long as there's no duplication of data, there's no denormalization. I know I can see your head going, wow, but it's true. Because as long as I can clearly express the relationships of the data without strong redundancies, it is a normalized data model. That's what most people don't understand. No SQL does not require denormalization. That's a decision you make.

And usually it happens when you have many-to-many relationships. Then we need to start duplicating the data. In many cases, at least my own experience, again, I am bad at computers. I find that the data model is not something that is sat out, that you sit down and consciously plan very often.

It's rather something that happens to you instead. I mean, realistically, using DynamoDB for this is aspirational. I just checked. And if I look at, so I started this, I started this newsletter back in March of 2017. I spun up this DynamoDB table that backs it. And I know it's the one that's in production because it has the word test in its name, because of course it does. And I'm that backs it. And I know it's the one that's in production because it has the word test in its name, because of course it does. And I'm looking into it and it has 8,700 items in it now.

And it's 3.7 megabytes. It's not for nothing. This could have been just as easily and probably less complex for my level of understanding at the time, a CSV file that I grab from a Lambda out of S3, do the thing to it, and then put it back. And from a performance and perspective side on my side, it would make no discernible difference. That's right. Because you're not making high velocity requests against the small object.

It's just a single request every now and then. S3 performance would probably be, you might even be less. It might even cost you less to use S3. Right. 30 to a hundred of the latest ones are the only things that are ever looked at in any given week. The rest of it is mostly dead stock that could be transitioned out elsewhere. But again, now that they have their lower cost infrequent access storage thing, great. It's not item level.

It's table level. Right. So what's the point? I can knock that $1.30 a month down to what? $1.10? Oh, well, yeah. No, I mean, again, Corey, for the small workloads, you know what? It's like, go with what you know, right?

But the reality is, look, as a developer, we should always want to know more. And we should always want to know new things. And we should always be aware of where the industry's headed. And honestly, I've heard through, I bet you I'm an old school relational guy. Okay. I cut my teeth on, oh God, I don't even know what version of MS SQL Server it was, but I was talking when I was interviewing at MongoDB, I was talking to Dan Pissett about the old enterprise manager where we did the schema designer and all this, and we were reminiscing

about back in the day. The reality of things is that if you don't get tuned into the new tooling, then you're going to get left behind sooner or later. And I know a lot of people who that has happened to over the years. There's a reason why I'm 56 years old and still relevant in tech. Okay. Mainframes, right? I kid. You're not that much older than I am. Let's be clear. I worked on them. Okay. And some of my peers, they never stopped. Still waiting for the AWS 400. We don't see them yet. I love it. I love that. But no, one of the things that you said, that I think it hit me really, and it's like, the data model isn't something you think about. The data model is something that just happens, right? And you know what? That is a problem.

Because this is exactly what developers today think. They think they know the relational database, but they don't. You talk to any DBA out there who's come in after the fact and cleaned up all the crappy SQL that people like me wrote, okay? I mean, honestly, I wrote some stuff in the day that I thought, this is perfect. There's no way there could be anything better than this, right? Nice derived table joins inside. And you know what? Then here comes the DBA when the server's running at 90% CPU and 100% memory utilization and page swapping like crazy. And you're saying, we got to start sharding the data set. And my director of

engineering at the time said, no, no, no. What we need is somebody to come in and clean up our SQL. I said, what do you mean? I wrote that SQL. He's like, like I said, we need someone to come in and clean up our SQL. I said, okay, fine. We brought the guy in. $1,500 an hour we paid this guy. I was like, there's no way that this guy is going to be worth that. A day and a half later, our servers are running at 50% CPU at 20% memory utilization. And we're thinking about, you know, canceling orders for additional hardware. This was back in the day before cloud. So, you know, developers think they know what they're doing. They don't know what they're doing when it comes to the database. And don't think just because it's a relational database and they can hack it easier that it's better.

Right. You know, there's no substitute for knowing what you're doing. That's what it comes down to. So, you know, if you're going to use a relational database and learn it. And honestly, it's a hell of a lot more complicated to learn a relational database and do it well than it is to learn how to model your data in NoSQL. So if you sit two developers down and you say you learned NoSQL and you learned relational, two months later, this guy is still going to be studying. This guy's going to be writing code for seven weeks.

Okay. So, you know, that's what it comes down to. You want to move fast, use NoSQL and you won't have any problems. I think that's a good place to leave it. If people want to learn more

about how you view these things, where's the best place to find you? You know, always hit me up on Twitter, right? I mean, at hulahanrick, that's my underbar Rick, that's my Twitter handle. And, you know, I apologize to folks who have hit me up on Twitter and gotten no response. My Twitter, as you probably have as well, my message request box is about 3000 deep. So, you know, every now and then I'll start going in there and I'll dig through and I'll reply to somebody who actually hit me up three months ago. You know, if I get that far down the queue, it is a last in first out. I try to keep things as current as possible.

My DMs are trash fire. Apologies as well. Of course, but links to it in the show notes. Absolutely. Thank you so much for your time. I really do appreciate it. It's always an education talking to you about this stuff. I really appreciate being on the show. Thanks a lot. Look forward to seeing where things go.

Likewise. All right. Rick Houlihan, Director of Developer Relations, Strategic Accounts at MongoDB. I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an upset comment talking about how we didn't go into the proper and purest expression of NoSQL non-relational data, DNS text records.

If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started. This has been a HumblePod production. Stay humble.