The Changelog: Software Development, Open Source - Massive scale and ultra-resilience (Interview)

Episode Date: July 9, 2021

This week we're sharing a recent episode from Founders Talk that we continuously hear about from listeners. Listen and subscribe to Founders Talk at founderstalk.fm and anywhere you listen to podcasts. On Founders Talk #75 — Adam talks with Spencer Kimball, CEO and Co-founder of Cockroach Labs — makers of CockroachDB an open source cloud-native distributed SQL database. Cockroach Labs recently raised $160 million dollars on a $2 billion dollar valuation. In this episode, Spencer shares his journey in open source, startups and entrepreneurship, and what they're doing to build CockroachCloud to meet the needs of applications that require massive scale and ultra-resilience.

This week, we're sharing a recent episode from Founders Talk that we continuously hear about from listeners. In this episode, Spencer Campbell shares his journey as an entrepreneur and software developer and the creation of CockroachDB. Spencer is CEO and one of three co-founders at Cockroach Labs. Listen and subscribe to Founders Talk at FoundersTalk.fm and anywhere you listen to podcasts. Here we go. In this episode, Spencer shares his journey in open source, startups, and entrepreneurship, and how they're building Cockroach Cloud to meet the needs of applications that need massive scale and ultra-resilience. Big thanks to our partners, Linode, Fastly, and LaunchDarkly. We trust Linode to keep it fast and simple. Check them out at linode.com slash changelog.

Our bandwidth is provided by Fastly. Learn more at fastly.com. And get your feature flags powered by LaunchDarkly. Get a demo at LaunchDarkly.com. This episode is brought to you by Sourcegraph. Sourcegraph is universal code search that lets you move fast, even in big code bases. Here's CTO and co-founder, Byung-Loo,

explaining the problems that Sourcegraph solves for software teams. Yeah, so at aLoo explaining the problems that Sourcegraph solves for software teams. Yeah, so at a high level, the problems that Sourcegraph solves, it's this problem of, for any given developer, there's kind of two types of code in the world, roughly speaking. There's the code that you wrote and understand, like the back of your hand. And then there's the code that some idiot out there wrote. Or, you know, alternatively, if you don't like the term idiot, it's the code that some inscrutable genius wrote and that you're trying to understand. And

oftentimes that inscrutable genius is like you from, you know, a year ago. And you're going back and trying to make heads or tails of what's going on. And really, Sourcegraph is about making that code that some idiot or inscrutable genius wrote feel more like the code that you wrote and understand kind of intuitively. It's all about helping you grok all the code that's out there, all the code that's in your organization, all the code that is relevant to you in open source, all the code that you need to understand in order to do your job, which is to build the feature, write the new code, fix the bug, etc. All right. Learn how Sourcegraph can help your team at info.sourcegraph.com slash changelog. Again, info.sourcegraph.com slash changelog.

Spencer, let's begin with building a company, I suppose, out of open source. What has been your experience with that? Obviously, you've gotten to a Series D, so it's been pretty much successful. But what's the challenges? What's the ups and downs of that kind of road? There's still a long ways to go. I think for us, building a database and trying to turn that into a company, an open source database, there wasn't really any other option. There's been some other examples of closed source

databases built in the last 10 years, and it's a pretty difficult uphill slog. There's some really good open source databases that have existed since the early aughts mysql postgres are some good examples and if you are going to play with those as alternatives and you're not open source it's very difficult to get developers attention i will say that the landscape for building an open source company has changed dramatically since we started Cockroach in 2015 and in 2020. I mean, five years doesn't seem like it should be a lot. It's a lot. These days, five years is more like 20 back when I started my career. Things are moving a lot faster. And the shift to the cloud, while it creates huge opportunities, is also changing what open source means to open source users,

open source developers, and in particular, the companies that might pay for an open source project, you know, as sold and supported by a commercial entity like Cockroach Labs. I think, you know, if we wanted to get into that, it's really about consumption model. I'm happy to talk more about that if it's interesting. That totally is. What do you mean by consumption model? Yeah, just think of this sort of generationally. I'm sure this has been true for at least partially for most of the listeners.

The older listeners will have a more visceral reaction to the way things were, let's say, pre 90s, you know, if you wanted to use software back in the 90s, the 80s, and, you know, also the aughts, and even today, if it were closed source, it was a pretty difficult procurement road. So, you know, you had to identify the piece of software that you're interested in, and then contact sales of, you know, whatever vendor was selling it. That would go through your procurement department. You had to get all kinds of different sign-offs and things. That's just to use the software. And then you got printed manuals that would come shipped to you.

There wasn't much of a community to ask questions of. I mean, you could contact support and so forth, but not, that's, all of these things were just much slower, more tedious, and, you know considerably slower let's call an order of magnitude potentially more in order to actually use software put it into production kick the tires whatever it is that you wanted to get done as a developer open source just dramatically improved that and i'd say that more so for example than than having the ideas free or even the price tag being free. You know, those are two aspects of free that people talk about with open source.

But it's the speed with which open source technologies could be downloaded, compiled locally, and then run and explored and even put into production. That was such an improvement and ultimately led to open source eating the world, as has been said by Andreessen Horowitz. That is the paradigm that existed, I'd say, in 2015 when cockroach was really conceived of as a product and then as a company. What's really interesting is that model is itself rapidly being overtaken by a new consumption model that's actually even easier than open source was compared to closed source and that's to use software as a service and you know

i did mention that you know when my description of open source you could download the source and compile and so forth obviously there were there were different evolutions within even that model where you would start to download binaries that were pre-compiled for a particular system and then even packages and things that sort of bundled things together i think that's the more common thing let's say for a like a docker container you know all of those are innovations but as a service is kind of a next generation fundamentally where you don't have any operations you don't have to learn how to become a system administrator or whatever DevOps requirements are necessary in order to understand and then run a system day one plus.

How to monitor it, how to understand, how to debug it. Those things now are still required to some degree, but you can obviate a lot of that labor. And especially when you're a larger entity trying to use software, this means that, you know, the investment necessary to use software has decreased accordingly. And ultimately, that feels like the writings on the wall for software to be consumed as a service increasingly. So, you know, the question is, how does open source fit into that? I believe in open source.

I've been doing it now for my entire career since I went to UC Berkeley. And Peter, my co-founder, is our CTO. Him and I started the GIMP back in 93. That world was magical to me when I first entered it. And I care deeply about open source, especially from the perspective of the free exchange of ideas.

But you can sort of squint right now and look at open source and the aughts and the tens, what do people call that decade? The tens, probably. Yeah, the tens. You can start to see it not vanishing, but changing almost unrecognizably.

If everything's consumed as a service, the interest in open source will necessarily wane. I don't think open source, just because it was free exchange of ideas, would have succeeded like it had if that's all it was. So when the consumption model of open source loses traction in favor of something that's even better from an average user's perspective,

what will the future hold for open source? And that's an interesting question. I would like to see it preserved. And so one of my big interests with Cockroach as we build Cockroach Cloud, which is Cockroach DB as a service, is how to preserve the best aspects of open source. There's a saying, ideas are crap,

execution's everything. And i suppose this consumption model as it relates to open source seems like that like open source is the idea the freely exchangeable idea forkable etc and the execution might be the service because that's what you say is like executions everything so if cockroach labs didn't create cockroach cloud you know then you've got service. Because that's what you say is like, execution is everything. So if Cockroach Labs didn't create Cockroach Cloud, you know, then you've got Amazon or XYZ cloud provider, essentially using your open source providing us a service and potentially getting paid and you not. That's the troubling model, I suppose, of the business you're in. And that's a part of the complexity you've been navigating this last at least several years,

maybe not so much in 2015, may have been starting to begin then, but it's really been prevalent since, say, 2017 to now, where that consumption model has drastically changed to where you create the open source and the community, obviously, and then the cloud provider doing the execution, the service, and potentially getting paid for it while you don't yeah it's a really interesting way to look at it and i think there's quite a bit of truth to it you know i would extend the definition of execution all the

way back into the continued investment in building the actual project you know if that doesn't progress then the service will start to look you you know, a little long in the tooth after a couple years and eventually not really be viable. So the execution, unless you want something to sort of die on the vine, has to extend to the investment in the open source project. And that's really what's wrong with Amazon's, people call it strip mining, but their exploitation of open core companies that are doing all the investment back into the core, the open source core of the project. And then Amazon swoops in and is able to use their platform advantage to really exploit a lot of that value that they're not reinvesting in. So I do think that Amazon's exploitive and predatory tactics around open core companies is just short term profit for Amazon and ultimately Amazon's customers. I don't really want to make a big value judgment about what Amazon's doing.

And yeah, it's true. If I use a word like predatory, there's an implied value judgment. But I don't fault Amazon for doing what they're doing. I think it makes perfect business sense. And it's in line with their mission and their value, which is to obsess about their customers. But nevertheless, it doesn't leave a lot of space for a company like Cockroach Labs if they were to use Cockroach Database

and simply repackage it and win the market because so many people use AWS. That's ultimately going to cause CockroachDB to cease being improved. Because if Cockroach Labs went out of business, or we had much less capital to work with, because of Amazon reselling our product successfully and sort of forcing us out of the market, you know, obviously, the improvements to CockroachDBDB would slow down to a trickle and maybe stop, and then what happens, right? I don't think anyone really benefits from that scenario. It mentioned a career in open source.

Take us back a little bit. If you want to start at 2015, that seems pretty shallow, but at least that's the beginning of Cockroach Labs and what you're doing with CockroachDB. Maybe take us back to, I suppose, your experience level with open source. You mentioned the GIMP. Did you mean the GIMP in terms of the editor of the GIMP? Yeah, that's right. So you're one of the co-founders of that or one of the co-creators of that?

Co-creators, that's probably the right term. Peter, Mattis, and I, I think in 1993, we had really become converts to Unix and free software and open source. And I had actually bought a used Sun Microsystems. I can't remember what the name of the actual model was, but it cost me a couple grand. It probably wasn't as good as even the high-end PCs that were on the market at that point in time. It was like 92, 93. And yet, it seemed revolutionary to me coming from Windows operating system.

Because it was counter-cultural. That era even, that time frame even was like, that's when Bill Gates was still CEO. He's a different guy now in many ways in terms of publicly because he seems likable and soft, whereas then he was ruthless. It was everything. I think he had an evolved outlook, or he has an evolved outlook.

I'm sure it continues. It's quite impressive to see that change. Back then, we were very impressed at so many aspects of the free software, open source world, and Unix in particular. And yet the desktop applications seemed to be a decade, I mean, they just were not on the same playing field

as what you could get in Mac and Windows at the time. And Photoshop was a really good example of that. And both Peter and I are really kind of graphics aficionados back then. Maybe still are to a certain extent. But we felt like, okay, we love so much about this new operating environment, but we can't get simple photo manipulation tasks done. And I remember one day, we were using XPaint and XV. Those are the two options, really, that were available to us.

We sat down at one point and just kind of wrote a manifesto. Hey, if we wrote something that could replace some of the things you use XP from, some of the things you use XPaint for, and make it look something like Photoshop in terms of its capabilities, that would really be the start of something. I wish we still had that manifesto because it was pretty peculiar in my recollection of it. I don't think the GIMP turned out anything like that manifesto. And we weren't really, you know, thinking that this would be. And sometimes the exclusion of our classwork and so forth. But what a learning experience to really dive headfirst into something that became so ambitious.

That's successful too. I mean, I knew many people still today even that use it. Are you involved in the project at all anymore? No, and that's again, part of the magic of open source. And part of what makes me so proud of it, I mean, I was kind of, in 97, Peter and I both stopped working on it. And it was sort of like we pushed it out of the nest

and it was either going to learn how to fly on its own or it's going to crash and burn and not have a future. And ultimately the open source community adopted it when there were a bunch of authors that had already been contributing to the game many of them continued even after peter and i uh left berkeley and went and started an industry uh but you know the game continues strong to this day i mean i download it every time i get a new computer and i'm extremely grateful it still exists because i don't really do enough photo manipulation work that I want to download or actually pay for Photoshop. So I'm really excited to use GIMP every time and see how it's improved.

Might be going a little bit a layer deeper, but you mentioned that you weren't planning it for it to be a GNU thing originally. Is that right? Did I hear you correctly? That's right. But yet its name is based upon GNU and the name. So, was the name come first or the software? When did they get the name? So, actually, it's a good question. The name came out right around when Peter and I saw Pulp Fiction.

So, you can guess the character it's named after. We just had more, I think my sense of humor is honestly pretty childish still, which is part of why Cockroach is called Cockroach. But we were thinking of names for it at that point in time. We'd already made some good progress with it. We still hadn't named it. But we thought, okay, well, we could call this like an imp, like X imp, we were thinking. An imp would be a little sort of familiar or something like that.

And that stood for Image Manipulation Program. And then because we were just seeing Pulp Fiction, Peter suggested, oh, wow, this is awesome. We should call it the GIMP. sort of familiar or something like that. And that stood for Image Manipulation Program. And then because we'd just seen Pulp Fiction, Peter suggested, oh, wow, this is awesome. We should call it The GIMP and we'll make it a GNU project. So that's really what sealed the fate that it would become part of the free software movement. And we were thinking, okay, it could be called GenRoll,

but then we ultimately called it GNU. That's a good movie. Gosh, such a good movie. Yeah, it really was. Okay, what's next then? So in terms of open source, what was your pathway from there? So UC Berkeley, you spent four years roughly on this based upon what you had just said there. Eventually you left the project because, hey, that's how open source works, and you moved on in your career. What was next for you? Well, interestingly, I wasn't super interested in just being a software developer

when I left Berkeley. I really wanted to potentially work on Wall Street or be a consultant and travel and see all kinds of different sort of businesses in situ. And I ended up taking a job at Accenture, which was called Anderson Consulting back in 1997. But I say there only four months because it wore pretty thin pretty quickly. You know, it wasn't the glamorous lifestyle I had imagined it would be. There was a lot of sitting around and working on kind of silly projects that weren't challenging

in the way that, for example, writing The Gimp had been. So I ended up going and working at a boutique investment bank for a year after that and that also wasn't quite to my liking it felt more like gambling than it did like deterministic software development but that was right in the middle of the dot-com boom so in 1999 i came back to silicon valley and started a company as a co-cto it was called we go systems no open source in there but it was content management system basically for hierarchical web presences it was pretty neat but it ran into the end of the dot com boom which became the dot com bust which was right pretty interesting experience to live through.

And that's where that project ended, or that company ended, and it was actually sold. And then Peter had actually done a similar move in terms of doing his own dot-com startup. He also ran into the dot-com bust and started at Google. And it was at Google in, I guess, 2002. Peter said, hey, you got to come work here. This place is amazing. And things are going great, which was a strange thing to hear in 2002

because 101, for example, if anyone on this podcast is from California, I imagine quite a few are, you know, it's this highway that runs north-south in California. And in the dot-com heyday, it was more like it is these days, pre-COVID. Just absolutely jam-packed with traffic at most of the reasonable hours of the day. And after the dot-com bust happened, it was like tumbleweed blowing across 101.

I mean, it was a really sad and sort of desolate stretch of highway for some of the busiest hours. That's what it felt like. Google, on the other hand, was just blowing up. It was a wonderful place to work with this exuberant culture and everything seemed to be going right. And so within three months, Peter started there, I started there, and Ben, the third co-founder for Cockroach Labs, started there. And we all started working together on just an incredible diversity of projects. So I'm not sure I've ever talked to anybody who's actually built a company into the bust of the dot-com era. So what kind of scars did you take away?

What kind of learning did you take away from, I suppose, that era of your life into maybe that still helps you make decisions today? Oh, I think one piece of advice I'd give any potential entrepreneur is start a company only with people, co-founders, that you have been in the trenches with. Preferably for considerably longer than a year, but I can say at least a year. And the trenches means, you know, there's been shells whistling overhead and not enough to eat for some of the time. It needs to be some good times, you know, but also a lot of bad times. And if you still maintain a lot of respect

for folks that have been with you in those situations, I think they can make really good co-founders. I've started three companies now. Cockroach Labs is the only company where it was just strictly co-founders that I had already been working with for, in this case, a decade plus. That has worked out very well.

You just really want co-founders to be people that you truly understand and respect. I suppose when you first start your career as an entrepreneur, you might have to get in the trenches with just anybody to some degree, which is where that advice comes from because you might eventually get your own battle scars and learn that lesson the hard way like you may have done. But you might be so eager to begin that you're like,

I will partner with anybody. I will go to a meetup to find my business partner, you know, which does happen successfully in some cases. No, it absolutely does. But I, I do agree with that.

Like in the trenches, I think is where life happens, you know, and life is not always fair. Life isn't always fun, but sometimes it is. Yeah. But being able to respect and appreciate the persons

and or person next to you that is leading your company is vital. Very vital. And that kind of leads to the other piece of advice I'd give to entrepreneurs. Exactly as you say, sometimes people just can't wait. And that's fine. I mean, I wouldn't say delay your startup idea if you've got one that's inspiring and you really believe in.

On the other hand, if you feel only a mediocre pull of gravity, let's say, for your startup idea, the recommendation I'd give to people is work at a company that looks like it's really going places. I think the sweet spot would be a startup that has this pre IPO that, you know, it's between 100 and 500 people, it really looks like it's starting to win its category. That is a just a it's a prime and fertile experience where you are going to meet people in the trenches that you will want to start that, you know, next company with. And there's a lot of ways to learn in sort of a negative sense of what doesn't work. I mean, you could spend an entire career doing that. And that experience is valuable. That's sort of battle scars. And, you know, hey, I've seen this done

before and it didn't work out so well. Maybe we should think of an alternative. But I think the sort of positive learning experience where you go somewhere and you see a company that has a great culture that seems to really be succeeding, those situations attract the very best and brightest. And so you end up with a reputational, let's call it experience that really gives you a reputation that can help you in terms of getting investors and attracting people to work at your new venture.

But you also, as I said, you end up being thrown into the trenches with really good soldiers to keep that metaphor going. And those end up being the lifelong friends and collaborators. You're also networking too. I mean, when you're in a company like that, you're obviously gonna be around people who have ambition,

have a desire for success. They're able to get hired by a company like that, stay employed, maybe even ship good stuff and deliver on what their promises might be. And people undervalue, I suppose, early on how to get to a network, how to build a network. I think you just start. You just make friends with one person, do your best to keep connected, and rinse and repeat. Yeah, absolutely. I can't tell you how impressive some of the outcomes of the folks that I worked at Google with back in 2002. It's just the diaspora of that cohort of Google employees

is something to behold. And so it's, yeah, it's exactly your point. I mean, there's exceptional people and that's really how you do the real networking right it's not i'm not saying you can't do it on linkedin uh it's a great tool but really working on solving uh interesting and difficult problems with the best and brightest that's that's how you you do the networking and you know the only way to start is just to to put a foot on the path and start walking. This episode is brought to you by Retool. Retool is a low-code platform built specifically for developers that makes it fast and easy to build internal tools.

Instead of building internal tools from scratch, the world's best teams, from startups to Fortune 500s, are using Retool to build their internal apps. Assemble your app in 30 seconds by dragging and dropping from the complete set of powerful pre-built components. From there, you write custom code, connect any data source, API, and build custom logic and queries to create exactly the right tools for your business. Spend your time getting UI in front of your stakeholders, not hunting down the best React table library.

Retool is also highly hackable, so you're never limited by what's available out of the box. If you can write it in JavaScript and an API, you can build it in Retool. Try Retool out for yourself at retool.com slash changelog. Again, retool.com slash changelog. Again, retool.com slash changelog. experience at Google, obviously. I understand you were at Square for a bit. You had a startup called Viewfinder, which you have since sold. You know, you got a lot of, you know, sort of in the trenches, bloody knuckles, and even time in the trenches with, you know, Peter and Ben, your two co-founders, to kind of get to a problem set, which is usually the crux of like why you're doing today what you're doing today. So how did you there what is that yeah so databases turns out that they have been extraordinarily central in my career back as early as the dot-com startup i did we go systems we built sharded oracle and sharded postgres those are

the two sort of flavors we supported and i gotta tell, when I was at Berkeley, I wasn't very interested in databases. I mentioned graphics. That was really probably my key interest. Databases, I didn't take until my first and only year of grad school, and I just kind of took it to get some credits. I ended up being pretty interested in the course, but I didn't really think they'd be central to my career. But as soon as I hit the quote-unquote real world, databases became a central problem, a big source of frustration at WeGo. And then when I got to Google, that was one of the first projects I got thrown onto, which was the AdWords system, which was nascent then in 2002, but it was running into problems with sharded MySQL.

And you hear this word sharded, but it really, you know, for listeners that aren't aware of what that implies, it's about taking a monolithic database like Postgres or MySQL or Oracle that really is meant for, you know, a single machine, even if that machine can be quite large. And you say, well, maybe this isn't going to be large enough. And this is the case of AdWords when I got put on that project. So you'd say, okay, we're going to use two databases. We'll put half our customers on the first database, half on the second. And maybe at some point you start reaching capacity on those two.

And so then you say, we're going to use four, or we're going to use five. It got up to about 32, I think, when I was at that project at Google. And all these different problems started to occur as you sharted. You know, the application complexity became quite high. You know, it's just one ridiculous practical example. The MySQL databases had too many connections coming into them. And, you know, that started to cause them to cavitate. And so we solved these problems.

Every morning, we had this ads war room to solve the latest set of problems related to this just basically scalability challenge with the database. And I will just say that in Google AdWords, by the time they replaced that sharded MySQL architecture, they'd gotten to a thousand shards. So it became a thousand MySQL instances.

And I've heard that Facebook has hundreds of thousands of MySQL instances. So there's kind of no end to, you know, both how scalable that architecture is, but also how much time you have to put in to truly keep scaling it. So that's a scalability challenge. There's also resilience challenges. And that's part of what we saw when we were at Square is certainly something we saw when we were at Google. And that is you really don't want to have a database that has a primary and a secondary. And that's been the standard way to operate databases for, you know, most of my lifetime. The problem with that solution is that the secondary is getting an asynchronous replication stream of data. And even if you put in another data center, so you have a really nice failure scenario, so you can lose the data center and failover, that failover might

imply data loss, because that asynchronous replication stream might not have fully made it over to the secondary when the primary dies. so you switch over to the secondary and you realize wait a second i thought i just sent that email out as an example but it's not in my outbox what happened well the replication stream just didn't get that email into the outbox on the secondary so it's almost like you've moved backwards in time you've regressed to an earlier version of the state that you had in an application. And that causes huge headaches. I mean, if a data center was lost at Google back in 2004, let's say, it would be many teams scrambling to figure out what might have gone wrong. Did we charge a customer twice? Are there consistency problems in the data? Because some of the stuff got replicated and some of the stuff didn't.

And you'd have to write cleaners and scripts that would go through things. And you just try to reason through what might have gone wrong with your use case. That's not the right way to do database replication, and certainly not in 2020. Google started to play around with better ways to do that as early as 2004, 2006. They built Bigtable, then they built something called Megastore,

and then they built something called Spanner. And Spanner is really what inspired Cockroach. And so there's scalability, there's resilience. Those are two of the biggest problems that I faced with databases in my career. The sort of gold standard these days with databases is to do what's called consistent synchronous-based replication. The popular ways to do this is something called Paxos or something called Raft.

And what they do is consensus. So instead of just writing to a primary and asynchronously replicating to a secondary, you actually write to three data centers or three replication sites. And you are going to be committed if the majority of the replication sites respond positively or affirmatively to any particular right. If, for example, you only write to one out of three data centers, that right can't be committed. So you need two out of the three. And so as long as you always have two out of three, if you lose any one data center out of those three, you always are guaranteed that one of the remaining two has the exact data that you need. So as long as you only lose the minority,

you have total operational sort of continuity. It's hard to overestimate just how important that advances for running these systems operationally. And you were in an era when this didn't exist. You had to invent it. You'd mentioned Spanner was inspirational to some degree. And even as you talk through the problem, it reminds me a little bit like RAID for hard drives, for example.

You know, you might choose RAID 0 because you want super fast disks. You may choose RAID 10 or RAID 5 or a couple different other flavors essentially it's how many disks you can have lost at once and it's similar it's like how many databases which is literally what the disk is it's a database of your files right it seems a lot like even that at a small level why did it take so long do you think to hit the problems of sharding with mysql or Postgres, or other to get to that point?

Was it technically not possible until around that time or was it just like no one thought about doing it? That's a good question. I'm just going to kind of think out loud on answering it. Certainly, your analogy to RAID disks is very accurate. That's exactly what it's like. I mean, not exactly, but it's pretty similar. Principle, yeah. Yeah.

The reason that, well, let me just say this. There's nothing new under the sun in computer science. Or maybe the number of new things are vanishingly small. Like everything's been thought of before. And so making sharding uh more automatic this has existed far earlier than google created big table and sort of launched the idea of no sequel i mean no sequel the word no sequel or the term predates google or at least big table by i think five or six years at least the earliest mention of it that I've been able to find. So ultimately, the popularization, as opposed to the innovation of these kinds of things,

whether it's consensus-based replication or sort of elastic scalability in a kind of cloud-native fashion, I think the popularization of these things and widespread adoption has to have a lot of different confluent factors all aligning. The cloud is a big example of why these things are possible. And Google had their own version of what looks like the public cloud to any startup today in 2020. They had that in the aughts. They had data centers all over the world and Borg to control access to resources in a very frictionless fashion. And once you start to have capabilities like that, you start to think that, hey, we could write databases differently. We could use all these commodity resources and build a bigger database than anyone's ever had. was instrumental to driving some of this innovation was the fact that after the dot-com boom, the idea of enterprise scale gave way to a whole new level of scale, which you could call

web scale. That's what people have called it. And I think there's additional levels of scale that are on the horizon or are probably already here. You know, I think of when we think of why you need something like Cockroach, which is an operational, what they call OLTP, online transaction processing database, it was used by an enterprise, and you had maybe 10 million customers, the biggest size enterprise, you know, Google started to say, okay, we might have a billion customers, and we need to store all that data. That's just a hugely different problem. And it demanded additional architectural sort of innovation for the database. But now what we're looking at is something that goes beyond the number of human beings that have smartphones, you start talking about IoT, you start talking about virtual agents, basically anything that

could hit a company's API, which interacts with a service that they're hosting that has a database that's backing it. You know, it used to be how many human beings had desktop computers, then it was how many human beings can operate smartphones. Now it's how many potentially non-humans can take some agency and access an API causing a database to be involved. And that number is already in the hundreds of billions and it's going to go to the trillions. So it's just the demands of scale

are probably pretty limitless when you actually look to the horizon. But all of these trends, the alignment of them is what pushes what might have been a research paper in the 90s, which is the case of Paxos, into production systems. It's just the demand has to be there. And so the stars align. It's really interesting to watch it. Yeah. Basically, you don't create software you don't need.

You create software you don't need. You create software you need today. So software that's in place, successful, adopted, useful, etc., is because it has a use. And so as the need changed,

you know, the idea of multiple data centers, etc., the need for how a database needed to work changed beyond what previously had been in place.

And you needed a new look a new database based on new infrastructure new problem sets yeah you don't build what you you can't use that's uh that's exactly accurate i mean and if you do you're probably wasting your time and you don't use what you shouldn't use so sometimes you're not good when you use google tools but i'm not google so i shouldn't use google tools i should use the database that makes sense for me and my problem. That's right. Which is a whole different subject. You know, so you're at a point now, obviously, where you're in the trenches with the right people. You're building the right technology, potentially being inspired. Did Cockroach, the software product, the initial of it, did it begin when you were at Google? Did it begin when you were outside of Google? How did the, you know, the beginning of it happen? When did you first try it, ship it,

see it be used by somebody else? Take me to that timeframe. Yeah, it was when we left Google. So that was 2012. We'd been there just under 10 years. Great time. But ultimately it felt like, you know, it was time to do something new. I even thought about going back to school, maybe I'd get an MBA and kind of take a, an MBA is really a two-year vacation where you network. But that sounded pretty good to me. And I thought maybe I'd go back and become a doctor. I just felt like, you know, I didn't necessarily want to spend my whole life being a Google engineer. It didn't matter how much fun or how challenging the work was. For me, that was just kind of part of my internal calculus. In the end, we decided, you know, hey, we could do another

startup. And what viewfinder was, was a private photo sharing so uh the same time that snapchat was getting started we were getting started and uh i think we did build the right thing snapchat clearly did so but it was it was really an amazing experience of course overall but when we left google we were a bit disappointed by what open source databases and open source infrastructure looked like in 2012 compared to what Google had been aggressively building. And that's where the idea of Cockroach was initially born. You know, it was really, okay, well, Spanner's great. We want to have Spanner-like capabilities,

but it has to be open source. And it has to be something that you can run on a laptop and it has to be something that any startup could use. And the idea of calling it Cockroach is really because cockroaches are so damn resilient. And they say after World War III, they'd be the only things left alive. That's probably true, actually,

based on my experience living in New York. That's right. the movie Wally that cockroach would not die it would it would last through everything it could be squashed and it would bounce right back yeah so you know I think it was during that early days of viewfinder that again it was another manifesto I like writing those it's kind of like okay well this what's exists right now it doesn't work well enough. It's fun to write without thinking about the practicality of any particular prescriptive solution. But what would be the ideal solution to this problem if there were just no barriers or limits?

Obviously, it's still grounded in what's conceptually possible based on what i knew and the beauty of having come from google so recently is that the the blueprint at least of the capabilities was was very well understood i mean they just published that paper too and that manifesto was super fun to write. But, you know, it was just this idea that, okay, there'd be these nodes and it's commodity hardware. And, you know, I was thinking of AWS EC2 at the time. And every node of Cockroach would essentially colonize the disk space you gave it. And it would try to reach equilibrium, but also be greedy about making sure its data

was replicated to any neighboring nodes that it would coordinate with. There wouldn't be any actual leader or central points of failure. Everything would be cooperative with sort of well-understood protocols, but capable of independent operation where necessary. And that was a super fun thing to ideate. But ultimately, we were trying to build private photo sharing, not a database. So that project really kind of was a passion project that had to be put on the back burner. We were then acquired by Square a couple years later in 2014. And when we got to Square, they sort of didn't really have a fixed project for us to work on. So we went around, talked to a lot of different groups, and a theme emerged. And it was a theme, as I've already mentioned,

speaking with you, that has been prevalent in my career, which is databases are a significant problem. And at Square, I think a lot of the problem was, how do you make sure that applications that are database-backed can survive a data center outage? Not just survive it in a kind of half working fashion, but to really have business continuity, no postmortems for application teams. And payment processing was the seminal example at Square. If you started

authorizing a credit card and then finally charged it or canceled the transaction. That's sort of a two-step process. And if it gets interrupted midstream, so you authorize the credit charge and then you aren't able to cancel it or confirm it, you might end up authorizing it twice when the thing restarts and you've failed over to a different data center. And that was problematic from a sort of customer perspective. twice when the thing restarts and you've failed over to a different data data center and that uh that was problematic from a sort of customer perspective you don't want to get that uh alert on your phone that you've been charged twice and then you know that causes problems for square and

so forth but you just all those kinds of problems if if you don't have a good solution to the real guts of the problem, the core, I laid out a fairly simple scenario, but the problem is these use cases, they get more and more complex, and the burden of maintaining it when there's gaps at scale becomes very onerous. So that was a big learning at Google. Basically, anything that can go wrong, any gap you have where you're like, well, it's pretty unlikely this is going to happen trust me at scale it will happen and it will happen and become a

huge problem that will blow up on you so it's it's kind of like theoretically when you build these kinds of systems you do not want to have any gaps like zero everything needs to theoretically work perfectly uh you know even even with disastrous sort of scenarios that you don't think are going to happen, like weird network partitions that are going to be, you know, so obscure that you just can't imagine they'll happen. Boy, they'll happen,

and they'll happen in like a month or two at scale. So that's really when we were at Square, just to sort of pick up that thread again, we came to the conclusion that Cockroach, as we had originally conceived it, you know, might, its time might have come. And, you know, I lobbied pretty hard for Square to support the Cockroach project. And, you know, there were definitely some people that were on board with it and others that weren't. And ultimately, Square said that I could work on it, but they weren't going to really adopt the project. So we started as a GitHub side project,

and I worked on it my nights and weekends. And eventually, I was able to work on it full time while I was at Square, which was really an amazing time in my life. For about six months, every day I'd come into the office and I'd say, oh, great, what's the next problem? How do I build the very best database I can conceive of? And there weren't any customers or managers or any process. No one's still in your time, yeah. Yeah, it was wonderful. Focus, right?

I mean, reminds me of John Wick. He's a sheer will, a man of focus. And it's like, what could you do with complete focus, right? Well, you know,

I think as any really dedicated programmer knows, those stretches of that sheer focus are some of the most pleasurable moments in the trade and the craft. And I think that's true of any artist. It's just when you get into that flow state, it's meditative in terms of its quality. And it's like a deep state of happiness. This episode is brought to you by Century.

Build better software faster. Diagnose, fix, and optimize the performance of your code. Over 1 million developers and 68,000 organizations already use Sentry, including us. Sentry also recently shipped a new SDK for Next.js applications. Check that show notes for links to more details. Best of all, our listeners get the team plan for free for three months. Head to Sentry.io and use the code TheChangelog when you sign up. Again, Sentry.io and use the code TheChangeLog.

You know, you're at a point where obviously you're really enjoying it. You mentioned this six months of working straight on it. I'm assuming at some point you're going to depart from Square and rethink your life and, you know, get influence to take investment and create a company. Is that roughly what happens next? Yeah. Well, you know, the interesting thing about Cockroach is, you know, to our earlier conversation, it was a technology whose time had come.

You know, people, I think their appetite was whetted by Google's paper about Spanner. Well, we need this. Yeah, interested, like, who's going to build the open source Spanner? Kind of like, you know, Hadoop with the open source MapReduce and, you know, there's other examples. And, you know, that was true, I think, more generally,

not just the VC community, but developers everywhere that were interested in databases. We had a lot of stars on GitHub, and that ultimately led to a number of VCs coming around and wondering whether we were interested in taking money and really making this a commercial entity. And I remember the idea was a little foreign at first, just coming out of a startup and actually enjoying my time at Square.

But I realized I really want to build another open source system. I think that was one of the most rewarding things that I'd done so far writing the GIMP and I felt like Cockroach could really be extremely useful and something that existed long after I started I stopped working on it maybe even after I was no longer alive you know it just uh it felt like you know it could be a system that really uh meant a lot and added a lot of value. And so I convinced Peter and Ben, which wasn't, Ben was totally on board with it. Peter was thinking that he might want to go back to Google, work on Go or something like that. And I said, come on, Peter, I know our last startup, you know, wasn't the huge success we hoped it would be, but let's jump on the bandwagon again

and, you know, build what, you know, something that we're probably a little bit better at, distributed infrastructure, as opposed to a private photo sharing company where you have to understand the fickle desires of the average consumer, which is maybe something I'm not so good at. Well, you're at a series D, which means you've gotten several rounds of funding so far,

which means people believe in you to build what you're trying to build. I know tons of people that use Cockroach and love it. So congrats on that. I know that Cockroach Cloud is now a thing and you're doing well with that. In terms of, I guess, success of a business right now, how do you feel you're performing as a business? Really well. There's always just existential concerns starting any company. There's been so many stages of growth. The early days

when we were pre-general availability, we had alpha and then beta. Those we could move so quickly and it was extremely enjoyable. It was just R&D. Building a relational database from scratch from the top to the bottom is a huge undertaking. And those are, I think, some of the most enjoyable just because of the extent of the challenge. But then the team started to grow. So, okay, you've got cultural issues and you have to manage so that everyone is pulling in the same direction instead of everyone doing something useful but you know pulling in opposite directions and then you get

customers and you know okay you got to respond to all of their issues and make them successful and you know and then it's kind of like you've seen the crossing the chasm idea where there's this bell curve of adopters and you have those innovators. And that's kind of where Cockroach and most of these kinds of technologies start. And then you get to the early adopters and then the early majority. Where are we in that journey? And it's just every new tranche of customers or people that are interested is a whole new challenge. So I feel like when I look at everything we've done,

it feels like we've come a long way. But when I look at everything that we need to do, at least what I can envision, it feels like we have a heck of a long way to go. So, you know, I think it's anything but certain that we've truly succeeded as a commercial entity. But it, you know, we've come a long way. You know, we have some of the biggest companies in the world now using Cockroach, and that includes the real blue chips, but it also includes the really fast-moving, high-tech growth companies.

And so both of those are extremely exciting. I think when I started with Cockroach, I was maybe a little intimidated or unsure of whether building something that would be enterprise software is really what I might be good at. But I found that helping these bigger companies adopt cloud native architectures and infrastructure is extremely rewarding. And that's something I'm happy about. But as we were talking about at the beginning of the conversation, the real challenge is how do you build and deliver cockroaches as a service?

And that's where I think the future of our success is going to be made or lost. And it's a transition. Right now, the world's biggest companies, they want to run a relational database themselves. They want to self-host it. They want to buy software licenses. They might want to put it in private data centers or hybrid across private and public clouds. On the other hand, in five years,

even those companies, much less every other startup and high-growth tech company, they're all going to be using databases as a service. In 10 years, the entire world will be. So we have to not just win where we originally set out to build CockroachDB as the way that you might run Oracle or Postgres or MySQL if you're running it yourself, but we have to also now succeed with Amazon

as a direct competitor, and Google and Microsoft, like these big clouds that are offering databases as a service and doing quite well with those businesses. So how do we deliver cockroaches a database as a service and effectively compete? There's a lot of really interesting answers to that question. It's by no means a foregone conclusion that a company like AWS, which is the cloud vendor incumbent, really has as many advantages as you might think they have. How do you do that then? Because on the landing page for Cockroach Cloud, you say Cockroach Cloud is the simplest way to deploy CockroachDB and is available instantly. And here's the key on AWS and Google Cloud. So what's your current answer? I'm sure over time your answer will evolve,

but what's your current solution to competing with these big players? There's a number of different aspects to the successful strategy. And as you say, ours will continue to evolve. One is you out-innovate. I think Google is probably the only of the cloud vendors that has a truly comparable technology.

Amazon's better at repackaging existing open source. And part of that out-innovating is, you may have read, we made some license changes to the core of Cockroach. So we adopted something called the BSL. And that's part of how you continue to out-innovate. It gives you a little bit of protection. Then there's the idea of being multi-cloud or cloud agnostic, and that includes private clouds. The deployment flexibility is extremely important to the world's big companies that have been around

for a couple decades and have lots of existing investments in data centers and high-value use cases that aren't going to be easily moved to the public cloud. That, I think is incredibly important. You know, part of something that's worth touching on further is just how much innovation can be done in the database as a service model. And that's something that we're pushing really hard on right now. Ultimately, we'd like to deliver databases with a lot less friction than they currently are delivered as a service.

Right now, when you get a database as a service, there's quite a bit of cost to it. I mean, even like a sort of production-ready encrypted instance of RDS, that's sort of the minimal footprint, still costs you about $100 a month, which is a lot. And you're choosing the size of nodes, where those nodes are located, and there's a number of decisions that increase the size of nodes, where those nodes are located. And there's a number of decisions that increase the friction of the process. We'd like to drive to a world where

databases are truly serverless, in the sense that when you get a relational database, it's something that you can pay for exactly what you use, not worry about what kind of machines, how many and so forth, or even where they're located, you just get a database database and that database is truly capable of global operation. Hey, if you only use it in the East Coast of the United States, great. You want to add the EU? That's extremely simple. It's as simple essentially as setting a different value for a column in a table specifying what region the data should be stored in or whether it should be global, as an example. And further, we actually think that price is a major impediment to using something like a relational database as a service.

We'd like to make these things perpetually free for developers for a pretty generous tier. So think about what Gmail did in 2003 where they're effectively making a gigabyte of email free. you know at the time you hadn't it was unheard of yahoo yeah it was like five megabytes what you got before which it's filled up with one mp3 somebody sent you or whatever a couple photos so this is a huge innovation obviously um just set a new standard for what webmail should feel like and while gmail is free

if you want 100 gigabytes you pay for it that that extra storage space and that's exactly how cockroach cloud is going to feel to a developer like we want to make perpetually free relational databases that are the seed of an extraordinarily powerful production database something that can scale to run you know retail banking for the world's largest banks that has geo replication for a high level of resilience. And that is capable of truly global operations so that even a startup could use the free tier of cockroach cloud and, you know, store data for customers in Brazil in Brazil, store is for customers in Japan in Japan, right? And give them a local experience. That's how big tech builds services and applications.

We really want to make that so that every company in the world, even every developer, even in a hackathon, can build that way. And it's ideally easier to build that way than it is to stand something up yourself in a single availability zone. That's ambitious, for sure, because one of the hardest parts is adoption and you're guaranteeing that by enabling that perpetually free tier that's generous so that you can tinker in a hackathon or scale your enterprise and it's the same cloud, right? It's the same cloud. It's not like a different version of it. It's the same version

regardless. Yeah, we want that to be a very continuous product experience. And I think the journey that is most evocative for me is, you know, you're starting a company, which, you know, I've done viewfinders, the canonical example I always use in my head, how much easier could we have made the viewfinder experience? And that's just great, great right to have that experience to to make product decisions is pretty fundamental but you know the idea would be okay you know hey you want to stand up your database your pre-production but you know you have developers that are pinging it and so forth you certainly don't have to pay for that you don't have to have this big server that's sitting there that's almost completely idle for months and And then you launch the first version of your software,

you get something into the app store, maybe it's in test flight or something, and you have 100 beta users that are poking at it and so forth. You're still under the free tier for sure, right? It's only when you really scale to get more product market fit and you start having sustained high throughput,

then you start to get into overages and you can pay for exactly what you're using over that free tier threshold. And then eventually, if your startup continues to succeed, you're going to want to move to sole tenancy, a dedicated cluster, as opposed to the cockroach cloud free tier and the overages where you're sharing a multi-tenancy cluster with other users. So for InfoSec reasons, so that you don't have noisy neighbors and you have a very guaranteed throughput, exactly what you expect, and there's no sort of variance in terms of your latencies and response times and so forth.

And also, in order to truly scale to big sizes, where the cost is more economical, that's where you'd move to the dedicated cluster. And there you can scale to really as far as your ambitions. It's very similar to the VPS analogy, where you might be on a virtual private server, you maybe have some noisy neighbors neighbors to use your example. But if you can go beyond that, maybe you get your own dedicated virtual private server where you're not sharing, you're not shared resources, you have your own dedicated.

That seems like a very similar analogy. So if you get that from that world, then you'll get that in the database world. That's exactly accurate as an analogy. And what's really wonderful about this capability that we're building, think of it as virtualizing a big cockroach cluster and allowing many tenants to share those resources. That's also something that's extremely interesting

to big enterprise customers. They would like to have their production use cases also run on a multi-tenancy dedicated cluster. So one of these big clusters that we might have a public, any developer can sign up with their GitHub OAuth login, but you might deliver that to a major financial services institution as a dedicated cluster,

but their internal teams get to share those resources in a pool. So they don't have to say, okay, for each one of these production use cases we have, we're going to have completely dedicated hardware, which we have to make sure is sized so it can handle our peak throughput. That's a lot of wasted resources over 100 production use cases. If you can pool all those resources and allow the overages from one to use additional resources from others that might not be at peak throughput,

then you get to have much more efficient resource sharing. So what we're building for the public at large to really connect CockroachDB to the large audience of developers out there in the world is also something that is extremely valuable to the high-end dedicated companies and customers. It's interesting how the ideas translate from small to big to big to small.

That's interesting. Let's close with this. I didn't preface this with you, so this is sort of a curveball to some degree, but what's lesser known or not known at all to, say, the general developer world of what you're doing? but what's lesser known or not known at all to say the general developer world of what you're doing? So what's on the horizon for you that not many people know about that you can share here today? Well, a lot of what I've been talking about, I'd say,

is understood by a still small audience. That's something to always keep in mind is that crossing the chasm thing. I think that the large pool of developers out there, and there's 10 million of them in the world, the majority of those probably have never even heard of Cockroach. That's also interesting. I imagine the people that listen to your podcast are closer onto that innovator side of the sort of bell curve. Yeah, the thing that I think might be extremely interesting that isn't necessarily obvious from what I've already talked about is just what we think the 2020s

holds in store for even a developer at a startup or a developer at one of the Fortune 500 companies or Fortune 10 companies even and that's really not just a database that's serverless but the entire stack above that database right if you really want to build an application the way that facebook or uber and netflix builds them so that wherever you do get customers around the world you can give them what feels like a local experience it's more than just a database. I mean, the database is clearly a foundational layer in the stack, but you need to have execution layer as well above it.

You're certainly going to need additional systems that are also global. You're going to need global DNS and global load balancing and so forth. Really, what's on the horizon for us is how do we partner with the clouds, with other technology companies that are complementary to what Cockroach Labs is doing in order to define the next generation of stack. You remember the LAMP stack which really drove a lot of the innovation in the aughts and beyond. You know, the big question for us, and I think what's extremely exciting,

is the emergence of a stack that allows a startup or a Fortune 500 company to build the way that Google builds and operates services and applications. And so I think that's where a lot of our thinking, and I'm sure a lot of the thinking of all of our contemporary peer companies, is going to be directed in the next five years. And part of that, I think, is 5G, interestingly enough. It's pretty unusual that there's a significant improvement in latency in communication networks. It's much more common that you have significant improvements in bandwidth.

Latency improvements happen somewhat infrequently. And they usually herald quite a bit of innovation. And so I think the widespread adoption of 5G in the next five years is going to mean that applications, especially on a smartphone, can feel substantially different than they do today. I think everyone's pretty used to hitting a button on a smartphone and maybe a second and a half later,

something changes. That is a pretty bad user experience, but it's just one we're all used to. Ultimately, you want that to be the 100 millisecond rule as popularized by Google Gmail and now more recently Superhuman. It's another email application. But this idea is 100 milliseconds is the threshold for human noticing something as taking time or being instantaneous. Less than 100 milliseconds is instantaneous. So if you can actually adhere to that latency

end to end in other words you hit a button on your smartphone and you get response all the way up to the backbone into the you know across the backbone to wherever the data center is through the application logic into the back-end database and then all the way back out that round trip time less than 100 milliseconds you can give people real-time experiences and obviously for gaming interactive media of all sorts self-driving ar vr these are obvious use cases where this kind of latency guarantee is transformational maybe even necessary yeah but i actually think that as this becomes both more desirable and that will happen by degrees at first and then all at once, but also more tractable, like it's not just Google being able to build these things or Facebook,

but a startup, even a hackathon, that's like the gold standard litmus test in my opinion, then you're going to see lots of innovation. And even things like what happens when you post on Twitter or what happens in your Facebook feed or your news feed. As little things start to happen and you start to see more than just a couple dots going across the screen when somebody else is typing, but you actually start to see genuine interactions. And that's going to make the virtual world that so many of us are spending so much time in feel substantially different. And applications that don't start to feel that way will increasingly feel antique and sort of out of touch and clunky, right? So that's going to just, I think, to our sort of point before, you know, why do these technologies find such widespread adoption? And all these stars have to align.

And when there's a huge demand that catalyzes across the ecosystem, that will be what everyone's building for in 2025. And that's really where we're interested in setting our sights. That's an interesting perspective. Just for humor me, is 100 milliseconds basically one-tenth of a second? That's right. Is that what it is? So I had to grok that in my own mind. I'm thinking like listeners, just so you know, 100 milliseconds is one-tenth of a second. So you're talking about as a full, quite a bit of an operation to go through the client device all

the way to the stack and back again in one tenth of a second. Yeah, and what's interesting is you simply can't do that for a user that's in Sydney, Australia, if your data center is in Virginia. It's too far. It's in fact going to be half a second. And you think, well, what difference does a half a second make? That's kind of ridiculous. Well, Google's found that their search results,

if they, you know, take 200 milliseconds instead of 100 milliseconds, just like a, or 300 instead of 200, there's this incredibly consistent, I guess, relationship that they observe between how many searches people do and how much of a latency they experience i mean even down to these fractions of a second and it's it's it's like uh the curve is is reproducible and it's it's uh you know especially over the amount of data they collect it's extremely consistent and that is you know a little bit mind blowing but how do you solve that problem i mean that means the only way because the speed of light really and speed of networks

aren't going to allow you to get that Australian user a local experience, you have to expand what your data architecture looks like what your whole stack looks like so that you're really running a global architecture, so that there's application logic in Australia running on servers in Australia, and there's databases that are running on servers in Australia. That's the only way to really do it. And that's also great because a lot of countries are introducing data sovereignty regulations. And they don't want users data, especially if it's, you know, personally identifiable to exit legal jurisdictions. And users don't want that either. So, you know, how do you grapple with all this stuff? And the answer is, okay, well, if you're

Google, you just build it all. If you're anyone else, you simply don't. You try to get everything to sort of work out of a single availability zone. And in order to solve this problem for a much more general audience, it's about improving the infrastructure. And so that's what we're doing. At least we're pushing a lot of those capabilities

and smarts down into the database. Very cool. Spencer, thank you so much for spending this time with us and sharing your story and Cockroach's story and this look into the future of what networks might be like and how you're planning for them to be, I suppose, reliable. Not so much the network,

but the data that might transpire there and the partnerships you might form as a result of this new found lack of latency in our future communication network. So thank you so much for sharing your time today and appreciate you coming on the show. Yeah, it's been my pleasure. Thank you, Adam. What's up, Adam here. Thank you so much for tuning into Founders Talk. If you enjoy this show, do me a favor. Share it on Twitter. Share it on Insta. Share it with a friend. Tell someone you love this show if you got value from it. As you know, we're backed by some awesome partners,

Linode, Fastly, and LaunchDarkly. Check them out. We get tremendous value from their services, and you might as well. Also, thanks to Breakmaster Cylinder for making all of our awesome beats. Breakmaster is our beats master in residence. Thanks again for tuning in. That's it for this episode. We'll see you next time. Thank you. Game on.