The Changelog: Software Development, Open Source - The Future of RethinkDB (Interview)

Episode Date: October 17, 2017

Mike Glukhovsky joined the show to talk about the future of RethinkDB. Mike was a co-founder of RethinkDB along-side Slava Akhmechet. RethinkDB shutdown a year ago officially on October 5, 2016 — and today we're talking through all the details with Mike. The shutdown, getting purchased by the CNCF, relicensing, buying back their IP and source code, community and governance, and some specific features that Mike and the rest of the community are excited about.

Bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. And we're hosted on Linode servers. Head to linode.com slash changelog. This episode is brought to you by CircleCI. CircleCI is how leading engineering teams deliver value faster. By automating the software development process, using continuous integration and continuous delivery,

you are free to focus on what matters most, which is building value for your customers. CircleCI is everything great teams need. Support for any language that builds on Linux, configurable resources, advanced caching options, custom environments, SSH access, security through full-level virtual machine isolation, interactive visual dashboard, first-class Docker support, and more. Get started with their free plan, which gives you unlimited projects and 1,500 bills per month. Plenty to get started with.

Head to circleci.com slash changelogpodcast. Thank you. show Jerry and I talked with Mike Lukowski about the future of RethinkDB. Mike was a co-founder of RethinkDB alongside Slava Akhmachet. We've talked with Slava in past episodes so check the show notes for links. RethinkDB shut down a year ago officially on October 5th 2016 and today we're talking through all those details with Mike. The shutdown, getting purchased by the CNCF, relicensing their code, buying back their IP and source code, community and governance, and some specific features of RethinkDB that Mike and the rest of the community are excited about. So Mike, we've had quite a journey with RethinkDB. We've covered on episode 181 and 114 with Slava,

kind of the background of Rethink as a technology, the company. And we're now at a point where we're looking back and Rethink the company had failed, went defunct, and ultimately has moved to this newer community-driven project. For those who are catching up, help people understand kind of who you are in this role here. You've co-founded RethinkDB with Slava. What's your portion of the story here? Yeah, thanks for having me on the changelog. I really appreciate it.

I have spent a lot of time working on RethinkDB. So looking back, Slava and I started RethinkDB back in 2009. So almost eight years of my life. And it's been pretty incredible. Slava and I started RethinkDB as a technology startup. We were backed by Y Combinator back in summer of 2009. And we decided to build something designed for modern hardware,

for solid-state drives, for multi-core CPUs. We wanted to build really low-level systems technology. When we originally started the company, we were focused on building a storage engine for databases. And as time went on, we really started to build not just a storage engine, but a full database. And eventually we decided to open source it. When we first released it to the world,

it took us almost four to five years to be able to build RethinkDB before we were ready to share it with the world and get it tested, get it out in the community. We really decided to flip that switch and become an open source company. My background is that Slava and I both studied computer science. My background is that Slava and I both studied computer science.

His background is in distributed systems and storage technology. Mine is in user experience and design. And so we come from very different backgrounds. But between the two of us, you got RethinkDB, which is a very capable, very powerful database. Scales linearly, is open source distributed, allows you to run hundreds of thousands of real-time streams in parallel and be able to solve some really interesting and really complex use cases. And I was really focused on making the database usable,

making it friendly, making it something that people could engage with. When we first looked at building RethinkDB, oftentimes people look at system software as being cold. You interface with it through a terminal. You read really complex and arcane manuals. You spend a lot of time deciphering

really complicated query languages and APIs, trying to understand how the thing works. And they tend not to help you along the way. But when we decided on RethinkDB, we really wanted it to feel very much like a consumer product, to feel accessible to developers who are just starting out building their first application. And so I spent a lot of time working on things

like the dashboard that's in RethinkDB. I spent many late nights designing and building the user experience of the database, thinking about how people use it. I built the community operations for RethinkDB. We had a really amazing worldwide community that was deeply engaged in the open source experience. Open source manifests as being a software development model,

as a license type, as a community. And I spent a lot of time thinking about how to make RethinkDB really accessible in terms of our community as well. And a lot of the work that we did in making RethinkDB really friendly and accessible and capable has borne through as the company shut down.

Because when the company actually shut down, the community just came up in arms and said, we're going to keep this thing going. Whatever it takes, we're going to champion the needs of RethinkDB and we're going to continue building it. And so my experience has been basically anything that required doing, I worked on it, Rethink. And these days, I have taken a lot of the experiences I've spent in the open source community and I work on developer relations at Stripe so I help run developer relations for Stripe which builds developer tools for commerce and for finance. Right you're at Stripe now is it because

of the transition of engineers and whatnot that was, I guess at that time, transition to Stripe, Stripe stepped up and said, hey, we'll hire. Is that why you're there? Are you there for other reasons? Is that what got you there? I guess what I'm asking. Yeah. So our team is pretty incredible. We have like a really incredible team of engineers who built Rethink TV. And when the company shut down, we had lots of options of where we could try to take our team to be able to feel like we could continue working together, continue building things together.

And Stripe was by far and away one of the most exciting options available to us. And I personally am really excited and dedicated to what Stripe is working on because having worked on developer tools for a long time, there's immense power in allowing creators and builders to be able to use technology to be able to solve problems that previously required teams of people

to solve. And Stripe is really building an abstraction for the world of commerce, for the world of business, to be able to allow people to redefine how they build companies, to be able to redefine how they build payments infrastructure around APIs and around developer experience. And the whole RethinkDB team has spent a lot of time thinking about how to abstract away the world. Software represents abstractions about the real world. And we've spent a lot of time thinking about how to help developers represent the real world and to be able to use those abstractions to build really powerful things.

And so Stripe is a really natural fit. When you look at the developer tools companies out there, there are a few of them, but Stripe is far and away one of the most exciting companies out there for developer tools. So it's been a really comfortable home for the RethinkDB team. Gotcha. So to paint some of the picture too for timelines, you said that Rethink began in 2009. That's roughly around when we started the change law too, by the way. So that's kind of crazy. But in terms of like overlapping there, operated as a company for several years, took VC,

out of Y Combinator, kind of glossing over some of the details. But then in 2016 was the RethinkDB shutdown post, kind of describing some of the things you just mentioned here. And Slaviv and Penda's own Y RethinkDB failed and kind of pulled out some thoughts on developer tools and how he had some speculation on the market and whatnot and shared some hindsight notes about how you chose a terrible market and optimized the product for the wrong metrics. So there's some downsides there, so to speak, but that's roughly the story there.

And now we're transitioning from a failed company, to put that loosely, I don't want to say like y'all messed up, but just easily said, failed company transitioning into a community-driven open source project. It's always been open source, but now it's community-driven. So help us kind of paint the picture from where you failed as a company to CNCF stepping in, even a lot of the details there around getting the code back and being able to use that. Help us fill in the gaps on those details. Yeah, absolutely. So it's really interesting to look at open source development.

The comparison that I always really appreciate is the bazaar in the cathedral, which people are probably familiar with. But I'll repeat for those who aren't that, you know, open source is the bazaar. It's messy. It's loud. It's complicated. There's lots of things happening. It's hard to keep track of what's going on. And in contrast, the cathedral is closed source. It's very polished software development that comes out with very precise, very clear products that are shipped. And you don't have a lot of interaction between people who are using it. It's very prescribed. And what's interesting about when you build open source as a company is that you can kind of strike this happy medium between the two, where essentially you have the resources of what

you would call the cathedral model, and yet you're able to still work with the bazaar and to open and invite people to be able to contribute, collaborate, and work on the project with you. So RethinkDB has like hundreds of thousands of users around the world who use the project. And throughout the company's history, people regularly were contributing major features to it. We would have pull requests that would come in. We had an open development model on GitHub. We did everything out in the open. It was no private trackers,

no private communications that were significant. Everything we did was in the public, in the open on GitHub. And so the community would come in and work with us and be able to build features together, close a product feedback loop, be able to help us ship things more quickly because we understood what their needs were.

And when the company shut down, the intellectual property was in a state where it was held by a third party because when companies shut down, typically investors will hold on to certain assets of the company and so we found ourselves in this position where the open source license was a gpl so people could continue using the project and feel comfortable using it but we were unable to

change the license or make any significant changes because the intellectual property was in the hands of a third party. So the first thing that happened is that people said, we're not going to let this die. It really doesn't matter what you guys do because it's open source. So we're going to keep going with this. But if we are able to retain control of the assets, we can make some changes we want to make. So for example, when the Linux Foundation ultimately ended up, and the CNCF ended up acquiring the assets of RethinkDB, they changed the license to the Apache software license, which some people consider to be more permissive, because it places less restrictions on how you use the software. That was one change that they

really desired, and I understand why. They also acquired a lot of the closed source assets of the company. We had a version of the database called RethinkDB Enterprise, which had only a small handful of features, things that were useful to very large organizations like audit logging, audit trails, things like that. And we're working on sharing those more broadly with the community and actually getting into the open source version. So essentially, people stepped up and said, we're not going to let this go away. So I agreed with them. I didn't want RethinkDB to stay in limbo. And so I worked very closely with the open source leadership team, which was comprised of former RethinkDB team members, including myself, and a number of folks from the open source community who really had made this a part of their lives. They had worked with us for years.

We had very close connections and relationships in that community. And they were our friends. And we all shared the same passion to be able to keep the project going. And after a few months, we decided to be as public as we could about the process. And we held regular meetings. We have a Slack channel where we had over 1,000, 1,500 suggested that they would be interested in acquiring the IP and assets of the software project. And so we worked with them to close a deal where we were able to move over the control of the assets and be able to relicense the project. So yesterday, we actually just shipped the first release for the open source project under the hands of the community, which is 2.3.6, which sums up a series of stability fixes,

bug fixes, and improvements. And it's also the first official release that has been fully under the Apache software license. And we're steadily working on releasing more of the things that we've built, including some of the closed source features that we had built in the past eight years. And other things like we had a full time resident artist illustrator who was creating art around the open source project. This is something that I really cared about because to me, software is a vehicle for ideas. But art is also a vehicle for ideas.

It helps you express the way you reason about the world, the way you think about the world. And so in building a community project, we had a lot of art that we created around it, which really helps people understand the ideas that we're communicating. And so all that art we're working on also open sourcing. A lot of the projects that we had internally for build testing, performance testing, load testing. They're all getting steadily open sourced. And so the challenge and the opportunity for the project is to figure out how to move to this community-based model. You guys have great artwork, absolutely. So hats off in that direction. Take us into the nitty gritty a little bit

and help us understand from the outside the legal ramifications of what you said back there with the AGPL and acquiring the assets. So if I understand you correctly, because it's AGPL, the community could have forked it, because it was open source and open license enough that they could fork it, rename it, and continue on. But there were drawbacks to that, such as, you said, the artwork assets, other things, the name, of course, other aspects of the intellectual property

that because the open source project was operated by a company that had creditors, people who had invested into the company, they had ownership of the intellectual property. Am I following this correctly? Yeah, so RethinkDB started as a closed the company, they had ownership of the intellectual property. Am I following this correctly? Yeah. So, I mean, RethinkDB started as a closed source company back in the day. And so the terms under which we were working with investors were very different than our

needs four or five years into the company when we really open sourced everything. And so the way that license and copyright work, they're very different. So the license is the terms under which you can use the project. So when it's AGPL, that means that someone could very well fork it, rename it, do whatever they like. In fact, lots of people have forked RethinkDB. And that's encouraged. That's the goal of copy left software is to every time a fork happens, the terms of the AGPL will continue to stick to those forks. And while that's something that was perfectly workable

for users, a lot of developers wanted to work on it, but they weren't necessarily comfortable with the AGPL. I have a neutral opinion on the AGPL. I see both its pros and cons. But the will of the community was very strong that a lot of folks were committed to working on it, especially if it was under the Apache software license. Because a third party owned the actual copyright to the code, we were unable to change the license. And it's not even about whether it's a change from AGPL to Apache software license or anything like that. It's about thinking what happens in 10 years or 15 years when a new license type emerges and we decide that we may want to shift the license for whatever reason or give ourselves any sort of forward flexibility. And so being able to actually retain the name, the common

law trademark usage of RethinkDB and not having to rename it, being able to keep our website, being able to keep our documentation, being able to keep domain names, all the collaboration and communication tools, the GitHub organization, the repository of issues that we had built up for thousands of issues that were recorded, all of those details. It's best for the forward velocity of the community project if the Linux Foundation and CNCF have that. And so when the LF acquired the IP and assets of the software project, it really made a lot of problems go away. Otherwise, you'd have to start at ground zero again, right?

You'd have to go back to, I mean, you would have the code to fork, but you'd have to... There's a lot of stuff you'd have to leave behind. You'd have to go back to, I mean, you would have the code to fork, but you'd have to... There's a lot of stuff you'd have to leave behind. You'd have to rethink everything. Re-rethink DBs. That's right. Tell us about the Cloud Native Computing Foundation, the CNCF, as you mentioned, which is associated with the Linux Foundation,

or is the Linux Foundation. Help us understand who that is and how they can swoop in and buy the assets. Tell us about that. The CNCF is very interesting. It's part of the Linux Foundation. The Linux Foundation has a number of foundations under its Aegis umbrella.

What they really focus on are technologies that are designed for the cloud-based environments that we're using today. So that's why they're called the Cloud Native Computing Foundation. Examples of projects that they work on are things like Kubernetes, which you guys may be familiar with, which to me is one of the most exciting, or if not the most exciting, infrastructure change or project that's popped up in the past 10 years. And a lot of the projects that they work on are designed to help build the infrastructure necessary for the next 10 years of what we're going to need for cloud computing. And so Brian Cantrell, who has been a longtime member of our community, we've collaborated with him on several projects, and he's been a joy to work with. So he approached us and said that the CNCF would be willing to work on acquiring these assets.

And between them and the LF, we've been able to make this change very, very quickly. So they had a lot of enthusiasm and energy for completing the transaction and for helping establish the future of the project. Right now, we are part of the Linux Foundation, but we are hoping to establish governance soon, community governance, to be able to decide whether we want to join the CNCF or remain as part of the Linux Foundation. And the CNCF has certain structure to it, certain ways they want projects to be run. So that's really something, it's a decision that needs to be made as a community together. So the CNCF slash Brian Cantrell found motivation to approach you to say, hey, let's help you

acquire your assets, which requires dropping a check, basically, and taking care of creditors to get that IP back and to be able to do all the things you just said there. Where do you think that motivation came from? Why is RethinkDB so important to the community, but also to those individuals involved and organizations involved to drop some money down to acquire the IP back and have full control of it?

So I can't speak for why lots of other people find RethinkDB to be so important. I can say why I think RethinkDB is important. And I think it's because RethinkDB is one of the system software projects that is entirely built from scratch. It's C++ and assembly all the way down up to JavaScript. It is as complete a software undertaking and endeavor as you can imagine. There's eight years of work in it with a team of up to 17 people working on RethinkDB. And that's a lot of man hours.

That's a lot of things that were built, a lot of really low- level, interesting software technology. And we also were at the forefront of a lot of changes that happened in infrastructure software. So, you know, being able to add real time streams to databases, thankfully, that idea has really started to pop up in the other parts of the community. People are starting to move to the idea of real time subscriptions, and thinking about how databases can can be shaped differently. There's the query language, which is really incredible because the way the software is built, it essentially is a giant distributed computing environment that allows you to run

functions that operate in parallel in a distributed cluster. And all of that is really, really, really neat technology. So to look at that and to consider that for a marginal cost, you can buy back the ability to direct the future of the project. I think that the CNCF and the Linux Foundation really felt like it was the most reasonable of investments. And in terms of why it's exciting, I think it's because for a long time, RethinkDB has been thinking about where databases should go, what the future of databases should look like. And being able to continue exploring those ideas as a community seems just deeply powerful and deeply useful to continue as an entity. And it's really exciting because even in the next version of RethinkDB that we have planned,

it already unlocks so many more ways that RethinkDB can be used to solve all sorts of different types of problems. And that'll be RethinkDB 2.4, which is being worked on right now. Coming up at this point in the story for Rethink, the company has shut down creditors are holding the IP and source code and meanwhile the community just wants things to carry on they want the software to carry on we cover what the actual value of the IP and source code is worth what they end up spending to buy it back from their creditors and we also talk through the new governance model and how they can move forward as a community-led project.

Stay tuned. This episode is brought to you by Bugsnag. Bugsnag is mission control for software quality. And on this segment, I'm talking with James Smith, co-founder and CEO of Bugsnag, about the core problem they're solving for software teams and why you should head to bugsnag.com slash changelog to test it out with your team. Let's start with, you mentioned you and Simon. So you guys obviously at one point didn't have this company, right? So as founders, as engineers, you got to a problem. What was that problem? Why does Bugsnag exist? Simon and I, my co-founder and I met in college. We went off to build software for other companies. I ended up in a startup. He ended up in enterprise software. And we had the same problem in both of these companies.

When things break, it's really hard to figure out how badly they're broken, who's impacted, and what to fix first. So we both had this problem ourselves. So we decided, hey, why is no one doing a good job of fixing this problem right now? So very much Bugsnag was born out of scratching our own itch, as they say. One thing that we find all the time is that there's this tension in software teams or in product companies where you want to deliver new features to your customers, or you want to build cool new stuff. But at the same time, you've got to fix bugs because no matter how good a coder you are, you're going to introduce bugs.

But there's no clear definition of where to set that slider. Should I be fixing bugs now or should I be releasing features? And so this tension exists, I think, in all product teams, all software teams. If you don't have a tool like Bugsnag, it's very difficult for you to figure out where to spend time. And so that's the idea here is we're trying to help teams understand whether they should be building or fixing because there's a bit of a delicate balance between both. Absolutely. I couldn't agree more.

So if your team is unsure of how to spend their time building or fixing, give Bugsnag a try. It's free to get started with a 45-day extended trial exclusive to our listeners. Head to bugsnag.com slash changelog. So the company shut down, Rethink shut down October-ish 2016. Creditors held the source code and clearly based on the story you've shared with us, Mike, the community and everyone else wanted things to move forward. And for the reasons you mentioned before of like not wanting to restart over or

fork the code and think of a new name or you know you wanted to reuse this past seven-ish years of work to move forward but that required someone to come in or basically to buy the assets back from the creditors that's you know vc that's whomever held the IP when the company went defunct. Can you share roughly what that figure might have been? I'm sure the community is thinking like, geez, this must be worth a lot of money or whatever. What are we talking about here? Yeah. So Brian approached us and was super excited to try to help figure out how to take next steps for the project. I think he really personally cared about Rethink2B's mission and figuring out what it would take to to unstick it. And thankfully the number was not that high.

I think largely because the folks involved who held the IP really recognized that the people who were most interested in the future of Rethink2B that really were dedicated to it were the community. They were the ones that were really asking for this shift to happen. And so we were able to negotiate a very reasonable number, which was $20,000. And I think that is a testament to the fact that the community was very vocal about its needs and that it was clear it wasn't a corporation that was acquiring the assets, the community was very vocal about its needs and that it was clear it wasn't a corporation that was acquiring the assets, that it was something for the community's benefit. And they made it really easy to amicably make the transition. And thankfully,

the Linux Foundation and CNCF stepped forward to be able to facilitate that. And they've given us a lot of the resources we need to be able to continue, like the infrastructure that you generally need to be able to do things like accept donations for future software development of the project or be able to help us with hosting of domains and things like that and be able to really give us the legs we need to stand on as a community-based software project going forward.

Well, just, I guess, mad respect to the negotiators and as well as to the creditors for negotiating that price, because that's an excellent price for what everybody, what the world got out of it, which is a free and clear RethinkDB that can be handed over to the community. Tell us what the community looks like, because everybody's got one, but they all are different. And I guess with RethinkDB, it's probably even a little bit tough to make that actual transition because it was a product from a product team

inside of a company for so long, even though y'all were developing it open source style. Like you said, it was a middle ground between the cathedral and the bazaar. So now you're going full bazaar and you gotta figure out what your bazaar is going to be shaped like. So what does it mean when you say community governance for RethinkDB? That's an excellent question.

We're lucky to have a lot of people care about the project, as I mentioned. It's funny because people used to say that after checking Twitter and Reddit, they would go to GitHub and they would just catch up on all the emails that were being sent between our developers to sort of just sit and observe how the project was unfolding. So on GitHub, we have like north of 19,000 stars, a few thousand people, I think, who watch the project. And when you watch a project, it essentially means that every email that gets sent out gets shared with every single person who's watching it. And it's funny because we

kind of had this like silent audience that was observing everything we were doing and watching everything unfold, just eating popcorn and watching as it progressed. And so now those voices started to make themselves heard. And in the Open Rethink TV channel on Slack, as I mentioned, we had like 1000 plus people all advocating for the future of the project. And while it was in limbo, we're trying to work things out. People were debating, should we try to fork it? Should we try to organize development somehow? Should we try to build a software development fund? And I think that the nice thing about being under the aegis of the Linux Foundation is that it allows us to have a very safe home from a legal perspective, from a structural perspective. But what we really are trying to figure out now is how to decide what decisions we're going to make for the future of RethinkDB in terms of its technology. And so the most important thing that we're going to be building is a technology steering committee to be able to

decide like what features are going to go into the next version, what is something that should go into the API and shouldn't go into the API. A lot of these decisions only happen through lots of conversation, lots of structure, lots of discussion. And at RethinkDB, we spend a lot of time. You can even just go, if you look at the issues for RethinkDB, take a look at, for example, the dates and times issue,

where we added date support to RethinkDB. And dates are notoriously complicated because everyone disagrees on what calendar you're using, especially if you go back through history. Calendars changed based upon political rule or based upon particular standards that would emerge. If you look at time zones, there's even time zones that are half an hour off,

45 minutes off. So these things get very complicated. And how you represent it in a database, very important because you're going to do a lot of queries based upon dates and times. And if you just go to that thread and just observe the hundreds of comments that unfolded, just trying to figure out every single angle, every single nook and cranny of how this should be added to the database, that kind of agitating

force is what's really powerful about open source. A lot of voices come to the conversation. They are able to agree on what the best thing to do is only through letting the ideas do violent conflict sometimes before you emerge with the best result. do is, only through letting the ideas do violent conflict sometimes before you emerge with the best result. And so building a group, like a committee, that will be able to think about these things and do so on a regular fashion is really important to the open source feature.

Aside from that, we're looking for folks who will be able to take over, for example, different language communities. So we have drivers that exist for Java, Python, Ruby, JavaScript, a lot of other languages. And we want people to be able to help make the experience of Rethink to be better in each of those language communities. I'm here on the issue tracker, and I'm trying to find that issue, but there is 1,400-ish issues. That's quite a bit. Is that potentially because of this cycle we've been in since last October

to get to where we're at now? Is that why there's so many issues? What's the state of issues? Just curious. So issues are a funny thing on GitHub because people think about issues differently for each project. Right. Like a to-do list or something like that. Should it be clear, basically? Yeah. And so basically, every time that we ever had an idea on what to do with the database, we would just create an issue for it.

And then it would allow the conversation to unfold for some crazy speculative idea that wouldn't see the light for even sometimes two or three years. And so if you look at a lot of the open issues, not all of them are bugs. A lot of them are like, I can't build in this particular environment, or I'm having difficulties with this particular workload. And aside from that, many of them are speculative. We should build some new feature. And if you look at the number closed, which is 4,500, it dwarfs it significantly. But if you just search, for example, for dates and times, you'll notice, which should be, I think it should just be just if you search for dates, it should pop up. Then you should be able to see just these proposals that are titled,

for example, recall proposal. And in a proposal, we basically start exploring a new idea to be added to the query language. And it's tagged with a particular tag. And then we just let the discussion roll for a long time until all the ideas are exercised. And that's why there's so many issues on RethinkDB. And sometimes people look at it and they wonder, like, is there a problem with the database? And the reality is that that's just how we approach. Or stagnation. Yeah. I mean, when you get those numbers, you kind of think, like, is it being managed? Is somebody triaging these things? You know, it's almost as if you have to have the precursor of like, this is how we do issues on GitHub kind of thing.

Like you just said there, basically, cause otherwise you come to it and you're thinking like, is it, you know, is somebody triaging these issues? Does it, you know, is there being, is really being tracked that kind of thing? Yeah. And that's a lot of the work that, um, folks on the leadership team are doing today is working through those issues, replying and triaging. And obviously we would appreciate a lot of help from more folks in the community and helping keep track of all these issues. Easy call up there. So you'd mentioned some things around governance, TC.

It sounds a lot like you're going through with RethinkDB, what Node went through roughly a year ago, year and a half ago when there was a major fork where in the case of RethinkDB had the acquisition of the IP not happened, there could have been, you know, an IOJS slash Node.js kind of scenario. I'm assuming here, but who are you leaning on in terms of, or what communities have you leaned on to get the insights you're getting to go the way forward that you're going? We're getting a lot of advice from folks at the CNCF and the Linux Foundation.

We also have some folks who have been involved in a number of other projects like Chris Abrams and Roscoe Kalinsky who have seen things happen in the Node world and have a lot of deep connections in the open source community. And so we're not looking to innovate as much as take the minimum viable organization that we need to be able to ship new releases on a regular basis and really feel comfortable for the next bit. Unlike Node, it's not terribly contentious. It's more collaborative.

Sometimes that so many people use Node.js and so many people cared about the future of it that split between I.O. and Node happened because people were dissatisfied with a lot of how things were unfolding. And in the case of RethinkDB, it's a lot more collaborative. Folks are just trying to understand the best way that we can all build this together going forward, rather than arguing over its future. Just going back, thinking about Adam said the word stagnation, and when I think about the cathedral and the bazaar metaphor,

and the one thing about the cathedral is you don't have to ask for anybody's opinion. You just push forward the way that you think is best inside of the product team. With the bizarre, like you said, things have to come out of violent arguments or sometimes violent agreement, whatever it is. Ideas have to prove themselves first and so there's a lot more conversation happening and sometimes that can result in a lot less agreements. If all you have is disagreements, then you're not exactly going anywhere. I'm curious what your thoughts are on that.

And the reason why I ask is because, and I'm not trying to hold you to the fire or anything because we're software people. We know about ship dates and stuff, but in the Linux Foundation post back in February, it said that RethinkDB 2.4 was coming a few days later, and as you said earlier,

2.36 just shipped July 17th. There was something that happened there, and I'm wondering if the transition may have caused some stagnation, or if there's actually potentially a problem of disagreements and generally slower moving product because of the community governance.

So I agree with you. It's definitely slower moving than I personally would like. One of the things that RethinkDB really pioneered in its development model is just rapidly shipping releases. So we had a goal of shipping a new version of the database every six to eight weeks. And each time it had major features like secondary indexes

or adding a distributed cluster using Raft. And each of these releases was the product of extremely careful scheduling and trying to figure out how to keep the pulse of the database moving quickly because it is the cathedral. I personally think that a slower pace is a healthy thing for a database. RethinkDB has only really been production ready for three or four years now.

And you look at something like Postgres or even MongoDB, and it takes a long time to let databases really bake. And once they're built, they don't really need to have, like you could argue that dramatic shifts for a database is not necessarily a good thing. Because people are relying on it. It's like the most core part of their infrastructure, which is very different from something like Node where you're building for a community

where the web is moving so quickly. People have framework fatigue, new features are being released, async, await, people desire it yesterday, and so there's a lot of pressure and a lot of loud voices. In the case of databases, I would argue that moving to a community process, it's probably healthy to slow down the pace of software releases. Stagnation is very different.

That means that you just stop work, right? One of the biggest problems that we faced when we wanted to ship a new release for RethinkDB was that we did not have our test and build infrastructure that we had built. We had RethinkDB when we were a company, racks of servers, some with like 32 cores on each machine, just working on producing builds for the database, working on testing every night, automated testing infrastructure. All of these things are really, really useful to have. And when you are going to rebuild as a community project, you need was letting all of the work that we needed to just catch up and then decide, okay, now we can ship our first community release for RethinkDB. And so it's really monumental in the amount of effort that's gone on behind the scenes. I would definitely name Marshall Cottrell, who's been a longtime member of the project, and Etienne Loren, who was a core member of the RethinkDB team and has just tirelessly continued to work on it, along with Sam Hughes and Ryan Paul. quietly working on tidying up all the issues that go into shipping a new release to multiple

platforms for distributions. 2.3.6 also included packages for new releases of Ubuntu for Yakety and Zesty and unifying and just making sure that all of the bug fixes and stability permits went into it. We also shipped multiple release candidates to give the community time to be able to try to air it out because we are leaning more heavily on them for the build and test aspects of this. And as time goes on, we plan to speed this up because we're going to have a lot of the community processes down. But I think that moving wisely and steadily is better than moving quickly when it comes to system software. Yeah, that's a great answer. And the other side of my question about stagnation was another question about instability, and perhaps not technical instability, but as the case has been over the last year with RethinkDB, at least since October 2016, when the shutdown

was announced, it was uncertainty of like, what's going to happen with this thing that we love or rely upon. And the last thing you want in terms of instability or uncertainty is with your data store. And so I think the silence as the transaction was happening, the negotiation that was happening, between October and February had probably a lot of people wondering what was going to happen. But since the joining of the Linux Foundation

and this transition and a move into the community governance a bit of a return to normalcy and maybe that normalcy is a little bit slower than it was previously but at a healthy pace is probably providing a lot of certainty and answers for people who are either running RethinkDB in production or considering picking it up for their next project. Yeah, I had a lot of folks in the community ask me, should I be worried when the company

shut down? And quite honestly, I mean, my answer has always been very clear that there is not a commercial entity that is behind the project. And if that affects your choice as to whether to use the database, you should factor that into your consideration. But you should also weigh it against the fact that it's been worked on for eight years by an extremely competent and really deeply technical team.

And it's been vetted and tested in some of the most rigorous and difficult benchmarks in the industry. The distributed systems test that was performed by Afer to test the new raft implementation for our distributed clusters passed with flying colors after our team worked on it for a year. And this is not something that goes away when the company goes away.

And the other side of it is that there are a lot of teams that are still building for it. So as an example, there's some teams at IBM that have been doing a lot of work on RethinkDB, despite the company background that's irrelevant to their projects. For example, porting it to PPC or Z, and those teams are just working on it in the assumption that it's a solid database, it continues to power a lot of infrastructure on the web, and there are really no extraordinary problems people have been encountering.

Up next we get into some specific features of RethinkDB that Mike and the rest of the community are pretty excited about. We talked through the power of RethinkDB's query language, Requel, and the idea of Modifier functions which lets you embed Requel natively into your programming language for the database to execute on every write operation, and also how they differ from stored procedures. And finally, we talk about the future of RethinkDB, the role of CNCF in that future,

and how you can get involved. So, stay tuned. is brought to you by Linode, our cloud server of choice. Everything we do here at Changelog is hosted on Linode servers. Pick a plan, pick a distro, and pick a location, and in seconds, deploy your virtual server. Jewel-worthy hardware, SSD cloud storage, 40 gigabit network, Intel E5 processors, simple, easy control panel, nine data centers, three regions,

anywhere in the world they've got you covered. Head to lyndo.com slash changelog and get $20 in hosting credit. So Mike, you've taken us through a little bit of the history of RethinkDB. You've taken us through the transition from RethinkDB, the company running RethinkDB, the project, to the community-led governance for RethinkDB, the company running RethinkDB, the project to the community-led governance for RethinkDB. We haven't talked much technical about the database itself, except for its merits and why so many people have come to use and rely upon it. But we want to look into the future a little bit and see where it's headed now that it has this complete freedom from IP constraints and this community-led governance. What are some technical aspects of the software

that are coming down the pipeline, things that maybe you're interested or excited about? One of the things that I'm most excited about is a new feature that we have had in the works. It's mostly implemented at this point that will improve the query language by adding what we call modifier functions. is a new feature that we have had in the works. It's mostly implemented at this point that will improve the query language by adding what we call modifier functions.

So basically, for those who aren't familiar, one of the biggest strengths of RethinkDB is that it has this query language that in most databases, when you write a query, you normally will write it in something like SQL or in the case of MongoDB, some custom JSON, and you will compose it using some of their syntax, and then send this query to the database and return a response. And in RethinkDB, our query language, which we call Requel,

allows you to basically write functional programming calls that embeds natively into your programming language. So as an example, r.table.filter.map.reduce. And each of those, what we call terms, like filter, map, and reduce, are just functions. And each function can accept things inside of it. You can do embedded functions. And what this essentially feels like is a functional programming language that's baked into the database and is also

embedded into your native programming language, which is really useful because you can do things like use your debugging tools and it doesn't feel like you're switching context. And at the end of these queries, you just send off the whole command stream to the database and then it will run in a distributed environment and return the computation back to you. And this just looks very simple on the face of it. Like you're just writing inserts and gets, and you're doing very fluent commands to be able to manipulate the data on any of those queries or on many of those queries. So you can then open a stream by just adding changes at the end. So the query language is really innovative. It feels really comfortable. People say that it

takes a little bit of time to really grok it, to really feel like they understand that these functions are running in this environment within the cluster. But once they do, they get addicted to it and they just want to use it all the time everywhere. And I genuinely think it's like one of the best parts of the database. Like it's one of the things that we're proudest of is a bunch of programming language design geeks got together and basically built this really beautiful functional programming language that allows you to operate on data in this distributed computation environment. And that's really exciting. But one of the things that we got as a consequence of building a really well-designed query language is that every time we added a feature to the database, we were able to reuse it in lots of different ways. Every single thing that we added enhanced the rest of the database. So modifier functions are like

this because basically it's a function that is applied to each write. So every time that you perform a write on the database, you can then use a modifier function to do something. And what this allows you to do is essentially add things like schema validation. So every time you perform a write, you can check if it matches a schema. To be able to rewrite documents or to add additional fields whenever you write something to the database. To be able to document expiration.

So you can say if this document is older than a certain date or time, then it should be deleted. And to do things like security rules. For example, you can imagine adding a modifier function that would say only inserts are allowed to run. Otherwise, any other type of write will be discarded. And so basically, this is the building block that will allow us to add things

that people have been asking for for a very long time, like document expiration and schema validation. And we expect that, like any good piece of technology, people will use it and reuse it in ways we don't expect. And we'll discover all sorts of things that they're pushing the query language to do

that it just wasn't capable of doing before. And when that happens, we can try to build porcelain commands that will very simply allow these features to be exposed and they'll be implemented in terms of these modifier functions.

So we're giving this toolkit out to the community to see what they're going to play with and what they can come up with to these modifier functions. So we're giving this toolkit out to the community to see what they're going to play with and what they can come up with to solve use cases that maybe we hadn't realized RethinkDB was capable of helping with. So where are you authoring these functions or these modifiers? Is it inside of this tool that you built, the web interface? Or how do you actually go about adding those to your

RethinkDB database? So basically you do it through the query language itself so you can specify the modifier function will live on a table. We may eventually expose that in the web interface as well. If you look at a corollary might be the statistics that we

built into the database. So we added a system table, which is essentially a table that tracks lots of details around the cluster and around the database. And it's exposed just as a table in RethinkDB. And that's really powerful because you could then query that table

using the query language. You can open a real-time stream on any aspect of that table. Or so you can say, for example, watch for whenever a certain threshold gets hit and then like, add more replicas within the cluster to react to that change. And that's all can be done with a rethink to be square language. And we also exposed it within the web interface. And it's just implemented in terms of rethink to be queries. So that's really powerful. And you can sort of see

what I mean by building a building block and then reusing it in lots of different ways. So by allowing modifier functions to be declared on tables, you can see how people can then use that to build on top of it. Things within the web UI, command line tools, and other rethink DB commands. So I'm coming from a kind of an old school RDBMS angle. So to me, this sounds like stored procedures. How is this different than what I'm thinking these functions are? So stored procedures are close because stored procedures are run on whenever a command will happen and the procedure is stored on the database. This is just RethinkDB's version of that. And it's expressed in terms of modifier functions because we are a functional

programming environment. But it's very similar. You can do a lot of the things that you want to do with store procedures with these modifier functions. So the age-old question is, how much logic do you put into the database and how much do you keep inside of your application code? I'm not asking you to answer that necessarily because I think that's a fit for the case-by-case scenario or even developer by developer, perhaps. But it sounds like with the way that RethinkDB's query language works, you could be writing these modifier functions, but it almost feels like your application code.

Is that what you're trying to say? Because of the style of it and the fluency of the recall? Yeah, absolutely. So take expiration as an example. If you wanted to be able to expire documents today within RethinkDB, what you can do is very simply open a change feed and subscribe to all documents that are older than a certain date or time, and then delete it whenever they pop up, whenever a new change appears within the table.

And that is done across the database application barriers. You have to have some sort of application process watching the database, subscribe to that stream, and then just periodically deleting it. And that's how a lot of people implement expiration today. It's totally reasonable. It works very well.

But there is some efficiency to be gained by doing that in the database because it has more knowledge about what's happening and you don't have to build as much infrastructure outside of it. I'm not arguing that you should always do things within the database. Every use case is different. It's about how much baggage you're willing to pick up and how much work you're willing to do.

It's about trade-offs. The pragmatic programmer likes trade-offs. Yes, and we're all pragmatic, aren't we? Hopefully. When do the modifier functions is that something that people can expect soon? Because as a building block, you kind of want those out there ASAP so that people can start doing things you never thought of.

Yeah, they're available in a branch on GitHub. I think there's still a little bit of work that's left to do on it, but there is an issue that's tracking them right now. And if folks are interested in learning more about it, they're welcome to join our Slack channel and just ask in hash open RethinkDB and be happy to direct them to it

if they want to test that feature. And in terms of diving into the background details too, you mentioned too that a lot of things propagate through issues and there's an issue for this. We'll link in the show notes.

It's got, you know, the specification for it, the introduction to this proposed feature and then 23 comments, so kind of going back and forth. And this is a closed issue, so it's something that I guess people can kind of come to and say, let's learn the history of this feature, basically. That's right.

Cool. Anything else, Mike, that you're interested in that's in development, coming soon? There's a number of ports that people are working on. I mentioned that people at IBM are working on porting to PPC and a couple other environments. There's also a very excited group of users who are working on porting to ARM64 because they want to be able to run on

things like ARM environments like Raspberry Pi and things like that, which is really awesome to see. People are really diving into figuring out how to bring RethinkDB to their environment, which at this point, it's a pretty mature and stable project. And so that's the question is, how do I get to use it more often than what can't it do? The other thing is really just open sourcing a lot of the features that were baked into RethinkDB Enterprise and other things that were in development but not finished up. And one example of this is like audit logging is an example. Being able to

tell all of the queries that were run in the system is something that a lot of people have asked for and we're excited to be able to just drop it into the next release. So as we look to the future of the project, one thing that we always have to think about with the future of anything is how is it going to get to the future? So we talk about sustainability. You mentioned earlier under the umbrella of the Linux Foundation, they helped you guys get set up to take donations and have some infrastructure around that. Tell us about that and how people and who is helping to financially or with developer

hours or however help support the future of Rethink2B? So right now, the Linux Foundation has enabled us to accept donations and folks have really stepped up. We've had a number of people contribute to the project, people to help maintain servers, keep the downloads available for the public, and hopefully in the future establish a way to maybe even hire developers to work on certain features for the public, and hopefully in the future establish a way to maybe even hire developers to work on certain features for the project. We've explored ideas like open source

bounty systems and other ways of organizing through tools like Patreon. Luckily, Stripe has been very generous and is matching up to $25,000 in donations from users. They've already helped us with the first set of donations that we've received. And every bit obviously helps. But the thing that matters more than financial support is strong C++ engineers who really want to participate

in what is a mature, powerful database. It's one of the largest projects on GitHub right now. And it's really deep, intricate, and people generally say very tidy and well-designed code base. And there are some really awesome features we want to build, especially around the real-time technology that is baked into the database. And being able to have more help from good systems engineers who really care about figuring out the future of databases would be really fantastic.

And outside of that, any technical contributions that people make are always appreciated. People speak with their words, but code matters more when you can build things that are helpful for the project. It just moves the state of the art forward. And if you want to just support us financially we would appreciate any anything that you guys can offer so on your site you've got slash contribute so rethink db.com slash contribute you've got some some of the things you just mentioned there and they become a contributor and even a form you can fill out i'm not really sure what this form is i guess it's just to say hello hey this is me i want to be involved somehow and you've also got the option to donate.

And you mentioned that Stripe, where you work at currently, which acquired many of the engineers, has generously agreed to donate or, I guess, match $25,000 in donations. And when I read that, I was thinking, super awesome, number one. And then number two was, how close are we? So how close are we? So how close are we? So we received a little bit north of $5,000 of foreign donations just based upon what the community has offered when we first announced the transition to the Linux Foundation. So there's still some ways to go. But we also are generally in pretty good shape because a lot of people have offered open source projects, all sorts of technical help. Services that have stepped forward include DNSimple, which is a DNS routing service.

And a number of other folks like Discourse has generously offered to host our Discourse forums for Horizon, which is a related JavaScript project that also has moved to the Linux Foundation along with RethinkDB. And we've received a lot of offers from other companies, which has been just super great. It's been really wonderful to see how the community has stepped up

to try to offer help in any way that they can. Definitely want to shout out to DNSimple because we use them as well. And knowing that they support this is super cool because we love them. Kind of curious about the future of donations. Is this just the start? What can people expect? A lot of people tend to either use Open Collective or, as you mentioned, Patreon. There's other obvious models. Is there a way that the Linux Foundation slash CNCF requires you to go about donating? What's the future of how you'll be able to sustain financially at least? There are no requirements.

We've had some companies who use RethinkDB offer to either donate a percentage of profits that they make using Rethink or provide some sort of recurring subscription. And the only thing that's really stopped us from doing that so far is just wanting to get our bearings as an open source project first. So we're excited to explore lots of options, but we also want to move steadily and carefully and be able to offer a way for people to provide financial support where we know exactly that it's going to be going to a development fund or hosting services and figure out what that looks like. But we're excited to explore a lot of different options. Great. You mentioned earlier, too, the technical steering committee.

That's how far out, like what are the plans in terms of that side of sustainability? I guess community driven part of it. Where are we at with that? We're doing well. We have a number of folks who have been thinking about how to structure it. But our goal and priority right now is on shipping 2.4 because we want to get that out the door to get a lot of the technical changes that were already underway and agreed on. And then we're going to start drafting the group that will pilot and shepherd the future for RethinkDB. Very cool.

So 2.4 roughly when? Just curious. I don't want to commit to any dates right now because this is happening a lot of people's personal time. It's sort of a ticket and cheat question, honestly. I was like, sure, yeah. sure yeah a few days later that's like i'll be speaking for people who are donating their time and like i don't want to do that gotcha all right are you are you planning like six months from now this year that's what i'm kind of asking not so much like a date but roughly it should be this year okay cool so that kind of paints a picture so get two four out sometime this year

and at that time maybe early next year start to look at the steering committee forming that. So governance getting in place. So that's a good pace for those listening and purveying what's happening here with RethinkDB. So very cool. What's some call to arms? I guess you kind of mentioned some. We don't really have to ask you directly that question, but ask you directly. C++ developers, get up there. Say what? Oh, yeah, C++ developers. I guess you kind of mentioned something we didn't really have to ask you directly, that question, but ask you directly.

C++ developers, get up there. Say what? Oh yeah, C++ developers. I'm just saying that's the big one. If you're a listener, if you're out there and you're a database or a systems person, you got C or C++ skills, it sounds like that's the best way to help out. But I'll let Mike expand and contract on that. One of the great things about open source is that open source always finds a way. Life finds a way. Jurassic Park line.

There you go. One of the reasons is that open source provides a space where people can explore the source code of a project and understand not just what it is, but how it was built. And really learn from the state of art and software development. And so the code base that's there, I mean, it's so much work that can be learned from. So much deep system software, things like the car routine engine that we built before car routine engines were features in modern programming languages, or the serializer

and the storage engine that was designed for multi-core CPUs and for solid-state drives. This stuff is all really interesting, really super cool. And if people are really good systems engineers, or are students and want to understand how these things are built, there's a lot to learn from. in the code standards that we used,

in the solutions that we provided. And open source allows people to examine it and look at it from every single angle and to think about how they can improve it, how they can see some change they can provide or add. And even just taking one issue on the project and just examining it, learning why is this an issue, not even thinking about maybe how to solve it, but just learning about

how the issue is produced, how the system operates, is just an incredible learning exercise. It really allows you to further your skills as an engineer and really learn how you can make a small change that can have drastic improvements for hundreds of thousands of people. And so if you are somebody who is a systems engineer, or you even just love a particular programming language and want to think about how to bring that functional programming experience from the database level into a language that you really care about, what should we think to be look like for the future of Node.js? What should it look like for Go or for Haskell? Your help is just deeply appreciated and an opportunity to really grow as an engineer and to help a project that really helps a lot of people. And so if someone out there is hearing this caught arms

from you, the best place to go would be slash contributor or what would the best place to go to say, hey, I'm one of these people, how can I step in? Or they just jump into issues and start planting their ideas. I would go to rethinkdb.com slash contribute to figure out how you can become a contributor. Also, just dive into an issue. GitHub is super open. So go into an issue and say, hey, I want to take a look at this. Ask for help when you need it. If you need help getting your build environment set up, there's lots of documentation on how to do that. But we're also very friendly on Slack. So if you want to just come and say, it, if you need help getting your build environment set up, there's lots of documentation on how to do that. But we're also very friendly on Slack. So if you want to just come and say, hey, I want to tackle this issue.

Can someone help me get my build environment set up? We're more than happy to help because we recognize that the more people involved, the better we'll have open source feature for the project going forward. Well, we're seven, maybe eight-ish years down the road since the beginning, I guess, right? Since you've been here since the beginning. A lot of people to think, I'm sure, along the way, but maybe specifically to like this last year, is there anybody or any particular groups you want to give a shout out to that's like, hey, without your help, we wouldn't be where we're at today.

I'm kind of putting you on the spot with that, I'm sure. But anybody you want to give a shout out to? Yes. I mean, I really want to thank the Linux Foundation and the Cloud Native Computing Foundation in particular for really stepping up and taking action. We are deeply appreciative. They removed a whole lot of work from our plates and really enabled an open source feature for the project.

Namely, Brian Cantrell and Dan Cohn. These two guys really did what was necessary to be able to make RethinkDB's future secure. And in terms of the OpenRethinkDB leadership team, to just name a few people, I mentioned them earlier, but Marshall Cottrell, Christina Keelan, Etienne Loren, Chris Abrams, Roscoe Galinsky, Ryan Paul, Sam Hughes, Yvonne Hebreken, and core members of the RethinkDB team that have stepped in to pitch their help in. And aside from that, I'm really appreciative to Stripe because they really care about RethinkDB's open source future. The fact that there are a few developer tools companies out there, we really regard each other with great respect. It's really funny because Stripe and RethinkDB both were in the same Y

Combinator class back in 2009. And so we've known each other and watched each other grow up for a while. And they've really enabled every bit of support that they can offer, including actual donations, to be able to help RethinkDB have a stable future. And all the folks in the open RethinkDB channel, of which there are hundreds, hundreds, hundreds of people, who would not let this go, even when it was hard, and have done nothing but support us and really celebrate all the work that people put into it for years. It's the community that makes this really possible. That's the only reason that Rethink

TV is alive today is because the community cares. And that gives me great confidence in its future because I know that no single person has a role in its destiny, but everyone together is building a future for it that is bright, very bright and really quite, quite moving. So I really extend just my heartfelt thanks to everybody who just has donated a dollar or even just said, I care about this and I want this to survive. So thank you. That's awesome.

We, I know from our perspective, we've kind of felt like we've been on this journey with you. As we mentioned, we had Slava on a couple of times to kind of talk about the history of rethink over its years. And it just made sense to have you back on to kind of cover this transition from a company to open source community led project and its future. So I kind of feel like we've been on this journey with you to some degree, maybe not all the years, but at least some of the years. And I can definitely echo that thanks to, because

without a thriving community behind something like this and that kind of passion, you just don't have much, right? It's, it's about the people, the code and the product that comes from it is obviously, you know, the point, but it's the people that really make everything happen. So that's awesome. Um, Mike, thanks so much, man. It was a pleasure to have you on the show finally, and, uh, to hear your side of the story and to share the future of Rethink DP. Thank you, man. Yeah. Thanks a lot. I really appreciate it. All right. Thank you for tuning into the change log this week. If you enjoyed the show, we ask for just one favor. Go rate it.

Go share it with a friend. Tell everyone you know about it. I guess that's more than one thing, but you get the picture. Thanks to our sponsors, CircleCI, Bugsnag, and Linode. Also, thanks to Fastly, our bandwidth partner. Head to fastly.com to learn more. We host everything we do on Linode cloud servers. Head to linode.com slash changelog.

Check them out. Support this show. The Changelog is hosted by myself, Adam Stachowiak, and Jared Santo. It's edited by Jonathan Youngblood. And the awesome music you've been hearing is produced by the mysterious Breakmaster Cylinder. You can find more episodes just like this at changelog.com or wherever you subscribe to podcasts. Thanks for listening. Thank you.