Software at Scale - Software at Scale 29 - Sugu Sougoumarane: CTO, PlanetScale

Episode Date: August 4, 2021

Sugu Sougoumarane is the CTO and Co-Founder of PlanetScale, a database as a service platform company. Previously, he worked at PayPal and YouTube on databases and other scalability problems, and he’s one of the creators of Vitess, a graduated CNCF database project that served as YouTube’s metadata storage for several years.Apple Podcasts | Spotify | Google PodcastsWe discuss his early days at PayPal and YouTube, their initial scalability problems, the technology that helped them scale out their products, the motivation behind Vitess, its evolution into PlanetScale, and some of the innovative ideas coming out of PlanetScale today.Highlights5:00 - Being interviewed, and hired by Elon Musk at X.com. Working with some of the PayPal mafia like Max Levchin and Reid Hoffman.9:00 - Solving PayPal’s unbalanced books via a better Activity Report15:00 - PayPal’s Oracle database and the initial idea of sharding. 20:00 - Early YouTube architecture and culture, and the consequences of explosive growth.24:00 - Sharding YouTube’s database. 32:00 - The story behind Vitess. It all started with a spreadsheet.40:00 - How a user with 250,000 auto-generated videos caused YouTube to go down. How Vitess fixed that, and the implications of a SQL proxy.45:00 - The motivation behind keeping Vitess open source, and interest from other companies dealing with similar problems. The motivation behind going from Vitess to PlanetScale53:00 - How PlanetScale can improve on some traditional relational database developer experience. How NoSQL was never actually about getting rid of SQL and was more about skipping traditionally rigid schemas. MongoDB’s support for SQL, and PlanetScale’s approach to solving schema changes.58:00 - Technical innovations coming out of PlanetScale.1:05:00 - Running databases in containers and the implicationsTranscriptUtsav Shah: [00:15] Hey, welcome to another episode of the Software at Scale podcast. Before we get started, I would just like to ask a quick favor from any listeners who have been enjoying the show. Please leave a review on Apple Podcasts. That will help me out a lot. Thank you.Joining me today is Sugu, the CTO and Co-Founder of PlanetScale, a managed database as a service platform company. Previously he worked at YouTube for a really long time from 2006 to 2018 on scalability and other things, including Vitess and Open Source CNCF project, and before that he was at PayPal. Welcome.Sugu Sougoumarane: [00:53] Thank you. Glad to be here.Utsav: [00:56] Could you maybe tell me a little bit of, you know, when you got interested in databases and MySQL? Cause it seems, at least from my understanding you've been interested in distributed systems, my sequel databases for like a really long time.Sugu: [01:11] Yeah. So, I would say, the first serious involvement with databases started when I joined Informix in 1993. That was the time when there were three huge database companies, Informix versus Cyber's versus Oracle. And I guess eventually Oracle won the war, but that was the first time when I came in touch with databases. I specifically did not work with the engine itself, but I worked on a development tool called 4GL, which was popular those days; not there anymore. So that was actually my first introduction to databases.Utsav: [02:08] Okay, and I guess you took this skillset or this interest when you moved to YouTube from PayPal. So, what were you doing at PayPal initially and what made you decide that you want to work at YouTube?Sugu: [02:25] So PayPal, I guess when I moved from Informix to PayPal it was mainly because this was around 2000. This was the time when the internet boom was happening, and it was clearly obvious that Informix had fallen behind it; like they were still trying to make a client-server work and that was pretty much a dying technology. That's when I kind of decided to make a change. It was somewhat of a career change when you realize that the technology, you're working on is not going to be around much longer. I don't know if other people saw it then, but it was pretty obvious to me. So that's the time when I decided that, you know. I kind of made a decision to start back from the beginning because you'd be surprised in year 99, not many people knew how a website worked. Now, it's probably common knowledge, but I had no idea how a website worked. 03:36 inaudible requests come in, you know, what servers do what, and those things. So, it was all kind of unknown science to me. So, for me, it was like I'm starting a new career; like even within software engineering, each of these things sounds like a different career, you know, there are choices. It is as if I stopped being software engineer A and now, I'm going to become software engineer B. So, the only skills I could carry over were my ability to write programs. Beyond that, I knew nothing. And I had to learn all those things from scratch again at PayPal. By the way, I did not join PayPal directly. I actually joined X.com, which was founded by Elon. So, Elon actually hired me and then later, X.com and PayPal merged, and that's how we became PayPal. Utsav: [04:41] I think if you can tell us a little bit about Elon Musk, I feel like this podcast will just shoot up in popularity because of anything to do with Elon Musk now.Sugu: [04:50] Alright. There is this story I think I've said that it's not really commonly known, but at least, I'd say I might've said that in one place, which is that, actually, I failed the interview. I went for the interview and Elon thought I was not worthy. They decided to pass on me. But then I was so excited when I found out, at that time I didn't know who he was, this was the year 2000. I mean he had just sold his previous company, but there were many other people who had done that. But when I found out what he was building, I got so excited that after I heard that he passed on me, I sent him an email saying why he should hire me. And somehow that email convinced him to bring me back for a second round. And then I got hired and the rest is history.Utsav: [05:55] So interesting. What is it like working with him? Did you work closely with him? How big was X.com at that time?Sugu: [06:02] It was very small. I think we were about 15 engineers, 12, 10, 15 engineers. I think the total was 50, but most of the other people were customer support type of thing because it was a bank, right, so there was a lot of customer support. Yeah, at that time, it's surprising many of those people that I worked with, not just Elon, are now celebrities like Jeremy Stoppelman who was like sitting right next to me. I remember asking, Jeremy Stoppelman is the CEO of Yelp in case you want to look it up. So, I remember I went there and he was the guy next to me. So, I said, "Hey, you know where's the bathroom?" [06:53] "Where do you get coffee?", and sometimes you say, "Yeah, it's there", sometimes you say, "I don't know". And finally, I asked him, "So how long have you been here?" "Oh, I joined yesterday". I literally joined the day after Jeremy; I got hired. Yeah, so those were good times. So, Elon himself, by the way, like SpaceX, he used to talk about that [07:21] as his dream, you know, going into space. And he's really inspiring when he talks about it; like he starts talking and everybody just shuts up and listens because he used to talk about him being broke. There was a time when he was broke, where he said he didn't have money for food. Did you know that Elon was broke once in a while?Utsav: [07:49] I feel like I've read about it or seen some YouTube video where; now there are so many YouTube videos where they take quotes of Elon and it's like inspirational stuff. I think I've seen some of those. That's so fascinating. And yeah, now that clearly became the PayPal mafia with so many people making so many successful companies and stuff. Like, did you work closely with any of the others, like Jeremy Stoppelman? Sugu: [08:15] Yeah. There were lots of them. So, I actually worked with Max a lot more right after the merger. So Max, definitely, Reid Hoffman, although I don't think I interacted much with Reid Hoffman, there is Roelof, for sure, Steve 'Chairman' Chen, obviously that's how I ended up at YouTube. My memory fails me, but if you look at the mafia, yeah, I have worked with all of them.Utsav: [08:52] Did you work on the Igor program, like the fraud detection stuff, which I've read about publicly?Sugu: [08:58] Oh, no, I did not. I worked on some fraud systems, but not Igor specifically. I am trying to remember who worked on it. It was probably Russel Simmons.Utsav: [09:14] So what did you do at PayPal? I am curious.Sugu: [09:18] So the thing is, this is a good lesson, the struggle I was having initially at PayPal was recognition. There were a couple of things. I was kind of an unknown there, nobody knew who I was, and I was part of the merger, right? I came from the X.com PayPal merger, and there was a lot of back and forth about which technology to use. Eventually, we decided to continue with the original PayPal technology and stuff using the X.com technology. So, the struggle I had was, how do I prove myself to be worthy. So, in software engineering, you have to show that you are good and then they give you good projects. You don't get to choose the projects because you get assigned projects, right? So at least that's how the situation was then. And by the fact that I was an unknown, all the good work was going to the people who were already recognized for how good they were. So, there was this one tool called The Activity Report, which basically tallied the total of all transactions on a daily basis, and say this is the total in this category and credit card category, ACH category, and that kind of stuff, and used to report those numbers. But the auditors came in and they said, you know, with these transaction numbers that this report produces, you need to tally them against your bank accounts; against the total balances that users have. The first time the auditors came, there were like a few million dollars off, you know?And they're asking like, where is all this money? I don't know if you remember when PayPal initially was founded, the business model was to make money off of the float. When we transfer money, there’s a float. We'll basically invest that money and whatever money we make out of that. So, people are like, 'oh, it must be the float', you know, that difference was because of the float and the auditors were like, 'no, that explanation is not sufficient. You need to show the numbers and know that they add up and that they tally’. And so, finally, we figured out where all it is on the tally. But the problem was that PayPal was software written by engineers and how do you create transactions? You insert a row in the transaction table and commit, you insert the row in the transaction table, update a user's balance, and then do a commit, right?So that's how you do it. But sometimes the transactions go pending, in which case we don't update the balance. Sometimes we complete the transaction, and then after that, you update the balance. And guess what happens when you start doing these things. There's this thing called the software bugs that creep up. And so, every once in a while, after they release the transactions wouldn't add up to the user's balance. And sometimes they won't tally with what's actually in the bank accounts. And so, every few days, the activity report is used to produce discrepancies. And because the software was written inorganically, there was no easy way to find out like you have, you know, million transactions. How do you figure out which one went wrong? The numbers don't add up to the totals here. And every time this used to happen an engineer was assigned to go and troubleshoot.It was the most hated job because it's like finding a needle in a haystack. Like you'd say, 'Ah, activity report conflict. Now I got assigned to do this. Why is my life so bad?' You know? So that was the situation. And then, at one point in time, I said, you know what? I'll own this problem. I'd say, 'Just give me this problem, forget it. I will investigate this every time it happens. And I will also fix the software to make it easier.' So, I took on this and I rewrote the entire program, such that, to the extent that, at the end of its maturity, it would find out exactly, previously it was all totals, that at the end, once it got fully evolved, it will say 'this particular account has this discrepancy and most likely this is the transaction that caused it'.So, it will produce a report or a list of conflicts with a very clear clue about where the problem is, and then you spend five minutes and you know what went wrong. It would find software bugs in it because sometimes somebody forgot to update the balance. It'll say, 'Hey, you created a transaction and didn't update the balance, you know, so here go fix it'. So that it used to do. And eventually what had happened was because it was so solid, it kind of became foundational for PayPal. Like people used to ask for reports about, you know, transactions, and they would say, 'How do you know this is correct?' and we'd say, 'Oh, I validated it against a not-so-good activity report'. 'Okay, then it's good. I trust these numbers. So that kind of stuff. So, making that software and taking that pain away from all the other engineers, got me accepted as one of the cool engineers. Utsav: [15:12] So interesting. How big was the engineering group at that time? Sugu: [15:19] So the PayPal by the X.com site, there were about 15 and on PayPal's side there were also 15. So, we were about 30 to 40 engineers.Utsav: [15:28] And once you got accepted by these people, what were your next projects, I guess? Sugu: [15:33] So having gotten accepted, I became the kind of person who 'd say, 'If you give a problem to Sugo, you know, you consider it a salty kind of thing'. So, which means that anytime PayPal had a struggle where you say, 'Oh, you need the top brains on this', I used to get called. And the biggest problems later were all about scalability. So, I kind of moved into the PayPal architecture team, and used to solve all problems related to scalability. And the scalability was always the Oracle database, because it was a single instance database, and every time, the only way we could scale it was vertically. So, the idea of sharding was actually born there for us. We needed to shard this database. But it eventually succeeded only on YouTube. We didn't succeed at sharding at PayPal itself, but we kind of knew how to do it. We kind of learned; we went through the learning process when we were at PayPal about how to shard a database, but by then the [16:46] had taken place. There was a drift and lots of things took place after that.Utsav: [16:50] That's so interesting. And is that where you met Mike?  You were discussing some of these stories, [16:55]Sugu: [16:56] Yeah, Mike was reporting to me. So, there was Mike, there was another person called Matt Rizzo. They were the top engineers, you know, one of the top engineers at PayPal. We still use some terms that only Mike and I would understand. We'd say, 'Oh, this is APCD right?' and then we'd laugh. Nobody else, I would say, would understand what that means. [17:28] ACH process completed deposits, Bash tool that he wrote. You probably know that Mike is really, really good with Bash.Utsav: [17:39] Yeah. He knows his Bash.Sugu: [17:42] He knows his batch. That was kind of my fault or doing because there was a time when I just told him a problem, you own this problem, you have to solve it. And he'd say, 'Do you think Bash is a good idea? Sounds cool'. I said, 'If you think it's a good idea, it's a good idea. It's up to you. It's your decision'. So, he went and he is still scarred by the fact that I let him do it and said, 'Why did you let me, why didn't you tell me it was a mistake?' Utsav: [18:15] I can imagine this very horrible process at PayPal, like adding and subtracting money and it's all written in bash.Sugu: [18:23] That wasn't written in Bash, but there were some tools that we wrote in Bash. Yeah, the money part was all SQL, SQL+ actually. [18:36] I guess, that could be called the Kubernetes of those days. You were writing an orchestration tool and that we wrote it in bash.Utsav: [18:46] Okay, that's so interesting. And then you're saying that the idea of sharding was born at PayPal, or like the idea of sharding or my SQL database, like you figured out how to do it. Why would you say you did not complete it just because there was not enough time or, at that point did you decide to leave and go to YouTube?Sugu: [19:04] There were a lot of reasons, some of which were changes in management at that time. There were the PayPal engineers and there were the eBay management that had acquired us and the relationships were very strained. So, it was kind of a very difficult thing. So, there was no coordinated, what do you call it effort? There was no focused, coordinated effort towards solving it. It was all over the map. There were a lot of committees spending a lot of time in conference rooms, discussing things, not coming to conclusions, you know, that kind of stuff where there were just 'too many cooks in the kitchen' type of thing. I think the core first kind of figured out how this needed to be done, but we were not able to, you know, push that idea forward, but you finally made that happen at YouTube. So, which proved, you know, it kind of absolved us of you know what you're doing.Utsav: [20:20] So, yeah, maybe you can talk about the earlier days at YouTube. What was YouTube like in 2006? How many people were there? Was this like pre-acquisition, post-acquisition? I don't know the exact -Sugu: [20:29] -just around the time of acquisition? So actually, around the time when YouTube was founded, I actually went to India to open the PayPal India office. But I used to visit YouTube every time I visited. I came to the US and I used to come pretty often. So, I've been closely following, at some point in time, Steve, and Matt kept hounding me saying, 'What the hell are you doing there? This thing is catching fire. You need to come and join us right now'. So finally, I came back and joined them. But yeah, the YouTube culture was kind of a carryover from PayPal. It was very similar, lots of heroics, you know, lots of, like each person was an impact player. Each person made a huge difference. Like one person owns all of the search, that kind of stuff. One person owned the entire browsing experience. There were only like 10 engineers. So, everybody owned a very important piece of the stack.Utsav: [21:50] Okay, and what were some of the early problems that you ran into? Or like, I guess what were the fires that you were fixing as soon as you started?Sugu: [21:59] So the one big thing I remember, there was this rule at YouTube where you join, you have to commit on your first date.Utsav: [22:11] Okay.Sugu: [22:12] I broke that rule. I said, I'm not going to commit on the first day. But I spent time learning the code and produced a huge PR, a huge pull request, that we wrote an entire module because it was badly written. So, the biggest problem I found, I felt at least at that time, was because the code was organically grown and incrementally improved, it was going to run out of steam as we added more features. So, I felt that we needed, you know, to clean it up so that you can put a better foundation underneath so that you can add more features to it. So that's basically the first work that I did. But then later after that, the bigger problem also became scalability because we were single, we were uncharted. There was only one database that was called Main and we had to do the re-sharding. So that was the biggest challenge that we solved. We solved the re-sharding [23:22] test later, and adopted that sharding method. But when we first started, it was at the app layer.Utsav: [23:33] Okay, at what scale did you have to start thinking about sharding? How many engineers were there? How much data was going in and what was being stored in this database? Was this just everything of YouTube’s metadata in this one large database?Sugu: [23:48] Yes, all the metadata, not the videos themselves because the videos themselves were video files and they were distributed to CDN, but all other metadata, like video title, likes, dislikes, everything else was stored in my SQL database. It was running out of capacity very soon, and the hardware wouldn't have kept up. Vertical scaling wouldn't have taken us much further. And that's when we said, 'Okay, we need to shard this'. We made a very tough call, which is that we are going to break cross short, transactional integrity. It was actually not as big a deal as we thought it was going to be because of the nature of the app, maybe because users just mostly work with their profile. There were a few things about comments because I posted a comment to you, so where do you put that comment? and some little bit of issues around those types of things, but otherwise, there was still quite a bit of rewriting because an application that asses an unsharded database and makes it [25:10] is non-trivial no matter how you put it. We had to build a layer under the app and write everything, change everything to go through that layer, follow certain rules. And then we changed that layer to accommodate sharding. That's basically how we solved the problem.Utsav: [25:27] Yeah. That layer has to basically decide which shard to go to. So, you were just running out of physical disc space on that My SQL box which seemed like the problem?Sugu: [25:37] I'm trying to remember, whether it was a disc, it might've been a CPU actually.Utsav: [25:42] Okay.Sugu: [25:44] I think it was the CPU. It's either CPU or memory. I think we could have a disc. I'm trying to remember if it's CPU, memory or IOPS, most likely memory, because of working set size issues. I could be wrong.Utsav: [26:14] I'm just curious, you know, do you know how many videos you had in YouTube at that time when you started running, [26:20]Sugu: [26:21] I remember celebrating. Let's see it was either a hundred million or 1 billion, I don't know.Utsav: [26:31] Okay, 1 billion videos and that's when you start running out of space? [26:35] total video, that just shows you the part of like my SQL and like [26:40]Sugu: [26:42] Yeah, it's amazing that I don't know the difference between a hundred million and 1 billion, you know, but something, yeah.Utsav: [26:52] Sounds like [26:53] I don't even know how many videos will be on YouTube now. I can't even imagine.Sugu: [26:59] The reason why it works is that most videos are inactive, right? You just insert a row and move on, and so it is more a question of what is a working set, right? How many of those videos are popular? They come and live-in memory and we had a Memcached layer also in the front, and so it was more a question of how many popular videos people are trying to hit and can all that fit in memory?Utsav: [27:31] Okay, that makes a lot of sense. Then you could put a caching layer in front so that you don't hit the database for all of these popular videos. That's so interesting. And, given all of this, it makes sense to me that you can shard this because videos, you can totally imagine, it's relatively easy to shard, it sounds like. But yeah, what do you think about resharding? That seems like, how do you solve recharging? That sounds like a very interesting problem. And maybe just to add some color to what I think resharding means, it's like you have a data on like my SQL box A and you want to move some of it to my SQL box B or like split from one database to two and two to four. And you have to somehow live transfer data from the database, like, how do you solve that?Sugu: [28:19] So, there's the principle and there is a mechanic, right? Typically, the mechanic is actually more, kind of straightforward, which is typically when you start sharding, the first thing you do is actually pull tables out. Like if there are 300 tables, you'll say, you know, these 20 tables can live in a separate database. It's not really called sharding or resharding cause you're not sharding, you're splitting. So, the first thing people usually do is split a database into smaller parts because then each one can grow on its own. The sharding aspect comes when a few sets of tables actually themselves become so big that they cannot fit in a database. So that's when you shard and when you make the sharding decision, what you do is, the tables that are related, that are together, you shard them the same way.Like in YouTube's case, when we did the sharding, we kept the users and their videos together in the same shard. So that was actually a conscious decision that users and videos stay together. There is actually a sharding model that allows you to represent that as a relationship. Now that we test actually exposes, which is basically there's some theory crafting behind there. But from a common-sense perspective, you can always use about 9 users and their videos, being together. So that's the mechanic part.On the principal part actually, the way you think about sharding is basically beyond the scale, you have to actually rethink your application as being fully distributed with independent operators working by themselves, working on their own. That means that some features you often have to forego for the sake of scalability. In other words, I mean, like crazy features, like what is the real time value of the total number of videos in the system? I want that up to the millisecond. That type of question is unreasonable for a sharding system. You could probably do a select count star, against a small database and can actually do it. But if you are having tens of thousands of servers answering that question becomes harder, right? And then at that time we are to start making trade-offs like, is it really important that you have to know it up to the, like, what if it is up to the minute? Is that good enough?So those kinds of traders here make at a high level and most people do because you cannot go to that scale unless you make these tradeoffs. But once you make these trade-offs, sharding becomes actually very natural, because the one thing that you make, for example, if you are going to have the entire world of your customer, you are going to have, you know, 7 billion rows in a user account. And sometimes many of them create multiple accounts. So, you are going to have billions of rows. You cannot build features that put them all together. You have to build features that keep them separate. And as soon as your features follow that pattern, then sharding also kind of drives that same decision. So, it’s kind of more or less becomes natural.Utsav: [32:21] Okay, that makes sense to me, and it seems like you said that you have a sharding solution that predated the tests. So maybe you can talk about the motivation behind the test, like, and maybe the year that this was in like, well, if you have any numbers on, you know, at what point did you seem [32:36]Sugu: [32:36] [32:37] I'm finding myself surprised that there are details I should know that I don't know, like the number of videos we had when we were sharding, right? It was 2007, 2007 is, actually wow,15,14 years ago. So, yeah it has been many years. So, in 2007 was I think when we did our first sharding, if I remember correctly. Maybe 2008? Around 2007 or 2008, one of those few years was the first time when we did sharding. But the test was not born because we needed to do sharding, obviously, because it was already sharded it was born because we couldn't keep up with the number of outages that the database was causing. That was actually why it was born. It was quite obvious that as the system was scaling, in spite of being sharded, there were a large number of things broken within the system that needed fixing for it to work.And the system overall, when I say system meets end to end the entire system, and that includes the engineers and their development process. So that's how big the problem was, but we didn't solve the entire problem. We said from a database perspective, what can we do to improve things, right? So that's kind of the scope of the problem. And more specifically Mike actually, took off, and I think, spent some time in, Dana Street coffee at Mountain View and came up with the spreadsheet, where he described, he kind of listed every outage that we have had and what solution and how did we solve it? Is there, what would be the best remedy to avoid such an outage? So, when we actually then sat down and studied that spreadsheet, it was obvious that a new proxy layer at any build and that is how we test was born.And the whole idea is to protect the database from the developer. For example, they were like, at that time 200 developers, if you look at it right, the developers, don't intentionally write [35:10] and a developer doesn't write a bad query all the time. But if a developer wrote a bad query, only once a year with 200 developers, 50 weeks a year, you do the math. How often do we see outages in the database? Almost every day. That's what it amounted to; we were seeing outages almost every day. And very often they were actually put 'quote and quote', bad queries coming from that one user role that they will never repeat, but they've fulfilled their quota for the year, you know? And so, what we did was we found common patterns where that would cause data.So, the big change that we made was if I wrote a bad query, I have to pay the price for it, not the entire team, right? Now today, if I wrote a bad query, that query runs on my SQL, it takes on the entire database. So, we wrote with tests in such a way that if you wrote a bad query, your query would fail. So, in other words, what we used to look at is how long is the query running? If the query is running too long, we kill it. If the query is fetching too many roles, we return an error. So those kinds of defenses we added in with tests early on and that pretty much took it a long way. The other big feature that we deployed was connection pooling, which my SQL was very bad at. The new SQL is slightly better. It's still not as good. So, the connection pooling feature was a lifesaver for them too.Utsav: [37:00] That makes a lot of sense to me. And maybe you can tell us a little bit about why my SQL didn't have the capability of defending itself. So, you might imagine like, you know, just from somebody who has no my SQL experience, they can just be like a [inaudible 37:16].Sugu: [37:17] It's very simple. The reason is because my SQL has, what do you call, claimed itself to be a relational database. And the relational database is required, you know, to not fail. So, we at YouTube could be opinionated about what query should work and what query should not work. That freedom my SQL doesn't have. Every query that is given to it, it has to return the results, right? So that's the rule that you had to, to be qualified as a relational database, you had to fulfill that pool and that pool was its curse.Utsav: [38:01] Okay, but you could configure maybe like a session timer or something, or like a query timer, or would that lose deference which is not good enough basically.Sugu: [38:09] They actually added those features much later, but they didn't. Yeah. The newer, my SQLs now do have those features, but I don't know how- but they are all behind. They are not as usable because you have to set it in a query parameter, you have to set it in your session, you know, so it's not as, like by default you just install my SQL, and start running it. You won't have this property, you have to configure it, which is another hurdle to cross. Whereas in Vitess you installed the test, you'll start failing, you know, people will complain like, we see the opposite problem, right? Oh, [38:53] My SQL, why is the test failing it? Because you are trying to pitch a million rows. If you want to do 10 million rows, you have to tell the tests that you want to get 10 million rows and then it will give you them. But if you just send a query, it won't give you a million rows.Utsav: [39:10] Yeah. What do you think was like the one really important? Which as you said, there's connection pooling, it's also limited, bad where he's like, how did it figure out that a query was not one, maybe heuristic is it's turning too many rows and it would just fail fast. What were some other key problems?Sugu: [39:27] Oh, there were, we added so many, the coolest one was, when engineers wrote code, they are, like me too, I'm guilty of the same thing. You think how many videos could a user possibly upload? And we were thinking manually in the [39:51]. For me to upload a video, I have to go make it, edit it and then upload it. You know, we will have, you know, 2000 videos where we'll be a number and, you know, look at how old YouTube was. It was two years old. How many videos can you produce in two years? So, 2000 videos. So, selecting all videos of users, not a problem, right? So, we never put a limit on how many videos we fetched. Then we selected a video by using, and then there was one user who ran a bot. I think there's no other way. Well now there are accounts that have billions of videos, but at that time that user had 250,000 videos.And that wasn't a problem per se, because that I couldn't have been around, but it became a problem when that account got high, got listed on a YouTube stream page. Yeah. The index page, right. The landing page. So, which means that every person that went to youtube.com, you should a query that pool $250,000, because there was, so the, one of the changes we made was that, any query that, that's why we added the limit, plus like, where you say, if you send an aquari with no limit class, we will put a limit. And if that limit is exceeded, we'll just return your arrow thing that you're trying to fish to NATO. So, there was one protection, but the cooler protection that we did was, there are some queries that are just inherently expensive, right? Like this one, if this, I think in this case, I think it might've been a select count star.If it's a select count, it doesn't fish to 50,000 rows, but it scans to 50,000 rows. So, the, the feature that we did was if a query got spam, like because of the fact that it's coming from the front page, which means that my school is receiving like a very high QPS at the same period in that case, what we do is we check if the query is already learning from some other request, if it is, if there are 10 other requests of the same query, they are not sent to my SQL. We wait for them to return and return that same dessert to all those requests.Utsav: [42:21] Okay. Would that be problematic from a data consistency perspective? Probably not. Yeah,Sugu: [42:26] It is actually, yes, it is but it is no worse than eventual consistency. Right. So, if you are reading, if you are reading from a replica, no problem. Right. If you are doing something transactional. So, the rule that we had at YouTube was if you're doing something transactional, so there's also another reason like the rule is if you read a rope, asse that rule is already stayed. Why? Because as soon as you read the rule, somebody could have gone and changed. So that stainless guarantee kind of carried over into this will also is that when you do a select, when you do a select of the query, it may be slightly outdated because one thing is you could be reading from the previous period that is running, or sometimes it could be going to a replica.So it wasn't, it was something that nobody even noticed that we had made this change, but it's like, we've never had an outage later to, you know, very spam after we did that feature. Yeah. But more and more, more importantly, the rule that we followed was if you want to make sure that the road doesn't change after you read it, you have to select for update. So, if you plan to update a row and update based on the value that you have read, you have to select for update. So that's a rule that all engineers followed.Utsav: [43:56] Yeah. It makes sense. And it's better to just be explicit about something like that.Sugu: [44:00] Yeah. Yeah. Because of the fact that the, the, my SQL NVCC model tells you, right. The MVCC, the consistently it's called consistently, that's actually the actual practical Lim is something else. but that model basically tells you that the role that my simple serves to you is technically obsolete. Utsav: [44:24] Okay, and then it just makes sense to me. And you’ll open source with us. I remember hearing the reason is you never wanted to write another way to test. So, you decided to open source it from day one. Can you maybe tell me a little bit about, you know, the motivations behind making it a product? So, it's a great piece of software, which I used to buy YouTube from my understanding for like a really, really long time. At what point did you decide, you know, we should actually make this available for other companies to use. And what is the transformation of the liquid test for PlanetScale? Sugu: [44:58] So even before PlanetScale, even as early as 2014, 2015, we felt that we should start promoting the test outside of YouTube. And one of the reasons was, it wasn't a very serious thing, but it was kind of a secondary motivation. We felt that even for YouTube, for the test to be credible, even within YouTube, this product needs to be used by other people beyond YouTube. In other words, you know, for a company to trust a product, a lot of other people must be using it. That was kind of the motivation that made us think that we should encourage other companies to also use, [45:46]. And actually, Flip card was the first one. I think it was 2015 or 16 that they approached us and they said, hey, it looks like this will perfectly solve the problem that we are trying to solve, and they were the first adopters of the test. Utsav: [46:05] Okay, so that makes sense. And that's like it was open source and you felt that other companies should use it 2015, it sounded like Flip card was interested, but then what is the transformation of from then to, you know, let's make it a product that people can buy because this it's an open-source technology that, how do you go from that to product?Sugu: [46:28] Yeah. So, I can talk about how PlanetScale came about, right? So, the other thing that started happening over time as the test started evolving as a project companies like Flip card started relying on it, Slack started using it, Square started using it. And they were all scared of using the test. Why, because this, this is basically defining technology. It starts something, it's basically a lifetime commitment type of change. And the question in their mind is what, like, what's the guarantee that, like what YouTube is interested in making sure that, you know, it will continue to work for slack, right? Why, like they were, a large number of companies were worried about that part, where, you know, you do focus his videos by like, how can we trust the technology?How can we trust the future of our company, right on technology? What if you abandon the project and that kind of stuff? So, overtime, I could sense that hesitancy growing among people. And this was one of the contributing factors to us starting planet scale, where we say, we are going to start a company at a company that is going to make a commitment, you know, to stand behind this open-source project. So, when we made that announcement that we are starting the company, there was a sigh of relief throughout the test community. You know, finally, I can-Utsav: [48:07] -depend on this really important.Sugu: [48:08] I can depend on this project. It's not like it is here to stay. There is a company who is going to make sure that this project stays healthy.Utsav: [48:18] That makes sense. Yeah.Sugu: [48:21] There were other factors also, but this was one of the, definitely one of the major contributing factors towards making. So, I think at the end of the day, this is something, this is generally a problem in open source. You know, that, you say open source is free, but there is an economy behind it because the engineer's time is not free. And for me to spend time on an open source, I have to be paid money because, you know, I have my family to take care of, you know, I need money to live. And so, I'm expecting engineers to contribute to open source in their free time. it can work for smaller projects, you know, but once the project becomes beyond certain sites, it is not a part-time, it cannot be a part-time contribution. It is a full-time contribution and the engineer has to be paid money for it. And so there has to be some kind of economy behind that.Utsav: [49:32] That makes a lot of sense. And I think it also tells me that the problem that slack and square and these other companies are facing was just so large that they were, that they wanted to use the project, given all of these issues with, you know, we don't know the future of this project, but there was no other solution for them given that existing, my SQL stack and their hyper-growth and the kind of problems that they'd be dealing with.Sugu: [49:59] Yeah, that's right. So that is something I still wonder about, right. If, if not retested, what could have, is that an ultimate, you know, solution, the alternative doesn't look as good either because the alternative would be a closed source product. Like, you know, that scares you, that even more scary, because what if the company COVID out of business, right. At least, in the case of Vitess the source code is there. If planet scale goes away, slack can employ engineers to take over that source code and continue to operate so that confidence is there. So that is one big advantage of something being open source, gives higher confidence that, you know, in the worst-case scenario, they have a way to move forward.Utsav: [50:53] And when you, when you finally decided to make a company and a product, like, did you, what are some things you've learned along the way that, you know, you need to have in with tests for more people to be able to use it at a first impression to me, it's like, there are not millions of companies running into these scale issues, but like, what are some interesting things that you've learned along the way? Yeah.Sugu: [51:14] All I can say is I am still learning. Yeah, every time I come there, I realize, oh, you know what I thought was important. It's not really that important. like somebody asked me, if you are agonizing so much about data integrity, why is Mongo DB so popular? Right? I mean, Mongo DB is pretty solid now, as far as I can tell, but it was not solid when it became popular. Right. They made it solid after it became popular. So, one good thing. If he doesn't, did not, for the longest time did not care about data integrity, but people still flock to it. So, there is always a trade- off in what is important. And you have to find out what that is and you're to basically meet people where they are.And in the case of the test, it’s actually usability, approachability, and usability. That is a bigger hurdle to cross before you cannot up your tests. And that is a big, much bigger hurdle than, you know, the data integrity guarantees that have people looking for. So those are for example, one lesson, where you have to, so the one thing we test is we used to listen to, we used to, we have our ears open. You're constantly listening to people, giving us feedback about what is not working for them and fixing those problems. It turns out that is insufficient. Who are we not listening to? We are not listening to the person, the quiet person that drives over the weekend, doesn't work for them and quietly walks away. Right? That's that voice we never heard. We only heard the voice of somebody who tried to use the test and like went past the problem of not being able to use it, got to use it, found a missing feature and is asking for it. So, we've been focusing on very specific features about how to make the test, but we completely forgot about, you know, how to make it more approachable. Right. So those are the problems we are now solving at PlanetScale. Okay.Utsav: [53:41] That makes a lot of sense. Yeah. Maybe you can tell us about one usability feature that you feel was extremely important that your live build.Sugu: [53:49] The biggest one for example, is a schema deployment. Subconsciously every developer will cringe. When you say, use a database and manage your schema, use a database, they would like, but as soon as you say schema, almost every developer cringes, because, oh my God, I have to deal with that. Bureaucracy, like send it to for approval. DBA is going to detect it. our schema is going to break something. Some app is going to break all those headaches I have to deal with. So, what we have done at planet scale is give you an experience very similar to what you would do as a developer with your source code. You do the same thing with your database. You can take your database ratchet, applied, schema changes, tested, and then have a workflow to much it back into production and very natural. And it looks almost similar to how you develop your source code, right? So, it's a very natural workflow and it applies to the database and you don't have to deal with, you know, that whole bureaucracy, it's all part of this nice, nice workflow. It handles conflicts. If multiple people want to change the schema at the same time, it pipelines them the correct way. And if they cannot go together, we will flag it for you. So those are really, really nice feature. And that seems to be really going well with the developers. Utsav: [55:28] Interesting. Yeah. And I think it speaks to, you know, why databases like Mongo DBS are popular?Sugu: [55:33] Yeah. No schema is not a no SQL that made Mongo DBS when it's the no schema part,Utsav: [55:43] Being able to just tweak, add one more field, not be blocked on like a team to make sure not exactly worry about back-filling and all of that. It is a game changer and maybe a little ashamed to admit actually, like I use Mongo DB at this job and it's not half bad. It's pretty good.Sugu: [56:00] Yeah. Yeah. Now they are trying to bring a skill back to Mongo DB and people like it. Right. So, the real problem was actually schemas, the fact that you can't just add something and move on. It's so hard to do it in a database. Utsav: [56:21] Yeah, and maybe you can tell me today, like, what's the difference between VTS open source and PlanetScale? Like what is the extra stuff?Sugu: [56:29] They are very orthogonal, right? Because what we are building in PlanetScale is a beautiful developer experience. And, what we test is giving you is the other part that, most people that have a good developer experience MIS, which is, a solid, scalable database in the backend, like we could build this developer experience on top of stock, my SQL, but people are hesitating to adopt that because they know that it won't scale. At some point of time, I'm going to hit a limit and I'm going to spend the 90% of management energy trying to figure out how to scale this. That's what companies experience when that problem is taken away from you. Right. You get a good developer experience, but when it comes to scaling with us, have you covered?Utsav: [57:23] Yeah. That makes a lot of sense. And does that mean that as a developer, let's say I want to start a startup, the model. I have a little bit of experience with my SQL. Can I just start with PlanetScale on day one?Sugu: [57:35] What would I get? That's exactly what [inaudible]. Yeah. You basically spend almost no time configuring your database. You start, you said, just go to planets, click, like you click a button, you instantly get a database and then you just start developing on it. So, zero bad years, which is what you want in a startup.Utsav: [58:02] And, and the experience I get is just stock my sequel to begin with, except for it would have all of these things like automatic limits on queries. And it would let me shard as soon as I become too big,Sugu: [58:15] As soon as yeah, exactly. But interesting. Yeah. So basically, what they have to do is make sure that, you know, kind of guide you towards good programming practices. You know what I mean? If you're just running unbounded queries, it's going to tell you, this is not good. So had bounced your [inaudible 58:36], that kind of stuff. So, we can be opinionated that way.Utsav: [56:37] So I'm curious about, you know, all of the technical innovation that I see your Twitter feed and YouTube talk about, you know, you're a handler, I'll have like an automatic benchmarking tool. You're like optimizing, protocol handling and letting go. What are some of the keys, like other innovations that you all are doing at that PlanetScale?Sugu: [58:56] So there is one, I don't know if you've seen the blog series about consensus, generalized consensus. Okay. So, I think that's a decent idea. I feel like it's a decent innovation. What we are doing is there's Paxus raft. You know, those in my eyes are rigid algorithms and they have rigid assumptions about how nodes should talk to each other. And those algorithms were opted for industry best practice. But if you flip it around, what problems are they solving, right? You identify the problem. And then you start with the problem first and come in and say, this is the problem I want to solve. What system would you build? I don't think a raft or Paxus would be the systems we would have been. Right. And what problems are we trying to solve?We are trying to solve the problem of durability, right? I have a distributed system and I say, save this data. And the system says, I have saved your data. And the guarantee that I want is that the system doesn't lose my data, essentially. That is the problem that all these consensus systems solve. But then when I come in here, I'm in a cloud environment, I'm on AWS. I have the ones that have regions, right. I can say for me, durability means my data is across two Lords. Or my, my notion of UWP is data is, at least in one other zone or one of the regions. Right? So, if you specify a system that can make sure that whenever you ask, ask people for the right data, it makes sure that the data has reached all those nodes or regions. And then it gives you, and then later, if there are failures, it recovers based on the fact that the data is elsewhere, right? So that is what the thing says. So, if you look at it, top-down, you come up with a very different approach to the problem where, in which case raft and Paxus are just one way to solve it., so what I have explained is what is the generic approach behind solving the problem of durability, how Paxus and raft solve it, but how we can build other systems that are more effective at more accurately meeting the driving requirements that are coming from the business.Utsav: [1:01:39] This reminds me of semi sync replication and like my SQL.Sugu: [1:01:44] Exactly. So, you could build a consensus system using my SQL semi application that gives you the same guarantees that raft gives you, for example.Utsav: [1:01:55] Okay.Sugu: [1:01:56] But then the theory behind this is kind of what I've explained in the blog feed. So that I think is one great innovation. And the benchmarking is another, is another good one, that may be a PhD that can make them out of that. Or at least a MTech thesis. I don't know what he's going to do, but float on is he's actually a student who has done an awesome job with them. So, some research work is going to come out of what you've done. Yeah.Utsav: [1:02:28] That's awesome to hear, because you don't generally hear of smaller companies and like academy work coming out of them, but plan skills, exactly. The kind of place where something like that would come out.Sugu: [1:02:40] Yeah. There's a lot more there. The other cool one is, the V replication or the materialization, feature, which is to be able to materialize anything into anything., and, it works so well within the model of, the Tessa sharding scheme, that you can actually, run, that is a foundation for a large number of migrations. It's a foundation for applying. DDLs like today in planet scale, you say, deploy this video, we do it in an offline fashion as in like with no downtime and the safeties, you can even rework it. Right. So those, so the way I see it is a DDL is a form of migration. And it's just one way of expressing a migration as a combined, but there are bigger migrations, very moving a table from one database to another where you Rashad a table. These are all DDLs of some sort. And if you can express them as videos, and if the system can do it offline in the background with no downtime and with reversibility, I think that is a very powerful mechanism to make available for you to use this. Like me, often some of these projects take months, years. Yeah, you can now do them in the test with just a simple command. Utsav: [1:04:16] Yeah, this seems like the kind of things that are just completely unheard of or impossible in any kind of other data system that you have.Sugu: [1:04:22] Yeah, near stone because it's science fiction, this is science fiction. Utsav: [1:04:31] And I also let that. That test can also be deployed within Kubernetes. So, you can get all of the benefits of, you know, like the same way as you get benefits of Kubernetes with stateless services that things can just move around. You can also use that to your benefit with tasks where your data can be moved around and stored in multiple places. Am I completely misremembering stuff for like, [cross-talking 1:04:59]?Sugu: [1:05:00] No, it's not like I can reveal to you that planet scale, the tests are all hosted in Kubernetes or everything runs in Kubernetes. And, we have a very large number of clusters. We may already be, or, may soon be, you know, the largest number of key spaces in the world. All in Kubernetes, the entire thing is in Kubernetes.Utsav: [1:95:29] So, that's like at least traditionally, so do you run on containers or is it just like, yeah, that's a tradition, yeah. If people say that you shouldn't run stateful stuff in containers and clearly bucking that trend. So maybe you can just talk through that a little bit.Sugu: [1:05:47] Yeah, totally. So that's, so the, the, the history goes behind the history behind this goes all the way back to, how we test evolve him YouTube and Google, right. On YouTube, originally, we wrote with testing our own dedicated hardware, but we were required to migrate it to Google's work, which is Google's cloud. And, Vitess had to, and the problem was that my sequel was not a first-class citizen in Google sport. Which means that you couldn't use it as a service. There was one team we actually collaborated with the team, but it was still not a first-class citizen. So, the way we did this was we actually landed with tests as, because, so the only way we could run with tests inside Google work was as a stateless application.So, we actually added the durability, survivability of nodes going down in, and Vitess such that anytime, the cloud can come and take down your server and we test should handle that. So that wasn't with Tessa's DNA that we had to build, and make it part of the test. So then later when Kubernetes came out, you had already for it because it was already running in what I call a data hostile environment., and therefore, like, because of the fact that we land at a massive scale inside Google board, we were able to, we have the confidence that it will run well. And he has shown them doesn't run well so far.Utsav: [1:07:34] Okay. Does that mean that the, my sequels basically live outside the Kubernetes and then you have a large part of like all of the tests it's running?Sugu: [1:07:39] Inside the container. Utsav: [1:07:40] Okay, Interesting. Sugu: [1:07:43] Yeah, it's my read on my sequels inside parts. We are not the first ones to do it by the way jd.com, HubSpot, and, there's another company. I forgot their name. They've been doing it for many years.Utsav: [1:08:03] Okay, and what are the benefits of running it inside a pot? I guess your perspective,Sugu: [1:08:06] Ah, manageability, right? Because it's a uniform environment. You don't treat tests as, you know, a special case like today, if you run a Kubernetes, that's the Kubernetes setup where you run your application and then you have to manage the database using different tools and technologies and methodologies. If you're running everything in Kubernetes, then you have a uniform set of management tools that work the same way. You want to add something, just push the ambles.Utsav: [1:08:40] That makes sense. And then you have to manage draining and basically hydrating nodes or pods. And you, do you have a custom logic to set that up?Sugu: [1:08:49] There is actually an operator that actually manages these things. We are actually trying to simplify that, you know, because we test, does have a lot of things for that, that we built when we were at war. I feel like at least our planet scale operator does a lot more than it should. It doesn't have to but yeah. There's some, a little bit of blue code is needed. I would sayUtsav: [1:09:18] That makes sense. Yeah. But it certainly sounds like science fiction reading all of your data running on these pods that can be moved around and changed. Anytime you don't even know which machine has all of your data, it sounds in the future.Sugu: [1:09:34] Yeah, it sounds like the future, but it is the future because there is no other way you can manage 10,000 nodes, right? Like if you're, if you're going from the unicorn world where there is this one server that you have to watch carefully to, it's going to be everywhere. It's going to be lots and lots of servers. How do you scale it? So, there's a term that one of our engineers used to use, you want off one manageability, which means that, as you scale the nodes, you shouldn't have to scale your team.Utsav: [1:10:12] That makes a lot of sense. Yeah. And what does your day-to-day job look like now? Like, what is your role or how has it changed over time? I'm guessing initially you must have already got itSugu: [1:10:21] Every time. Yeah. Like, it keeps changing. So, like until, initially I spent like 99% of my time on with deaths, like the early days of planet scale, the last seven, eight months, I spent a lot of time on our internal operator because there was one particular part that was broken that I said, okay, I'll go fix it. So that I'm actually winding down. So, I have to, so I, like, I have a feeling ... eventually what will happen is I think I will have to spend a lot more time publishing stuff. I have not been publishing enough. Like people have been asking me, when are you going to finish this consensus series? I still haven't had time to do it. So probably spend more time doing that, and probably also speak at conferences. I scaled that back a bit because I was too busy, not writing code internally, so it keeps changing. So, it will remain like that over because it's the way I see it is whatever is needed. It's like PlanetScale, right. Whatever is more important. That's the problem I am working on.Utsav: [1:11:44] Yep. That makes sense. And in some ways, it's like going back to your role at PayPal, they're just working on the important projects.Sugu: [1:11:52] Yeah, yeah, yeah. Keep it afloat.Utsav: [1:11:59] Yeah. And maybe to wrap things up a little bit, like I have one question for you, which is, have you seen the shape of your customers change over time? Like, you know, I'm, I'm sure initially with, with testers, like super large customers, but have you started seeing now with all of these usability improvements, smaller and smaller teams and companies like, you know, what we just want to future proof ourselves.Sugu: [1:12:18] Yeah, totally. It's night and day. That type of like, you know, some of like, even the words they use are very different. Like things like, you know, previous customers are more like very correctness, durability, scalability, and now like the newer customers talk about, you know, these two factors are not working, you know, I have this Ruby program. I have this rails thing. Even the language is very different. Yeah. It does change.Utsav: [1:12:54] Well, I think that that's a great sign though. Cause you're basically attracting a lot of people who traditionally would have never even thought about using something like with ESSA plan scale. So that's a great thingSugu: [1:13:04] To hear. Yeah. Yeah. Yeah.Utsav: [1:13:08] Anyways, well, thanks so much for being a guest. I think this was, yeah, this was a lot of fun for me. I love that trip down memory lane and understanding, knowing how the product is supposed to work and yeah. Again, thank you so much. Sugu: [1:13:12] Yeah. Thank you. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Welcome to Software at Scale, a podcast where we discuss the technical stories behind large software applications. I'm your host, Utsav Shah, and thank you for listening. Hey, welcome to another episode of the Software at Scale podcast. Before we get started, I would just like to ask a quick favor from any listeners who've been enjoying the show. Please leave a review on Apple Podcasts. That will help me out a lot. Thank you. Joining me today is Sugu, the CTO and co-founder of PlanetScale,

a managed database as a service platform company. Previously, he worked at YouTube for a really long time, from 2006 to 2018 on scalability and other things, including Vitesse, an open source CNCF project, and before that he was at PayPal. Welcome. Thank you. Glad to be here.

Good. Could you maybe tell me a little bit of when you got interested in databases and MySQL? Because it seems, at least from my understanding, you've been interested in distributed systems, MySQL databases for like a really long time. Yeah, so I would say the first database related stuff, the first serious involvement with databases started with when I joined Informix in 1993, I think. That was the time when there were three huge database companies. It was

Informix versus Sybiz versus Oracle. I guess eventually Oracle won the war. But that was the first time when I came in touch with databases. I specifically I did not work with the engine itself, but I worked on a development tool called 4GL, which was popular those days, and not there anymore. So that was actually my first introduction to databases. And I guess you took this skill set or this interest when you moved to YouTube from PayPal. So what were you doing at PayPal initially and what made you decide that you want to work at YouTube? So, PayPal, I guess I went to move from Informix to PayPal was mainly because this was around 2000.

This was the time when the internet boom was happening. And it was clearly obvious that informics had fallen behind it like they were still trying to make client server work and that was pretty much dying technology that's when I kind of decided to make a change it was kind somewhat of a career change when you realize that when you realize that the technology you're working on is not going to be around much longer. I don't know if other people saw it there, but it was pretty obvious to me.

So that's the time when I decided that, you know, I kind of made a decision to start back from the beginning because you'd be surprised in year 99, not many people knew how a website worked. Now it's probably common knowledge, but I had no idea how a website worked. Like how do requests come in, what servers do what, and those things. So it was all kind of unknown science to me so for me it was like I am starting a new career like like even within software engineering each of these things sound like a different career you know their choices it's as if I stopped doing being software engineer a and now I'm going to become software engineer B. So the only skills I could

carry over were my ability to write programs. Beyond that, I knew nothing. And I had to learn all those things kind of from scratch again at PayPal. By the way, I did not join PayPal directly. I actually joined x.com, which was founded by Elon. So Elon actually hired me. And then later, X.com and PayPal merged. And that's how we became PayPal. And yeah. I think if you can tell us a little bit about Elon Musk,

I feel like this podcast will just shoot up in popularity because anything to do with Elon Musk now. Well, there's a story, I think I've said that, it's not very commonly known, but at least I might have said that in one place, which is that actually I failed the interview. I went for the interview and Elon thought me as not worthy, felt that, they decided to pass on me. But then I was so excited about when I found out what, at that time I didn't know who he was, this is year 2000. I mean, you knew he had just sold his previous company, but there were many other people who had done that. on me, I sent him an email saying why he should hire me.

And somehow that email convinced him to bring me back for a second round. And then I got hired and the rest is history. So interesting. What is it like working with him? Did you work closely with him? How big was x.com at that time? It was very small. It was, I think, we were about 15 engineers, 10-15 engineers. I think the total was 50, but most of the other people were

customer support type of thing, because it was a bank, right? So there was a lot of customer support. Yeah, I got to, at it's uh it's surprising uh many of those people that i worked with not just elon uh are now celebrities like jeremy stoppelman was like sitting right next to me you know i remember i i remember asking jeremy stoppelman is CEO of Yelp, in case you want to look it up. So I remember, like, I go there and he's the guy next to me. So I said, hey, you know, where's the bathroom? Where do you get coffee? And sometimes you say, yeah, yeah, it's there.

Sometimes you say, I don't know. And finally ask him, so how long have you been here? Oh, I joined yesterday. I literally joined the day after jeremy i got hired uh yeah so it's a good times um so elon himself by the way uh like spacex he used to talk about that at x.com as his dream, going into space. And he is really that way, really inspiring when he talks about it. He starts talking, everybody just shuts up and listens.

He used to be broke. There was a time when he was broke where he says he didn't have money for food. Did you know that Elon was broke where he says he didn't have money for food. Did you know that Elon was broke once in a while? I feel like I've read about it or seen some YouTube video where now there's like so many YouTube videos where they take quotes of Elon and it's like inspirational stuff. I think I've seen some of those. That's so fascinating. And yeah, now that clearly became the PayPal mafia with so many people making so many successful companies and stuff like did you work closely with any of the other like so Jeremy Stoppelman?

Yeah, there's lots of them. So I actually worked with Max a lot more later, right? After the merger. So Max definitely, Reid Hoffman. Although I don't think I interacted much with Reid Hoffman. There's Roloff for sure, Steve Chen and Chad. Obviously, that's how I ended up at YouTube. My memory fails me, but if you look at the mafia, I worked with all of them. Did you work on the Igor program, like the fraud detection stuff, which I have read about publicly? Oh, no, I did not. I worked on some fraud systems, but not Igor specifically. I am trying to remember who worked on it.

It was probably Russ Simmons. So what did you do at PayPal? I am curious. So the thing, this is a good lesson. The thing I did at like the struggle I was having initially at PayPal was recognition. There was a couple of things. I was kind of an unknown there. Nobody knew who I was. And I was part of the merger. I came from the x.com PayPal merger. And there was a lot of back and forth about which technology to use. And eventually the original PayPal technology we decided to continue with instead of using the x.com technology. So the struggle I had was how do I prove myself to be worthy?

So in software engineering, you have to show that you are good and then they give you good projects. You don't get to choose the projects because you get assigned projects. So at least that's how the situation was then. And by the fact that I was an unknown, all the good work was going to the people who were already recognized for how good they were. So there was this one tool called the Activity Report, which used to basically tally the total of all transactions on a daily basis and say this is the total in this

category, in credit card category, ACH category and that kind of stuff, and used to report those numbers. But the auditors came in and they said, these transaction numbers that this report produces, you need to tally them against your bank accounts, against the total balances that users have, against the bank accounts that you have. the first time the auditors came there was like a few million dollars off you know and you're asking like where is all this money yeah I

don't know if you remember the when we when PayPal initially was founded the business model was to make money off of the float. It was when we are transferring money, there is a float. We will basically invest that money and whatever money we make out of that. So people are like, oh, it must be the float, that difference. It's because of the float. And the auditors are like, no, that explanation

is not sufficient. You need to show the numbers that they add up and that they tally and so uh finally we figured out where where all it is and it started tallying but the problem was paypal was software written by engineers and how do you how do you create transaction you insert a row into the transaction table and commit you insert a row into the transaction table and commit. You insert a row in the transaction table and commit. You insert a row in the transaction table, update the user's balance and then do a commit. So that's how you do it. But sometimes the transactions go pending, in which case you don't update the balance. Sometimes you complete the transaction and then after that you update

the balance. And guess what happens when you start doing these things, there is this thing called software bugs that creep up. And so every once in a while after every release the transactions would not add up to the user's balance and sometimes they would not tally with what is actually in the bank accounts. And so every few days the activity report used to produce discrepancies. And because the software was written organically, there was no easy way to find out. Like you have million transactions, how do you figure out which one went wrong? Numbers do not add up to the totals here. And every time this used to happen, an engineer was

assigned to go and troubleshoot. It was the most hated job because it's like finding a needle in the haystack. Like you say, oh, activity report conflict. Now I got assigned to do this. Why is my life so bad? So that was the situation. And then at one point of time, I said, you know what, I'll own this problem. I said, just give me this problem, forget it. I will investigate this every time it happens. And I will also fix the software to make it easier. So I took on this and I rewrote the entire program such that to the extent that at the end of its maturity, it would find out exactly, previously it was all totals. But at the end, once it got fully evolved, it will say this particular account has this discrepancy and most likely this is the transaction that caused it.

So it will produce a report, it will produce a list of conflicts with very clear clue about where the problem is and then you spend five minutes and you know what went wrong. It will find software bugs in it because sometimes somebody forgot to update the balance, you know, and say, hey, you created a transaction, didn't update the balance, you know, so here go fix it. So that kind of, so that it used to do. And eventually what that happened was, because it was so solid, it kind of became foundational for paypal like people used to ask for reports about you know transaction and they say how do you know this is correct say oh i

validated against you know sugoi's activity report ah okay then it's good i trust these numbers so that kind of stuff so that making that software and taking that pain away from all the other engineers got me accepted as one of the cool engineers. So interesting. How big was the engineering group at that time? So the paper, the x.com side, there were about 15. Paper side was also 15. So we were about 30 to 40 engineers. And once you got accepted by these people, what were your next projects, I guess? So having gotten accepted, so I became like the kind of the person who says if you give a problem to Sugu, you know, consider it solved kind of thing. And so which means that anytime PayPal had a struggle where you say, oh, you

need the top brains on this, I used to get caught. And the biggest problems later were all about scalability. So I kind of moved into the PayPal architecture team and used to solve all problems related to scalability and the scalability was always the Oracle database, because it was a single instance database. And every time the only way we could scale it was vertically. So the idea of sharding was actually born there for us that we need to shard this database and but it eventually succeeded only at YouTube. We did not succeed at sharding at PayPal itself, but we kind of knew how to do it. We kind of learned, went through the learning process when we were at PayPal about how to

shard a database. But by then the eBay acquisition had taken place, there was attrition, lots of things happened after that. That's so interesting. And is that where you met Mike and you were discussing some of these? Yeah, Mike was reporting to me. So there's Mike, there was another person called Matt Rizzo. They were the top engineers, one of the top engineers at PayPal. And yeah, so we still kid about some terms, we still use some terms that only Mike and I would understand. You say, oh, this is APCD, right? And then

we use, nobody else outside would say would understand what that means. It was an ACH process completed deposits, that's tool that he wrote. Like you probably know that Mike is really, really, really, really, really good with bash. Yeah, he knows his bash. He knows his bash. He knows his bash. That was kind of my fault or slash doing. Because there was a time when I just told him a problem. You own this problem. You have to solve it.

And he says, do you think bash is a good idea? It sounds cool. I said, if you think it's a good idea, it's a good idea. It's up to you. It's your decision. So he went and he still feels scarred by it by the fact that I let him do it. Why did you let me? Why did you tell me it was a mistake? I can't imagine this really horrible process at PayFab, like adding and subtracting money,

and it's all written in bash. That wasn't written in bash, but there was some tooling that he wrote in bash. The money part was all C code, C++ actually. So he was writing, I guess that could be called the Kubernetes of those days. He was writing an orchestration tool and that he wrote it in bash. Okay, that's so interesting. He was writing an orchestration tool and that he wrote it in bash. Okay. And that's so interesting. And then you're saying that the idea of sharding was born at PayPal or like the idea of sharding or MySQL database, like you figured out how to do it.

Why would you say it did not complete it just because there was not enough time or and at that point you decided to leave and go to YouTube? There were a lot of reasons. Some of it was, there was also the management change. At that time, there was the PayPal engineers, there was the eBay management that had acquired us and the relationships were very strained. So it was kind of a very difficult thing. So there was no coordinated effort. There was no focused coordinated effort towards solving it. It was all over the map and there was a lot of committees spending a lot of time in conference rooms discussing things, not coming to conclusions, that kind of stuff.

Where there was just too much, too many cooks in the kitchen type of thing. I think a core of us kind of figured out how this needs to be done, but we were not able to push that idea forward. We finally made that happen at YouTube, which proved, which kind of absolved us of, we know what we are doing type of thing. So maybe you can talk about the early days at YouTube. What was YouTube like in 2006? How many people were there? Was this pre-acquisition, post-acquisition?

I don't know the exact date. I joined just around the time of acquisition. But I have been… So actually around the time when YouTube was founded, I actually went to India to open the PayPal India office. But I used to visit YouTube every time I visited. I came to US and I used to come pretty often. So I've been closely following. And at some point of time, you know, Steve and Matt Rizzo, the other, kept hounding me saying that, what the hell are

you doing there? this thing is catching fire you need to come and join us right now so finally i came back and joined them but yeah so the the youtube culture was kind of a carryover from paypal he was it was very very similar Lots of heroics, you know, lots of, like each person was an impact player. Each person made a huge, huge difference. Like one person owned search, all of search, that kind of stuff. One person owned the entire browsing experience, that type of. There were only like 10 engineers. So everybody owned a very important piece of the stack.

And what were some of the early problems that you ran into? I guess what were the fires that you were fixing as soon as you started? So the one big thing I remember, so there was this rule at YouTube where you join, you have to commit on your first day. I broke that rule. I said I'm not going to commit on the first day. spent time learning the code and produced a huge PR, a huge pull request that rewrote an entire module because it was badly written. So the biggest problem I found, I felt at least at that time was because the code was organically grown and increment know improved, it was going to run out of steam

as we added more features. So, I felt that you needed to do you know you need to clean it up so that it can you can you can with a put a better foundation underneath so that you can add more features to it. So, that is basically the first work that I did. But then the later after that the bigger problem also became scalability because we were single, we were unsharded, we have those only one database that was called main and we had to do the resharding. So that was the biggest challenge that we solved. We solved resharding pre VITAS. V Vitas came later and adopted that sharding method. But when we first sharded it was at the app layer. Okay. At what scale did you have to start thinking about like sharding? Like how many

engineers were there? How much data was going in? And like what was being stored in this like database? Was this just everything of YouTube's metadata was in this one large database? Yes, yes. All the metadata, not the videos themselves, because the videos themselves were video files and they were distributed through CDN. But all other metadata like video title, likes, dislikes, everything else was stored in the MySQL database. And it was running out of capacity very soon and the hardware wouldn't have kept up to vertical scaling wouldn't have taken us much, much further.

And that's when we said, okay, we need to, we need to short this. We made a very tough call, which is that we are going to break cross short transactional integrity. It was actually not as big a deal as we thought it was going to be. Because of the nature of the app, maybe because users just mostly work with their profile. There were a few things about comments because I post a comment to you, where do you put that comment and some little bit of issues around those types of things. But otherwise, there was still quite a bit of rewrite because an application that assumes an unshardened

database and make it shard aware is non-trivial, no matter how you put it. We had to build a layer under the app and write everything, change everything to go through that layer, follow certain rules and then we change that layer to accommodate sharding. That's basically how we solve the problem. That layer has to basically decide which shard to go to. So you were just running out of physical disk space on that MySQL box, which seems like the problem. I am trying to remember whether it was disk.

It might have been CPU actually. I think it was disk, it might have been CPU actually. I think it was CPU. CPU is either CPU or memory. I think we could have, disk we could have grown. to remember it is CPU memory or IOPS, most likely memory because of working set size issues. I could be wrong. I am just curious, do you know how many videos you had in YouTube at that time when you started running out of issues?

I remember celebrating. Let me see. Was it, it's either 100 million or 1 billion. I don't know. Okay, 1 billion videos and that's when you start running out of space. Total videos. Total videos. That just shows you the power of like MySQL and like all of these. Yeah, it is very, yes. It's amazing that I don not know the difference between 100 million

and 1 billion, you know. But something, yeah. Sounds like accurate, I do not even know how many videos will be on YouTube now, I cannot even imagine. The reason why it works is, most videos are inactive, right? You insert, you just insert a row and move on and so it is more a question of what is the working set right how many of those videos are popular they come and live in memory and we had a memcache layer also in in the front and and so it

was uh it was more a question of how many uh how many popular videos people are trying to hit and can all that fit in memory. Okay, that makes a lot of sense. And yeah, then you could put like a caching layer in front so that you don't hit the database for all of these popular videos. That's so interesting. And given all of this, like, it makes sense to me that you can shard this because videos you can totally imagine, it's relatively easy to shard it sounds like. But

how do you think about resharding? How do you solve resharding? That sounds like a very interesting problem. And maybe just to add some color to what I think resharding means, you have a data on MySQL box A and you want to move some of it to MySQL box B or split from one database to two and two to four and you have to somehow live transfer data from A to B. How do you solve that? So there's the principle and there is a mechanic, right. So, there is a typically the mechanic is actually more kind of straightforward, which is typically when you start sharding, the first thing you do is actually split tables out, whole tables out. Like if there are 300 tables, you will say, these 20 tables can live in

a separate database. It is not really called sharding or resharding because you are not sharding, you are splitting. So the first thing people usually do is split a database into smaller parts, because then each one can grow on its own. The sharding aspect comes when few set of tables actually themselves become so big that they cannot fit in a database. So, that is when you shard and when you shard, when you make that sharding decision, what you do is tables that are related that are together, you shard them the same way. Like in the YouTube's case when we did the sharding, we kept the users and their videos together in the same shard. So that was actually a conscious decision that users and videos stay together, there is actually a sharding model

that allows you to represent that as a relationship that VITES actually exposes, which is basically there is some theory crafting behind that. But from a common sense perspective, you can always reason about users and their videos being together. So that is the mechanic part. On the principle part actually, the way you think about sharding is basically beyond a scale, you have to actually rethink little bit of your application as being fully distributed and fully, what do you call, fully distributed with independent operators working by themselves, working on their own. That means that some features you, some features you actually often have to forego

for the sake of scalability. In other words, I mean, crazy like some like crazy features like what is the total number, what is the real time value of the total number of videos in the system. I want that up to the millisecond. That type of question is unreasonable for a sharding system. You could probably do a select count star against a small database and can actually do it. But if you are having tens of thousands of servers, answering that question becomes harder, right? And then at that time, you have to start making trade offs like do you, is it really important that you have to know it up to the, like what if it is up to the minute? Is that good enough? You know? So those kinds of trade offs you

have to make at a high level. And most people do because you cannot go to that scale unless you make these trade offs. But unless you make these trade-offs. But once you make these trade-offs, sharding becomes actually very natural. Because the one thing that you make, like if you are, for example, if you are going to have the entire world as your customer, you are going to have, you know, 7 billion rows in your user account. And sometimes many of them create multiple accounts. So you are going to have billions of rows.

You cannot build features that puts them all together. You have to build features that keeps them separate. And as soon as your features follow that pattern, then sharding also kind of drives that same decision. So it more or less becomes natural. Okay, that makes sense to me. And it seems like you said that you have a sharding solution that predated Vitesse. So maybe you can talk about the motivation behind Vitesse like and maybe the year that this was in like, well, if you have any numbers on you know, at what

point did you probably it's been too long, like I'm surprised I'm finding myself surprised that there are details I should know that I don't know like number of videos we had when we did sharding, right? It is 2007, 2007 is actually wow, 14 years ago. So yeah, it has been many years. So in 2007 was I think when we did our first sharding, if I remember correctly. Maybe 2008, around 2007 or 2008, one of those two years was the first time when we did sharding, if I remember correctly, maybe 2008, around 2007 or 2008, one of those two years was the first time when we did sharding.

But VITAS was not born because we needed to do sharding, obviously, because we were already sharded. It was born because we could not keep up with the number of outages that the database was causing. That was actually the database was causing. That was actually why it was born. It was quite obvious that as the system was scaling, in spite of being sharded, there were a large number of things broken within the system that needed fixing for it to work. And system overall, when I say system, it is end to end

the entire system. And that includes the engineers and their development process. So, that is how big the problem was, but we did not solve the entire problem. We said from a database perspective, what can we do to improve things? That's kind of the scope of the problem. And more specifically, Mike actually took off and I think spent some time in Dana Street Coffee at Mountain View and came up with this spreadsheet where he described, he kind of listed every outage that we have had and what solution and what is, how did we solve it?

Is there, what would be the best remedy for to avoid such an outage? So when we actually then sat down and studied that spreadsheet, it was obvious that new proxy layer had to be built. And that is how Vitesse was born. And the whole idea is to protect the database from the developer.

Like for example, developer, so there were like at that time there were like 200 developers. If you look at it right, the developers do not intentionally write bad queries. And do not, developers, a developer does not write a bad query all the time. But if a developer wrote a bad query only once a year, with 200 developers, 50 weeks a year, you do the math. How often do we see outages in the database? Almost every day. That's what it amounted to, is we were seeing outages almost every day. And very often they were actually put quote unquote bad queries coming from that one user wrote that they will never repeat,

but they've fulfilled their quota for the year. And so what we did was we found common patterns where that would cause data. So the big change that we made was if i wrote a bad query i have to pay the price for it not the entire team right now today if i wrote a bad query that query runs on my sql it takes on the entire database so we wrote with tests in such a way that if you wrote a bad query your query would fail so in other words words, what we used to look at is how long is the query running. The query is running too long, we kill it. If the query is fetching too many rows, we return an error.

So those kind of defenses we added in the test early on. And that pretty much took it a long way. The other big feature that we deployed was connection pooling, which MySQL was very bad at. The newer MySQLs are slightly better, they are still not as good. So the connection pooling feature was a lifesaver for YouTube. That makes a lot of sense to me. And maybe you can tell us a little bit about why can't MySQL or why didn't MySQL at that time have the capability of defending itself. So you might imagine like you know just from somebody who has no MySQL experience they

can just be like a timeout that you can put. It's very simple. The reason is because MySQL has what you call claims itself to be a relational database and a relational database is required to not fail. So we at YouTube could be opinionated about what query should work and what query should not work. That freedom MySQL does not have. Every query that is given to it, it has to return you the results.

So that is the rule that you have to be qualified as a relational database, you have to fulfill that rule and that rule was its curse. But like you could configure maybe like a session timeout or something or like a query timeout or would those defenses were just not good enough, basically. They actually added those features much later, but they didn't. Yeah, they are the newer MySQLs now do have those features. But I don't know how, but they are all behind. They are not as usable because you have to set it in your query parameter, you have to

set it in your session. So it's not as like by default, you just install MySQL and start running it, you won't have this property. You have to configure it, which is another hurdle to cross. Whereas in Vitas, you install Vitas, it'll start failing, you know? People complain like we see the opposite problem, right? Oh, this query runs in MySQL, why is Vitas failing it?

Because you know, you're trying to fetch a million rows. If you want to fetch a million rows, you have to tell Vitas that you want to fetch million rows and then it'll give you that. But if you just send a query, it won't give fetch a million rows, you have to tell with us that you want to fetch million rows and then it will give you them. But if you just send a query, it won't give you the million rows. What do you think was like the one really important feature? You said there's connection pooling, it also limited bad queries. Like how did it figure out that a query was not so one maybe heuristic is it's returning

too many rows and it would just fail fast. What were some other key problems it solved at that time? Oh, there were, we added so many. The coolest one was when engineers wrote code, they asked like, and me too, right? I am guilty of the same thing. You think how many videos could a user possibly upload? And you are thinking manually in the back of your mind. For me to upload a video, I have to go make it, you know, edit it, create an MPEG file and then upload it. You know, we will have, you know, 2000 videos would be a number. And you know, look at how old YouTube was, it was 2 years old.

How many videos can you produce in 2 years? So 2000 videos, so selecting all videos of the user is not a problem, right? So we never put a limit on how many videos we fetched, then we selected a video by user. And then there was one user who ran a bot, I think, there is no other way. Well, now there are accounts that have millions of videos, but at that time that user had 250,000 videos. And that was not a problem per se, because that account had been around, but it became a problem when that account got listed on YouTube's front page, the index page, the landing page. So which means that every person that went to YouTube.com issued a query that pulled

250,000 rows. Because there was, so one of the changes we made was that any query that that is where we added the limit class right where you say if you send a query with no limit class we will put a limit and if that limit is exceeded we will just return your error saying that you are trying to fetch too many rows. So that was one protection. But cooler protection that we did was, there are some queries that are just inherently expensive like this one. I think in this case, I think it might have been a select count start. If it is a select count, it does not fetch 250,000 rows but it scans 250,000 rows. So the feature that we did was if a query got spammed

like because of the fact it is coming from the front page, which means that that MySQL is receiving like a very high QPS of the same query. In that case what we do is we check if the query is already running from some other request. If there are 10 other requests of the same query, they are not sent to my SQL. We wait for the query to return and return that same result to all those requested. Would that be problematic from a data consistency perspective? Probably not. It is actually, yes, it is a, but it is no worse than eventual consistent. So if you are reading, if you are like reading from a replica, no problem. If you are doing something

transactional, so the rule that we had at YouTube was if you are doing something transactional. So the reason, there's also another reason. Like the rule is if you read a row, assume that row is already stale. Why? Because as soon as you read the row, somebody could have gone and changed it. So that stainless guarantee kind of carried over into this rule also is that when you do a select of the query, it may be slightly outdated because one thing is you could be

reading from a previous query that is running or sometimes it could be going to a replica. So it was something that nobody even noticed that we had made this change. But it's like we've never had an outage related to, you know, query spam after we did that feature. Yeah. But more importantly, the rule that we followed was if you want to make sure that a row doesn't change after you've read it, You have to select for update. So if you plan to update a row and update it based on the value that you have read, you have to select for update. So that's the rule that all engineers followed.

Yeah, and that makes sense. And it's better to just be explicit about something like that. Yeah, yeah. Because of the fact that the MySQL MVCC model tells you, right? In MVCC, the read, the consistent read, it is called consistent read, that is actually the actual technical name is something else. But that model basically tells you that the row that MySQL serves to you is technically obsolete. Okay. And then this makes sense to me and you all open source with us.

I remember hearing the reason is you never wanted to write another with us. So you decided to open source it from day one. Can you maybe tell me a little bit about the motivations behind making it a product? So it's a great piece of software which used to power YouTube from my understanding for like a really, really long time. At what point did you decide we should actually make this available for other companies to use and what is the transformation of Vitesse to PlanetScale? So even before PlanetScale, even as early as 2014-2015, we felt that we should start promoting Vitesse outside

of YouTube. And one of the reasons was, it was not a very serious thing, but it was kind of a secondary motivation. We felt that even for YouTube, for Vitesse to be credible even within YouTube, this product needs to be used by other people beyond YouTube. In other words, for a company to trust a product, a lot of other people must be using it. That was kind of the motivation that made us think that we should encourage other companies to also use Vitesse. And actually Flipkart was the first one, I think it was 2015 or 2016 that they approached

us and they said, hey, it looks like this will perfectly solve the problem that we are trying to solve. And they were the first adopters of Vitesse. Okay. So that makes sense. And that's when you open, like it was open source, and you felt that other companies should use it. 2015, it sounds like Flipkart was interested. But then what is the transformation of from then to, you know, let's make it a product

that people can buy? Because this is, it's an it's an open source technology, right? How do you go from that to product? Yeah, so this is, I can talk about how PlanetScale came about. So the other thing that started happening over time, as Vitesse started evolving as a project, companies like Flipkart started relying on it, Slack started using it, Square started using it.

And they were all scared of using Vitesse. Why? Because this is basically defining technology. It's not something, it's basically a lifetime commitment type of change. And the question in their mind is, what, like, what's the guarantee that, like, what is YouTube's interest in making sure that, you know, it will continue to work for Slack, right? Why like, they were a large number of companies were worried about that part where you know youtube's focus is videos by like how can we trust a technology how can we trust the future of our company right on a technology what if youtube

abandons the project and and that kind of stuff so uh there was a over time i could sense that hesitancy growing among people and this is one of the contributing factors to us starting PlanetScale, where we said we are going to start a company, that company is going to make a commitment to stand behind this open source project. So when we made that announcement that we are starting the company, there was a sigh of relief throughout the test community. Finally, I can depend on this really important technology. I can depend on this project.

It is not like it is here to stay type of thing. That there is a company who is going to make sure that this project stays healthy. That makes sense. There were other factors also, but this was one of, definitely one of the major contributing factors towards making. So at the end of the day, this is something, this is generally a problem in open source. That you say open source is free, but there is an economy behind it because the engineer's time is not free and for me to spend time on an open source project i have to be paid money because

you know i have my family to take care of you know need money to live. And so expecting engineers to contribute to open source in their free time, it can work for smaller projects. But once a project becomes beyond certain size, it is not a part-time. It cannot be a part-time contribution. It is a full-time contribution, and the engineer has to be paid money for it. And so there has to be some kind of economy behind that. That makes a lot of sense.

And I think it also tells me that the problem that Slack and Square and these other companies were facing was just so large that they wanted to use the project given all of these issues with, we don't know the future of this project, but there was no other solution for them given their existing MySQL stack and their hyper growth and the kind of problems that they'd be dealing with. Yeah, that's right. So that is something I still wonder about, right? If not Vitesse, what could have, is there an alternate solution?

The alternate does not look as good either because the alternate would be a closed source product like that scales you that even more scary because what if that company comes out of business right at least in the case of vitesse the source code is there you know i if if planet scale goes away slack can employ engineers you know to take over that source code and continue to operate so that that confidence is there so that is one big advantage of something being open source. It gives higher confidence that, you know, worst case scenario, they have a way to move forward.

And when you finally decided to make a company and a product, what are some things you've learned along the way that you need to have in Vitesse for more people to be able to use it? At a first impression to me, it's like there are not millions of companies running into these scale issues, but like, what are some interesting things that you've learned along the way? Yeah.

All I can say is I am still learning. Yeah. Uh, every time I come there, I realize, oh, you know what, uh, what I thought was, was important is not really that important like somebody asked me if if you if you are agonizing so much about data integrity why is mongodb so popular right i mean mongodb is pretty solid now as far as I can tell, but it was not solid when it became popular. They made it solid after it became popular. So MongoDB did not, like for the longest time did not care about data integrity, but people still flocked for it.

So there are things, there is always a trade off in what is important and you have to find out what that is. And you have to basically meet people where they are. And in the case of VITES, it's actually usability, approachability and usability. That is a bigger hurdle to cross before you can adopt VITES. And that is a much bigger hurdle than the data integrity guarantees that people look for. So that's, for example, one lesson. So the one thing with VITES is we have our ears open. We are constantly listening to people giving us feedback about what is not working for them and fixing those problems.

It turns out that is insufficient. Who are we not listening to? We are not listening to the person, the quiet person that tries to test over the weekend, does not work for them and quietly walks away. Right? That's that voice we never heard. We only heard the voice of somebody who tried to use Vitesse and like went past the problem of being able not being able to use it, got to use it, found a missing feature and is asking for it. So we have been focusing on very specific features about how to make Vitas do this. But we completely forgot about how to make Vitas more approachable. So those are the problems we are now solving at PlanetScale.

Okay. That makes a lot of sense. Maybe you can tell us about like one usability feature that you feel was like extremely important that you all have built. The biggest one, for example, is schema deployment. Subconsciously every developer will cringe when you say, use a database and manage your schema. Use a database they would like, but as soon as you say schema, almost every developer cringes because, oh my god, I have to deal with that bureaucracy. Like, send it for approval, DBA is going to reject it,

or the schema is going to break something, some app is going to break. All those headaches I have to deal with. So what we have done at PlanetScale is give you an experience very similar to what you would do as a developer with your source code. You do the same thing with your database. You can take a database, branch it, apply schema changes, test it and then

have a workflow to merge it back into production. And very natural and it looks almost similar to how you develop your source code, right? So it is a very natural workflow and it applies to the database and you do't have to deal with that whole bureaucracy. It's all part of this nice workflow. It handles conflicts if multiple people want to change the schema at the same time. It pipelines them the correct way and if they cannot go together, it will flag it for you. So those are really, really nice features and that seems to be really going well with the

developers. Interesting. Yeah. And I think it speaks to you know, why database like MongoDB are so popular. Yeah, no schema. It's not the no SQL that made MongoDB when it's the no schema part. Being able to just tweak add one more field not be blocked on like a team to make sure not to worry about backfilling and all of that. It is a game changer. And I'm maybe a little ashamed to admit I actually like I use MongoDB at this job and it's not half bad. It's pretty decent.

Yeah, yeah. It's a the end and to and now they are trying to bring SQL back to MongoDB and people like Yeah, yeah. And now they are trying to bring SQL back to MongoDB and people like it, right? So the real problem was actually schemas. The fact that you can't just add something and move on. It's so hard to do it in a database. And maybe you can tell me today, like, what's the difference between Vitesse, Open Source and PlanetScale? What are the extra stuff you get?

They are very orthogonal, right? Because what we are building in PlanetScale is a beautiful developer experience. And what Vitesse is giving you is the other part that most people that have a good developer experience miss which is a solid scalable database in the back end like we could build this developer experience on top of stock mysql but people are hesitating to adopt that because they know that it won't scale at some point of time i'm going to hit a limit and I'm going to spend 90% of my energy trying to figure out how to scale this. That's what companies experience. That problem is taken away from you. You get a good developer experience but when

it comes to scaling, Vitesse has you covered. That makes a lot of sense and does that mean that as a developer today let's say I want to start a startup tomorrow, I have a little bit of experience with like, MySQL, can I just start with PlanetScale on day one? What would I get? Exactly. Yeah, that's exactly what PlanetScale is built for. Yeah, you like with basically you spend almost no time configuring your database.

You start, you said just go to PlanX, click like you click a button, you instantly get a database and then you just start developing on it. So zero barriers, which is what you want in a startup. Okay, and the experience I get is just stock my SQL to begin with except for it would have all of these things like automatic limits on queries and it would let me shard as soon as I become too big. As soon as yes exactly. Exactly. So basically what it will do is make sure that it will kind of guide you towards good programming practices. You know what I mean?

If you are just running unbounded queries, it's going to tell you this is not good. So add bounds to your queries, that kind of stuff. So we can be opinionated that way. So I'm curious about all of the technical innovation. I see your Twitter feed and you talk about, you know, you'll have like an automatic benchmarking tool, you're like optimizing protobufs handling and like go, what are some of the key like other innovations that you're doing at Planet Scale? So there is a, there is one, I don't know if you have seen the blog series about consensus, generalized

consensus. Okay, so I think that's a decent, I feel like it's a decent innovation. What we are doing is there's Paxos, there's Raft there is raft, those in my eyes are rigid algorithms and they have rigid assumptions about how nodes should talk to each other and those algorithms were adopted for industry best practice. But if you flip it around saying that what problems are they solving, you identify the problem and then you start with the problem first and come in and say this is the problem I want to solve, what system would you build?

I do not think Raft or Paxos would be the systems we would have built. And what problems are we trying to solve? We are trying to solve the problem of durability. I have a distributed system and I say save this data and the system says I have saved your data and the guarantee that I want is that the system does not lose my data. Essentially that is the problem that all these consensus systems solve. But then when I come in here, I am in a cloud environment, I am on AWS, I have zones, I have regions, right? I can say for me, durability means my data is across two nodes or my notion of durability

is data is at least in one other zone or one other region, right? So if you specify that, a system that can make sure that whenever you ask it to write data, it makes sure that the data has reached all those nodes or regions. And then it gives you, and then later, if there are failures, it recovers based on the fact that the data is elsewhere, right? So that is what the thing solves. So if you looked at it top down, you come up with a very different approach to the problem. Where in which case raft and Paxos are just one way to solve it. So what I have explained is what is the generic approach behind solving the problem of durability,

how Paxos and Raft solve it, but how we can build other systems that are more effective at more accurately meeting the driving requirements that are coming from the business. So this reminds me of like semi-sync replication and like MySQL. Exactly. So you could build a consensus system using MySQL semi-sync replication and like MySQL. Exactly. So you could build a consensus system using MySQL semi-sync replication. That gives you the same guarantees that draft gives you for example. And the theory behind this is kind of what I have explained in the blog feed. So that I think is one great innovation.

And the benchmarking is another good one. There may be a PhD that may come out of that or at least M.Tech thesis, I don't know what he is going to do but Florent is actually a student who has done an awesome job. So some research work is going to come out of what he has done. Yeah, that's awesome to hear because you don't generally hear of like smaller companies and like academic work coming out of them. But plant skills exactly the kind of place where something like that would come out. Yeah, there's a lot more. The other cool one is the V replication or the materialization feature, which

is to be able to materialize anything into anything. And it works so well within the model of the BITES starting scheme that you can actually run. That is a foundation for a large number of migrations. It's a foundation for applying DDLs. Like today in PlanetScale, you say deploy this DDL. We do it in an offline fashion, as in like with no downtime. And the safety is you can even revert it, right? So those, so the way I see it is a DDL is a form of migration.

And it is just one way of expressing a migration as a command. But there are bigger migrations where you move a table from one database to another, where you reshard a table. These are all DDLs of some sort. And if you can express them as DDLs, and if the system can do it offline in the background with no downtime and with reversibility, I think that is a very powerful mechanism

to make available to users. Like often somehow these projects take months, years. You can now do them in the test with just a simple command. They seem like the kind of things that are just completely unheard of or impossible in any kind of other data system that you have. Yeah, our engineer Shl he calls it science fiction. This is science fiction. And I also read that Vitas can also be deployed within Kubernetes. So you can get all of the benefits of, you know, like, the same as same way as you get

benefits of Kubernetes with stateless services that things can just move around, you can also use that to your benefit in Vitesse where your data can be moved around and stored in multiple places. Am I completely misremembering stuff? What's the advantage of that? No, it's completely accurate. I can reveal to you that PlanetScale, the Vitas are all hosted in Kubernetes. All, everything runs in Kubernetes.

And we have a very large number of clusters. We may already be, or may soon be, you know, the largest number of key spaces in the world. All in Kubernetes. The world. All in Kubernetes. The entire thing is in Kubernetes. So that like, at least traditionally, so do you run on containers? Or is it just like, yeah, yeah, that's traditionally Yeah, people say that you shouldn't run stateful stuff in containers.

And clearly you're bucking that trend. So maybe you can just talk through that a little bit. Yeah totally so that's so the the history goes behind the history behind this goes all the way back to how Vitesse evolved in YouTube and Google right in YouTube we were in originally we wrote Vitesse in our own dedicated hardware but we were required to migrate it to Google's board, which is Google's cloud. And with us had to end the problem was that MySQL was not a first class citizen in Google's board, which means that you couldn't use it as a service.

There was one team. We actually collaborated with that team, but it was still not a first class citizen. So the way we did this was we actually ran VITAS as, because, so the only way we could run VITAS inside Google fork was as a stateless application. So we actually added the durability, survivability of nodes going down in Vitesse such that any time the cloud can come and take down your server, and Vitesse should handle that. So that was in Vitesse's DNA that we had to build and make

it part of Vitesse. So when later, when Kubernetes came out, we were all ready for it because with this was already running in what I call a data host style environment. And therefore, like because of the fact that we ran at massive scale inside Google's board, we were able to, we have the confidence that it will run well and it has shown that it has run well so far. Okay. So does that mean that the MySQLs basically live outside the Kubernetes and then you have

a large part of like all of the Vitesse service? MySQL is running inside the container. Okay. Okay. Interesting. Yeah. It's my, we run MySQLs inside pods. We are not the first ones to do it by the way.

JD.com, Hubspot and there's another company, I forgot their name. They've been doing it for many years. And what are the benefits of running it inside a pod, I guess, from your perspective? Manageability, right? Because it's a uniform environment. You don't treat VITAS as a special case. Like today, if you run Kubernetes, it's the Kubernetes setup where you run your application and then you have to manage the database using different tools and technologies and methodologies. If you are running everything in Kubernetes, then you have a uniform set of management tools that work the same way. You want to add something, just push YAML. That makes sense. And then you

have to manage draining and basically hydrating nodes or pods and you do you have like custom logic to set that up there is actually uh there is actually an operator that that we use that actually manages these things we are actually trying to simplify that you know because uh vites vites does have a lot of things for that, that we built when we were at work. I feel like at least our PlanetScale operator does a lot more than it should. It doesn't have to. But yeah, a little bit of blue code is needed. That makes sense.

But it certainly sounds like science fiction, right? All of your data is running on these pods that can be moved around, changed anytime. You don't even know which machine has all of your data. It sounds in the future. It sounds in the future, but it is the future because there is no other way you can manage 10,000 nodes. Right? Like if you are going from the unicorn world where there is this one server that you have to watch carefully

to it's going to be everywhere, it's going to be lots and lots of servers. How do you scale, right? So there's a term that one of our engineers used to use. You want O of one manageability, which means that as you scale the nodes, you shouldn't have to scale your team. That makes a lot of sense.

Yeah, and what is your day-to-day job look like now? Like what is your role? How has it like changed over time? I'm guessing initially you must have written a lot of code. It keeps changing every time. Yeah. Like it keeps changing. So like until like initially I spent 99% of my time on Vitesse, the early days of Planet

Scale. The last 7-8 months, I spent a lot of time on our internal operator because there was one particular part that was broken that I said, okay, I'll go fix it. So that I'm actually winding down. So I have a feeling eventually what will happen is I think I will have to spend a lot more time publishing stuff. uh eventually what will happen is i think i will have to spend a lot more time publishing stuff i have not been publishing enough like people have been asking me when are you going to finish this consensus series i still haven't had time to do it so probably spend more time to do that and probably also speak at conferences i scaled that back a bit because I was too busy writing code internally.

So it keeps changing. So I think it will remain like that over because it's the way I see it is whatever is needed inside PlanetScale, right? Whatever is more important, that's the problem I'd be working on. That makes sense. And in some ways, it's like going back to your role at PayPal, just working on the important projects. Yeah, yeah, yeah.

Keep it afloat. And maybe to wrap things up a little bit, like I have one question for you, which is have you seen the shape of your customers change? Over time like, you know, I'm sure initially with Vitesse there's like super large customers But have you start seeing now with all of these usability improvements smaller and smaller? Teams and companies are like, you know what? We just want to future-proof ourselves Yeah, totally. It's a it's it's like night and day the type of like

Some of like even the words they use is very different you know like things like you know the previous customers are more like query correctness durability scalability and now like the newer customers talk about you know this two factor is not working you you know, like, I have this Ruby program, I have this Rails thing, the even the language is very different. Yeah, it does change. Oh, I think that that's a great sign, though, because you're basically attracting a lot of people who traditionally would have never even thought about using something like Vitesse or PlanetScale. So that's a good thing to hear.

Yeah, yeah. Anyways, well, thanks so much for being a guest. something like Vitesse or PlanetScale. So that's a great thing to hear. Yeah. Anyways, thanks so much for being a guest. Yeah, thank you. This was a lot of fun for me. I love that drip down memory lane and understanding how the product and stuff also works. And yeah, again, thank you so much. Yeah, thank

you.