Postgres FM - Xata

Episode Date: July 31, 2026

Nik and Michael are joined by Tudor Golubenco, CTO of Xata, to discuss their architecture, progress, and open source tooling. Here are some links to things they mentioned: Tudor Golubenco https://postgres.fm/people/tudor-golubencoXata https://xata.ioXata is now open source https://xata.io/blog/xata-is-now-open-sourceDBLab Engine https://postgres.ai/docs/database-labpgstream https://github.com/xataio/pgstreamXata acquires Privacy Dynamics for advanced anonymization https://xata.io/blog/xata-acquires-privacy-dynamicsTonic AI https://www.tonic.aiA thousand Postgres branches for one dollar https://xata.io/blog/a-thousand-postgres-branches-for-1pgroll https://github.com/xataio/pgrollCloudNativePG https://github.com/cloudnative-pg/cloudnative-pgPGSimCity https://nikolays.github.io/PGSimCityDeltaX https://github.com/xataio/deltax~~~What did you like or not like? What should we discuss next time? Let us know via a YouTube comment, on social media, or by commenting on our Google doc!~~~Postgres FM is produced by:Michael Christofides, founder of pgMustardNikolay Samokhvalov, founder of Postgres.aiWith credit to:Jessie Draws for the elephant artwork

Hello and welcome to Postgres FM, a weekly show about all things PostgresQL. I am Michael, founder of PG Mustard, and I'm joined as always by Nick, founder of Postgres AI. Hey Nick. Well, Michael. Hello, and we have a special guest today, Tudor Golbenko from Zeta, CTO there. Really nice to have you, Tudor, thanks for joining us. Hi, Michael, hi Nick, hi everyone. Thank you so much for having me. It is our pleasure. So to kick us off, I wonder if you could give us a bit of the background story of Zeta and take us through to where we are today.

Yeah, so I will actually start with what it is today because we did change once like a pivot or like a, I call it more like a technical pivot. So it's a Postgres platform, which has as differentiating features, a very fast copy on right branching, so it can create a branching in a matter of seconds even for databases that are terabyte in space.

sizes and scale to zero, which means that if a database is inactive, the compute part shuts down and then on the next connection, it automatically wakes up. And this wake up is very fast to target in one second. So that makes it very convenient, especially for developer workflows. So these non-production use cases, things like if you want a Postgres branch for each pool request, or if you want to create a sort of an ephemeral database for each agent run or for your agents and so on, these are our more common use cases. But on the Zeta platform, we can and do run production workloads as well.

So the differentiating features are on the developer experience if you want, but you can use it as a normal Postgres hosting provider as well. So that's what Zeta is today. It didn't really start like that. The initial idea, five years ago, my partner, Monica Sarbu, she wanted to do a database that's really easy to use. It was a lot of inspiration from Airbase or spreadsheets and because they were like a developer using Airbase as a database or Google Sheets as a database. So the initial idea was to make one that's as easy as to use as a spreadsheet, but that scales up so you don't have to switch at some point. of having features from both.

And we did that for a while with some moderate success, but at some point we decided that it's pretty hard to build a long-term business on that audience and never really took off with the primary database for more production. This is used as teams and things. I was just for interrupting because you're packing a lot of stuff in very short time. I just wanted to pose a little bit here. And this reminds me BitTio, right? They also like drag and drop CSP file.

You don't think about schema or anything, but it's Postgres behind it. So you had a similar thing, right? Yeah, but B.O were actual Postgres. Hours, we were actually hiding Postgres behind our own APIs. So it was a Rest API that you interacted with data, a bit like Postgres or something like this. nicer, I would say, but that type of interfaces.

So I like this idea a lot, but Bitio was closed. I remember like a very short note, like two weeks notice or something, people were like outraged. It was yet another, not fully postgues platform, but I have something, I store my data somewhere, but then it's closed and I have very short notice to grab my data. And you're telling us right now that also this model didn't work well, right? Yeah. Yeah, yeah, like there's definitely companies that do, that address this sort of audience successfully. Like I'm thinking of Superbase, for example, right?

But they have like millions of developers and such. And out of those millions, you're going to have a small minority that will become large databases. So they will pay a lot and so on, right? And we had a few tens of thousands. I think at the peak, we had 50,000 users. Turns out of those there's not enough paying customers to sustain such a business. Thank you for sharing. This is super interesting. 50,000. That's actually a lot.

Yeah, yeah, it was quite a lot. We had the growth in a number of users, but that didn't reflect in the growth in revenue. Many of them are not paying. Most of them are not. I feel you well. I understand this problem also. kinds of businesses. Yeah, it's hard to find business model on this. Unless you're super based. This is a good comparison. Thank you so much. It's super. Like before this recording, I said we were going to focus on technical stuff, but it started very different.

Very big. Thank you so much. I think it's really useful context to then dive into the technical side, because I remember one of your posts that I thought was really good about the architecture, which I guess might have been the architecture all along, but it explained how you could offer a free plan and how the free plan didn't actually cost you very much because of your architectural choices. And that was a really nice post from the business perspective,

but also from a technical perspective, like understanding how those tradeoffs actually help you as a business. Yeah, so on the new platform, we don't have a free tier. This is, again, the business side, and our experience with the old platform. So we call the old platform Zeta Light. It's now closed. We migrate the users that we could and so on.

But it's now close, so now it's only Zeta. But the old platform had a free tier, and when doing the new platform, we were not very eager to add another one, although with our current architecture, we can do it again, because we have this scale to zero, which is also very efficient from that point of view.

But we're not very eager because it adds a lot of noise, and like I was saying earlier, you have some growth, but as long as if that doesn't really result in the type of business that you want, you get a bunch of feedback that's not really useful. And it confuses things a little bit more on the business side. We might come back to that decision and maybe eventually we'll add again a three-tier plan, but currently we don't have one.

Yeah, let's shift to more technical stuff because I know you use cloud native PG, you have branching, you have very interesting open source tools like PG Stream developed and others, like pger roll right i'm very curious to talk with you about any of them yeah maybe we start at the zeta store it's our storage engine that makes a lot of this possible and then we can go about how we use cloud native pg to get the very fast wake-up times that's also i think interesting copy on write and branching yeah yeah so we do copy on right branching but we do it at the underlying volume level. So because our feature set is pretty similar with neon right also like copy and right branching and then scale to zero with fast wake up times this type of thing.

Cup of storage on compute right? Yeah. And you recently made everything open source. Apache 2.0 or not quite everything but enough to run the whole platform yourself. But it won't scale to the same levels that we scale our own platform. and you won't get as fast times. So with our platform, I can go into more details about exactly why. But we have open source enough to be running it yourself and to be useful, but it's not quite everything open source either. Cool. So if we talk about storage, like what's interesting there?

Yeah, so Zeta store, we call it like this Zeta store, and this is like one of the proprietary pieces. And we have quite a long history into that. that. We tried a bunch of products and Kubernetes tools for managing the storage layer. As in the end, we decided to build our own, but also use a bunch of ZFS functionality, so it's based on that. The way it works is I'm not sure how familiar you are with ZFS. Let me interrupt you here. My company, PosgousaI, has database lab engine, which implemented this branching in 2019 before Neonon and before

data, right? And I have a lot of questions related to your experience with database branching and how you convince users that it's really needed because we do it seven years and some of them are solved easily, but some of them still struggle and like, oh, production data, we need to change it and so on. Let's talk about this separately, but I just, to give you a context that you probably don't have. And it's on ZTFS. Oh, are you using with ZTFS? I thought you are using BTRFS. No, it wasn't stable when we started, and I don't know it's terrible now. Okay, I was actually remembering wrong then.

It was actually interesting. When I created it first, it came out of experience with GitLab. We just needed to give backend engineers capabilities to experiment and see explain plans, not touching a few database experts, right? And it was great, it worked great, and then we scaled it. And the problem always was that first of all, production data, how to protect this, right? And also when new started working with branching, it's very different. So we do it very locally and we achieve static cost for all clones and branches.

So you pay once and you have everything like multi-tenancy in one virtual machine, basically. You have very different model. So I'm very curious how it works and what's your, what do you observe, how people use, it. And because my honest take, I spent a lot of time in this area, market is still not understanding that branching is super powerful. And I hope AI will change. Yeah. I think it's growing pretty nicely. But yeah, like the use cases that we see most often are branch per PR. So they do PR and have some sort of preview environments. This could be like VERS or their own staff or different other platforms for ephemeral environments or preview environments.

But then they want also the database to be part of that with the data, right? So then a platform like ours makes that possible. You have the database with the data. It doesn't have to be the exact production data. And the way we typically recommend people to do is we tell them, look, keep where it is, let's say it's RDS. And then we have, this is why we develop PGStream. We create a replica of it.

And the PGStream can copy the data once. So for small databases, we recommend people to set up a nightly job or a weekly job or something like this, because it's simpler. For large databases, PGStream can also do the initial snapshot and then keep it up via a logical replication. And the deal. And ddl, yes. And you solve the deal.

I looked at it briefly and I'm very curious like how if you do it and what's your scale? Is it working really well? If it's working really well, why it doesn't go to core of post-gars in that form? Yeah, I think there's some work in progress to make it in. There are several ways. And if it's not super progress, it's not progressing. Not progressing, yeah. Actually, a little bit of backstory on PGStream. We were doing in the old product, so Zeta Light. We were doing, we are offering this Rest API and most of it was powered by Postgres. But we also had a free tech search with BM25 and relevance and so on. And that was actually served from Elasticsearch. So we wanted to have this replication between Postgres and Elasticsearch. And we could do, because we had like our own API for modifying the schema and such, you could do it in both at the same time.

But that's also tricky because if you replicate data, but the schema changes come out of order, then again you have a problem because it's not going to fit the schema, the classical CDC type of problems, right? So it's actually better if the schema changes are in the same replication stream as the actually. actual data, it's better that way. So that's why we did this event hook approach, listening for schema changes, and then apply them that. There, with Data Light, we had this advantage that we, you couldn't do arbitrary schemas in Postgres. It was a particular set of data types and so on.

It was a little bit more into our control. So we knew we could test all possible schema changes and make sure they replicate correctly and these sort of things. But then we generalize this approach a little bit more. Yeah, so that's interesting. So you have a limited set of what's happening, right? And then using event triggers. Yeah, yeah.

Yeah, that's our approach. And of course, we need to make sure it's in the right place in the replication stream. And the way we do it is we write in a kind of in a shadow table, we write what has changed and with what altered the same of the same of that happened and that gets put into the logical replication stream and then PG stream reads it from there and applies it on the target in the right order as the general approach. I'm very curious at what scale it is working and are you fully satisfied or there are some like plans to improve or something. Yeah we have people using PG stream at like four or five

terabytes of data and the initial snapshot takes like 24 hours or something like this and then that depends on a number of factors or something like this and then it's kept up to date but then it like it it's from company to company and sometimes we have to work on pg stream a little bit to make it work for them but in the meantime we had we it had enough battles let's say that now we're feeling like it covers already the majority of cases pretty well battle proven already. Yeah, to some degree. Like, they are still problems.

This CDC logical replication is still quite a play. I would love if DDL would be supported in core. Everyone waits for it. But it's super not trivial, as I saw last time when I checked this topic. That's cool. Like a small non-technical side note, side question. These pieces are, like, you mentioned some proprietary stuff, but PG role and PG stream and others.

Yeah. open source. Yeah, that's an Apache 2.0 right? Yes. Yeah. Yeah, and especially PG stream, it's important for us to be open source because we recommend people to run it in their environment, because it also does anonymization. This is the, you know, like kind of what we started from. It's important to do anonymization so that when you do this copy on right branching, the data is already anonymized and you have branches, very fast branches that don't contain the sensitive data. Like it's up to you what transformation rules you set and so on, but in many cases, like 80% of the data is fine as it is. And you have some data that it's sensitive and you need to transform it and make sure it's not in any developer copy and so on.

And this, I think, is the perfect balance of things if you want. But it does depend on this replication story. And we want to run it in their environment so they have full control over it. That's why it needs to be open source, really. Yeah, the answer is my question. On the anonymization front, I saw you acquired a company. I saw some blog posts, I think you wrote, was the beginning of this year.

Is that what you've then rolled into this? Yeah, partially. Partially, it's called Privacy Dynamics, this company, and they were like really deep into the anonymization front. It's like there's levels to this stuff. In PGStream, we already had anonymization. which was simple column rule base. Like you can say, okay, this column,

you can hash it like this deterministically, and it's hashed. But then, like for example, for HIPAA compliance and so on, you need to do the type of anonymization where, if you, let's say there's patients, they die and you have a woman that's 46 years old and so on and has this email address or something like this. And if you just anonymize the email address,

but it's the only woman in the data set with those characteristics, then you haven't really anonymized anything because you can, like even if the name is not there, there's enough information, unique information there, that you can still walk backwards what it is. So they're doing these sort of things where they see exactly,

combining the data set into groups and figuring out, okay, we need to anonymize this data so that no group is really small. Every group has more than 10 data points and such. So you cannot find the individual. And this is more like HIPAA compliance sort of things and such. So that's not, that's also something that's not in PG stream at the moment. Maybe we would roll out more of those features in there. That we set up more like manually. for customers that actually need this complex stuff. Yeah, that's great. So these two pieces are, they're interesting themselves. They could be separate episodes, honestly. This is what I do.

Because if we dive deeper, there are so many levels. As I said, I'm in that space as well. We usually use some other tools. Like we didn't build and we, I remember also there was a company Tonica. I'm curious what happened to it. Also, they still exist, I think, right? Yeah. So if you look to what they build, you understand that so many layers go there. Yeah, yeah, absolutely. Absolutely. Yes, and Tonic AI, I think they still exist, but it's just proprietary. And they open source like a simple version of it, but it doesn't scale at all. And the important to also remember, like use cases are very different for branching. If you just want to, like you said, you don't recommend to take production and if it's like simple development,

it's better to have some like preview environments. It can be some synthetic data, for example, right? You don't need to synchronize it. But if you bring data for a various, like closer to production testing, and especially if it's related to performance testing, I was always concerned about what plants we will be getting. Because if you change data, I stop trusting those plants. Yeah, of course.

So I usually try to convince people to keep like red data as is, production data and just limit this and consider it as part of production and still have branching and test it closer to production. Because this is the only way you can have really realistic performance testing. What you think about this? Yeah. So there is this deterministic hashing, which basically means if you hash a value, it will always hash to the same thing, which is also useful for referential integrity. Let's say, like if you have a full, pouring key on a string column or something like this, then you want it to anonymize the same in both places.

So that helps a little bit with this. But yeah, with the Postgres planner, there can be a lot of nuances. Like as I said, this could be a whole episode. If we dive deeper, like when you say hashing, okay, Pige stream takes data from RDS, understand this. You have ability. Like you copy anyway, right? But if you talk about your own platform, you have physical data.

the directory. Right. Right. Yeah. When you touch it with any update, with anything, it's better. Like you have, we have thing cloning, branching on physical level. And if you start touching it, you already will, Michael will tell better what, what will happen with plans because he builds PigeMaster, right, which is focusing only on the plans. So if you physically change something, you like different number of blocks and so on, it's already on plans, right? Yeah. can relative apples numbers will become different statistics might start different even if you don't if you say update some row set column equals the same column ID something logically nothing changed but physically changed and eventually if it's master scale it can drift plans

yeah and logical replication already like kind of throws that way because it's right but if it's if we go back to ztf s and thing cloning the copy and right yeah my point was always always like let's not touch the data and we will have ability to have the exact same. Yeah, and you can still do that. You can have like if you run production on ZAPE, what we typically recommend is have a prod projects and then something like a staging project or something like this and PG stream between them to do the anonymization. But then the prod project still supports copy on right branching and that would be like ZFS clone type of thing. And you can use that. But you would only only use it when you actually need it for like performance. And like I see Michael wants to ask something, but I will ask very last question because I sit in this topic seven years already.

So I like I'm super curious to find guys who have like also spent time there. Last question from me, unit economics. If we want to build like pipelines which will test performance plans, right? If we keep data and Zeta production. build this pipelines will each every CI pipeline lead to provisioning on the whole node will i be paying separately for whole for frm 10 or 100 pipelines you need to pay for everyone right because we do this separation of storage and compute right and on the storage side there's this thin copy right so you only pay for the difference that's on the

so it's basically doesn't change if i don't change data it's the same price i don't Right. With, of course, some gachas there as well, but at the high level, let's say that. And on the compute side, we use Kubernetes, so that they will be, they will be starting on different compute nodes, potentially, but not each new branch will be a new node necessarily, right? They will fill existing nodes and such. And there you only pay, if you use our Zeta cloud platform, because you also have bring your own cloud, which is a little bit different. But if you use our cloud platform, you only pay for the time the branch is active. And with this scale to zero, for development, it makes a lot of sense because it wakes up in a second when you connect to it. And you can set like a five minutes expiration time.

This is like a lot of our customers do. It's like you pay almost nothing. I actually have a blog post. You can run a thousand. It's just how the math works. you can run a thousand branches for one dollar basically a tenth of a cent each if they run for five minutes yeah that's great so this is much better than what i observe with thing cloning at up in aurora in new right because there is absolutely it's linear scaling if you want if you want many

pipelines in ci to test you amplify your spending budgets and these limits this is super important because this makes engineers to stop testing enough and they start. Yeah, yeah, yeah. And it discouraged to have a branch for HPR, which is like, yeah, Oh, that's like that's cool. As I understand it's not like a constant price, but definitely you've thought about this. And because this is super important to make testing more like intensive and complete. That's cool.

Yeah. Yeah. Yeah. So definitely neon, like you get 20 branches for free. You still pick the compute. They're not completely free. You pay for the compute, but above 20 branches, you pay a fixed amount for each of them, even if they are inactive. So we don't do that. We don't charge that. That's why you can have a thousand branches with only $1, which, I don't know, maybe in a year we'll have another call.

And I will say that it doesn't work economically. You said there are nuances. I know exactly which nuance. If you have some branch or clone or like it holds a snapshot and if production way to the future, you keep you start paying for all data to be stored, right? Yeah, exactly. Or it's some, yeah, like it's sometimes hard to explain because you create a branch, its cost is zero on the storage. And then you delete all the data into it. And then you would expect, you wouldn't expect the cost to go up.

but that's actually what happens because now it's different from the parent, right? So it's like, hey, why am I paying more by deleting data, right? That can be confusing sometimes. Just as exactly what we also had and still having. Yeah, that's cool. Cool. Yeah, Michael, I apologize, so you had questions. No, I was going to ask more, but actually now I'm thinking,

I think this is great in terms of using Zeta for your development. environments, let's say like pre-production, those early, maybe per agent, per branch, that kind of thing. But I think you've also got some interesting stuff on the developer experience of running post-gris in production, like PG-Roll, for example, but I suspect you'll have opinions on a lot of the other things that are currently quite hard to do as a developer running post-gris in production. So I wondered about getting your opinion on some of those. So the big one for me, I think still that's difficult for people to manage is major version upgrades. Yeah,

that's like things like that. I wonder if you have any takes or any interesting features that most managed services don't offer or things that you want to do in the future to make things like that easier. Yeah, we did put a lot of effort into the schema changes part of it. It felt to us like one of the biggest pain points in Postgres. There's a lot of gotchas there as well. Like you do. schema change and it works on your small database that you test with, but then you run it in production and it locks the whole table for a long time. So then you have to do this process and orchestrate multiple PRs, change the application, do that, do the backfield and so on.

So that's how we like how we started PG roll. It's also related to the old platform, which again, it was all about making the database as easy as possible to use. So that's, Then we had an API to do schema changes and that was not as we couldn't do as much as Postgres can do, but everything that you did there was guaranteed to be safe. You could never lock your database and such. But then again, you hit the problem that people want to do those things. So it's like that Postgres functionality was not added for nothing and then you end up having to support everything, which was, which was our problem. So yeah, PG-ROL is like, it's an attempt to to make that as simple as possible.

It's working. It also has some gachas that currently we're not super focused on it because we're focused primarily on the Zeta platform itself with copy and right branching and supporting many agents and these sort of things that's where our focus is. So we didn't have that much time for PG role lately. Yeah, that's where we are. So now focuses to support to give database to agents, right?

right? And that can be in two ways. One is during development, like you give, like I was saying, earlier branches per PR. Now with agents, you create a lot more PRs and we're seeing like we're seeing companies that have two engineers and they have 250 open PRs and they're actually all making progress and such. We see people that that don't do reviews. It's only AI reviews and they get merge automatically. The tests are running. A lot of these things are. happening now and will probably happen more, right? So that's one. And the second way is like AI platforms that need a Postgres database as part of their product, right?

And then they want to spin up like an FML database for that application, which could be an experiment that they throw away. So then you want this cheap databases that spin up and down very dynamically and scale a lot, scaling to many databases if you want. Yeah, this guys even don't need H.A. at all, right? Just single node, no fell over needed. Yeah, exactly.

Exactly. Usually you don't need replicas for those use cases. Yes. Yeah. So no availability needed. Maybe just backups and that's it, right? Like you also feel this like new demand. Yeah. Yeah.

And we do take advantage of that because the way we solve this very fast wakeup times is by doing what we call warm pools. We keep CNPG clusters ready to go. And then when there's a connection for a hibernated branch or database, we take one already provisioned clustered from the pool, and we connect it to the right ZEFS volume on the Zeta store storage node, and then serve it to the user. And that's how it happens so fast.

But this means the clusters in the pool have to all look the same because if they don't look the same and they have different RAM and so on, that's obvious, but also different Postgres settings, if that causes a restart, then you did nothing because you need to restart it then to. So we take advantage of that. The cluster pools have no read replicas. They're very fast, but it is what it is. How fast it is?

So like we're targeting a second. Yeah, provisioning a new cluster is the simplest. operation because we just take one of the from the pool and give it as yeah also second but it's that most of the cause there is really the control plane we have a single global control plane region so we have to go to it and so on but that's fast the more the wake up is actually more interesting because there you cannot just take one from the pool and serve it you have to take one from the pool connected to the right volume and then serve it right and this is what the this is what the Zeta store enables and this separation of storage and compute to some degree.

And we have a bit of a trick that Cloud Native PG doesn't support any of this. So we are maintaining kind of our fork of it to enable some of that. But there's the instance manager in Cloud Native PG and we modified it to essentially wait for the PG data folder to show up. And until then it's waiting. So the pod is completely provisioned. It's just waiting for that folder to show up before letting Postgres start. And behind the scenes, we mount it in the right place.

And as soon as that shows up there, the instance manager will let Postgres start. And Postgres starts in 300 milliseconds or so. It depends as well if the checkpoint was done and things like this. But if it doesn't have to do the recovery, it's actually really fast. But what's expensive is provisioning the whole stuff in Kubernetes around it, the pod and the innate containers and these sort of things. That's where the time was going. So that's why we do all that thing. And then the connection, the work that we do at the connection time is actually minimal.

That's why it's so fast. That's impressive. That's cool. I have a tricky question. Oh, okay. Michael. I maybe have the same tricky question. I wondered about your, in a development.

environment, you probably want the same extensions as production or you might need them in order to test various things. And also you might have changed a lot of configuration parameters. I'm wondering how maybe it's not quite relevant to the same pooling system. But does that that system you just described limit your use of extensions and changing of configuration? I just want super user. Okay. Two different questions. Let me start with the one from Michael.

Yeah, it does because the problem is you cannot keep a cluster pool. This is how we call them. You cannot keep a cluster pool for each combination of like you have instant size and then you have a postgres setting or list of extensions and so on. Preload libraries with more relevant because the extensions, they can be on the image. you just don't enable them or enable them. That's fine. That's not a problem.

But if you need a different set of preload libraries, that is a problem. So we just chose to enable more common stuff in the preload libraries by default. And then you have to keep in mind that if you are not on the happy path and we don't have a cluster pool for you, it still works. It's just slower. It will take a few seconds then. six to 10 seconds instead of one if you're not on the happy path. And then the other thing is that we haven't done yet, but we want to do, which is auto-scaling for the cluster pools,

because some configurations, if you want, are going to be a lot more common than others, right? The default settings that we have, you would do a cluster pool of 100 waiting to go, right? so that you can do 100 at once and you don't it's not overloaded. If you have a more complex one, it could be just one is enough, right? Because you start that one and then you immediately provision the replacement for it. That will take 10 seconds.

As long as someone else doesn't do the same operation in those 10 seconds, you're okay. So it's enough to keep one around. But making this via static configuration definitely annoying. So that's why we want to do some sort of auto-scaling based on that. So depending on the demand that we've seen historically, we would make these pools smaller or bigger, and that would automatically optimize costs for us. That's something that we haven't done yet, but want to look into it. Yeah, super user is like, I happen to know your opinion on that.

We don't give super user to our user. get a good at least yeah the issue is because you run on Kubernetes it's pretty hard to secure it if you can run copy program let's let's wrap up this question i have much more interesting and we don't have a lot of time so question to you tricky one if you started this platform to build this once again right now from scratch would you choose kubernetes still we do use a lot of stuff from kubernetes and it's good yeah i'm happy with overall. That's most important.

Okay, cool. Are you happy with Cloud Native PG? I'm, yeah, I'm like that's more on the fence, I would say. You know my opinion about it? I know your opinion about it. Yeah, and more is coming actually. Literally this weekend I didn't work only on this PG, the SimCity. I worked on...

Oh, that's awesome, by the way. Yeah, thank you. It was a simple one prompt. That was not expected. But this thing, like, what's happening? We have a winning leader among Kubernetes operators, which is completely like ignoring the facts like about split brains and data loss and so on. Do you use synchronization if you have multi-node setup? It's configurable.

It's configurable, but it's not on by default, right? It's not on by default. Which means I have demonstration of split brains which is still possible and multiple. in multiple ways. Tudia, do you mean you let your users configure it? Yeah, we can enable it for them. And in the cloud needs to do it's not enabled by default.

It's configurable. It's the same. Yeah, that's the problem. And so haven't you already like maybe I'm over like maybe we don't need proper H.A. Speed rate protection and so on. Like maybe this is like it's okay to run this thing. And if your users, if you don't know about your

users suffering at your scale, that this is good enough already, right? like it's okay what's your take on this no I think it's definitely a problem that like I'm yeah I'm also a little bit disappointed by the response to that issue I was expecting like to be closed sooner I understand it's tricky because in CNPG there's no there's no proxy layer so they rely on Kubernetes services to expose things and to move them around that makes it I think a little bit more complex for them compared to Patroni for example the funny thing is we do have a gateway it's not complex it's impossible it's impossible you can there is no consensus algorithm used

there is only quorum and there is mixing of concepts like quorum is needed to handle transactions it's not it's not for deciding who is the primary so i have demonstrations you can have multiple you have two primaries you can have split-dain you can have data loss and more to be published soon. I'm pissed off because of their popularity. Clients keep asking us. I literally have right now two asks from two different companies. So explain, should we use CloudyspG?

Because it's super popular. And now, like, I meet you and you use it at scale. So you don't have problems. Nobody lost data. Nobody dealt with split brains. Do we have, maybe we have already good cloud resources. which don't experience like outages, like maybe this is it. Maybe we don't need the H.A. replicas, maybe even, right?

Like, it's already good enough to live with a single node. So there's not such a problem as H.A. High reliability. No more such problems. Or you had such issues, like split brains or data losses, because there's no consensus algorithm. Yeah, like I would have to go back and look how often. Like, I do remember maybe one case where, to be honest, we couldn't figure out exactly what happened. In these things, there's always a combination of factors.

But I do remember once we thought, like, this might actually be the split-brain scenario hitting us. So, like, I think it's a bug, really. Unfortunately, in their DNA, like, this is a basic decision. If you read recent posts, I have an answer unpublished. I'm going to publish it. So more stuff is coming. It's coming. And I sent you LinkedIn requests.

Please accept it. Let's keep in touch on this topic because I want the industry, understand that consensus algorithm is needed. And either Cloud Native PG needs to be to change or people should stop using it. Or maybe we're like again like one node is enough. We don't need the H.A. Everything is good. AWS and Google Cloud. They stopped being unreliable as it was 10 to 15 years ago. Yeah. On our side, we're already, I told you we were on a fork of it.

It's not that I would not recommend people to use our fork because it's open source as well, but we didn't make it with the goal of becoming like a CNPG successor or anything like that. But we did go a bit in a different direction when it comes to backups. For example, we're using PG backrest and we made it in the operator itself rather than the plugger. solution which has a number of disadvantages. So we have the like we have the precedent of us going a little bit in different directions regarding these things. Also we do have a gateway anyway. We need a gateway like a SQL gateway. So then like the main benefit that you are proxy less actually doesn't really apply to us. So it might that might be also something that we would

choose to do a little bit different there. Yeah, and by default, Cloud NewsPG has a barman or for backups? It's a plugin. It used to be barman in the core, but then now it's barman as a plugin. I think as of right now, it's the only, though, plug-in that exists. I think there's a one for PG-Bgress pattern. So if you replace and if you have PG-Bcrest, this next step is just to add Patroni. Yeah. And then my concerns are fully resolved, I'm a call.

So algorithm works well and so on. But I understand it can be super challenging at your scale because if you have a lot of smaller clusters and so on, ETCD or something, like you should, you need to properly engineer it. I think it's done by some guys, but it's not straightforward to not like because we don't, if they have a lot of small clusters, we cannot afford a TCD for everyone, right? We need to have shared the TCD and there are some. Yeah, yeah. Yeah, exactly. Exactly. But it's like it could be natural step.

Next time, like you bump into a couple of other weird cases and maybe you should add Patroni. Yeah, yeah. But that's like there's the advantage that we don't really do this yet, but it does make sense to have a bit of different configuration for your development branches versus your production branches. Because the thousands of branches and so on, the really large numbers, don't even have replicas like you're saying, because that's like the point is that you shut them down,

they come back up, the storage is separated and it's secure no matter what. It doesn't depend on Postgres at all. So then you just don't use read replicas at all for those use cases. And the production ones are just much fewer and then you can. And computer is smaller, right? So like less, shed buffers are small, everything like this. Right. But then you can fool a planner and make it work to choose the same plans as in production.

This is magic I realized some time ago and it's super cool. That you can have a tiny, like memory small, shared buffers is tiny. But you say effective cash size is huge. Other planner parameters are huge, some other things. And you get exactly same plans as in production. Oh, that's cool. Yeah, and then you need to just adjust, you need a small mind shift and engineer minds to make them focus on buffers numbers more because timing might be different. This is what we talk about buffers, three years already or four years.

I'm super conscious of time, but Tudor, is there any last things or shout-ups you wanted to give on the Zeta side? I'm very aware we haven't even talked about Delta X, which looks really promising. Maybe we do another one in a few. Yeah, let's have part two quite soon. Yeah. It was cool. Thank you for coming. I enjoyed a lot.

I think we do similar things like in terms of where we had it. I think everyone should use database branching. It's super cool. We will have all the links in show notes, right? And so I encourage everyone to try out if haven't. And I guess you just need to tell your agent to try to go and use it, right? That's it. Yeah, pretty much.

Yes. That's cool. Thank you for coming. Let's have a part of this. Yeah, thanks so much, Judah. Thank you. It was really nice. Thanks both.