Postgres FM - Decoupled storage and compute

Episode Date: August 25, 2023

Nikolay and Michael discuss a listener question — about products that take Postgres and transform it to something that decouples compute from storage (RDS Aurora, GC AlloyDB, Neon etc.) and whether they see something like this landing upstream in the medium term.Here are some links to some things they mentioned:Amazon Aurora https://aws.amazon.com/rds/aurora/Google Cloud AlloyDB for PostgreSQL https://cloud.google.com/alloydbNeon https://neon.tech/ Google Cloud Spanner https://cloud.google.com/spannerIs Aurora PostgreSQL really faster and cheaper than RDS PostgreSQL? (blog post by Avinash Vallarapu from MigOps) https://www.migops.com/blog/is-aurora-postgresql-really-faster-and-cheaper-than-rds-postgresql-benchmarking/ Deep dive on Amazon Aurora with PostgreSQL compatibility (presentation by Grant McAllister) https://www.youtube.com/watch?v=HQg8wqlxefo Intro to Aurora PostgreSQL Query Plan Management https://aws.amazon.com/blogs/database/introduction-to-aurora-postgresql-query-plan-management/ Michael Stonebraker Turing Award Lecture https://www.youtube.com/watch?v=BbGeKi6T6QIInterview with Stas Kelvich from Neon on Postgres TV https://www.youtube.com/watch?v=4PUKNznq_eM Interview with Ben Vandiver from Google Cloud Spanner on Postgres TV https://www.youtube.com/watch?v=BW-Uexhv-bk Timescale Cloud bottomless storage feature (data tiering to Amazon S3) https://www.timescale.com/blog/expanding-the-boundaries-of-postgresql-announcing-a-bottomless-consumption-based-object-storage-layer-built-on-amazon-s3/ Testing Database Changes the Right Way (Heap Analytics article) https://www.heap.io/blog/testing-database-changes-right-way  ~~~What did you like or not like? What should we discuss next time? Let us know via a YouTube comment, on social media, or by commenting on our Google doc!~~~Postgres FM is brought to you by:Nikolay Samokhvalov, founder of Postgres.aiMichael Christofides, founder of pgMustardWith special thanks to:Jessie Draws for the amazing artwork 

Hello, hello, this is PostgresFM episode number 60. My name is Nikolai and together with me today is Michael, as usual. Hi, Michael. Hello, Nikolai. And we are going to talk about what? Serverless or separation of storage from compute? Or how do you name it? Well, I think we had a really good request and they used the wording decouples compute from storage. So maybe even though I do like the short titles for the images for this podcast, I think that probably is more accurate to what we wanted to talk about. Not serverless. I mean, I think serverless is part of this, but some of the other providers, I'm not sure.

I think there are providers decoupling compute from storage that you probably wouldn't class as serverless. Okay, I'm lost in this new beautiful world. I'm an old schooler. And I have my skepticism, but I also try to keep my mind open. So I'm trying to understand what's the hype about and what is it, why do we need it, and so on. So where should we start? Great. Can I read out the question we got? Because I think that brings it in quite practical terms. So what do you guys think about all these products that take Postgres and transform it to something that decouples compute from storage. For example, RDS Aurora, Google Cloud Alloy DB, Neon, etc. And then the follow-up to that was, do you see something like this landing upstream in the medium term? To Postgres itself.

Yeah, exactly. I'm not 100% sure what they mean by how that would exist, but I think they're quite interesting questions. And they've basically asked for our opinions I think on those providers and anything else we had class in a similar category I think that's like we've each had customers on at least one of those I know AuroraDB is the oldest so that's the one I've seen in the wild but yeah I've definitely read the marketing materials of the others and seen people playing around with them seen some people excited by various aspects of some of them so i think there are some compelling at least marketing arguments and it's an interesting future ahead and then there's also the question of which parts of this if any could be upstreamed. For example, I know the Neon folks are working to make Neon

as little of a fork as possible, trying to get some of the code upstreamed so that what they do is less custom. Same as AureolaDB. Yeah, exactly. So I think that's how I interpreted this question. So yeah, how does that sound? Yeah, that sounds great.

But let's discuss what is it. So let's talk about what some of these providers offer, or at least claim. And there's a few things that I'm not sure about all of them, but the things I keep seeing come up are around performance, around scalability, and they're around things that aren't necessarily always that easy to do in Postgres. Like, for example, major version upgrades. So things that you can do potentially without the same consequences or the same downsides. So, yeah, again, I've seen some good arguments that maybe the performance claims are slightly overstated by some of these providers potentially not even the case once

things are tuned differently and there's a really good blog post for example by the team at migops that i found interesting comparing let's discuss like yeah what is separation so if i keep pg data on ec2 on ebs volume which is network attached cloud disk and have ec2 instance i can upgrade this instance just very quickly well not maybe very quickly but i can do it If I run containers on that EC2 instance, or maybe if I have a layer or something, and for example, I get metal, and I attach EBS volume, and on metal, I have multiple,

I attach many volumes, and then I have, using Firecracker, micro VMs, which can be provisioned very quickly, and then I distribute load of my huge instance like micro VMs, which can be provisioned very quickly. And then I distribute load of my huge instance among these smaller guys. I can change this micro VM very quickly. Is this separation? Why do I need to think about any Postgres changes at all?

I have a network attached volume, storage is already separated. What else? Why? Like, what are we talking about? What kind of separation? This is a question. What your materials tell you? What do they tell you? To be honest, a lot of them gloss over this quite quickly. So I'm not quite sure what would count. And maybe that would count by strict definition. But I think a lot of these providers, like let's say Amazon with Aurora, Google Cloud with Spanner and AlloyDB. Those two, for example, have invested a lot in their cloud infrastructure and specifically storage so they've got very good at replicating storage across regions with redundancy with with being able to you know put data close to users um and

that part of what they've invested in is at the disk level you know that's storage layer now they've had interfaces to those things that aren't postgres in the past like aurora for example started well i don't know if it started but it definitely had mysql compatibility before postgres compatibility so it's added postgres compatibility later so that for me is like a is one way of looking at it so they already had this this storage infrastructure that they had invested in and they wanted to put a different interface on top so that they could support people's applications that already had right but what i understand about this approach

they store my data directory in object storage so s3 right and then we have we need to but it's if you just attach it, there are some there are some projects like that can if you take your S3 bucket and then you attach it as like it looks like a disk in your Linux, it's possible to do like with fuse or something like it's possible, but it will be slow and bad and so on of course in this case if we want to to keep our data directory in s3 and we need to work with it with good performance we need also to cache it so we need to cache it on regular abs volume and so on and i as understand this is what Aurora and Neon do. I might be mistaken here, like architecture. I'm not an expert here at all, but let's discuss about features for users, of course, definitely.

If bottomless, serverless, autoscaling, what kind of features? We have branching as well, right? So we separated. I remember Aurora claimed one of the biggest benefits they achieved is lack of the need to care about checkpoints. So checkpoints are fully detached from our compute node. They are done in the ground, we don't care about them. So we don't have penalty from checkpoints anymore. We still have vacuum, unfortunately, in Aurora Postgres, but we don't have checkpoints.

And this is good in terms of if you have right heavy workloads, it's much better. So performance doesn't drop when checkpoint happens. Latency doesn't spike. This is the benefit. What else? Well, interesting. I actually didn't see that at least on the front page.

So that's super interesting. But Aurora, well, it was the original presentation on all Postgres conferences from Grant McAllister, engineer from Aurora RDS team. He explained it very well. I recommend it even for those who don't use Aurora

because he explains problems with checkpoints and full page rights and details and then how they solved and not all technical details are clear there. But it's good for understanding of Postgres. These talks are good for understanding Postgres as well. And it was like five years ago or so. And yeah, so they don't have checkpoints, they eliminated, they don't fully eliminated them, because they need to replay false. And so like, they need to take care of checkpointing still, but it's as I understand, but it's happening

fully behind the curtain. So our compute node doesn't see any overhead of checkpoints. I think by this moment, experienced engineers already understood that we are not experts in this at all. Of course. But we are trying to understand this. Again, I'm quite skeptical. I see that it adds complexity a lot, this approach. We definitely can discuss further, and we will, but we have the question in the end.

Do you see it will go upstream to Postgres? Let me explain my opinion right now. I think, yes, it has chances, but only after autofailover, puller, and other such things, which are waiting to be included to Postgres engine itself. Because, for example autofailover is all about multiple node clusters. So you have primary, you have replicas and all

autofailover is essential there. Postgres has improvements for example we discussed recently in libpq fresh improvements in postgres 15. I keep forgetting or 14 even this load balance feature. When you connect, you can specify how to load balance. So these features exist, but it's kind of like making Socket better working for other tools, like for Patroni. For Patroni, instead of including whole feature into the engine, the engine can be adjusted. So this feature, which is implemented externally using additional tool, the engine can be adjusted to work better. Same is with backup tools. There are improvements.

There is API for backups now and so on. Maybe it means that pgbackrest and wulg will be in core in future somehow. Maybe not. Still not clear to me. But what I'm trying to say, this separation might be seen as some adjustments, slight adjustments of things in Postgres,

but I don't see how it can come as a whole thing into Postgres in the nearest years. I think I understand. Are you saying you don't think it should be included before those things, or you don't think it's possible, it makes any sense before some of those things? Some adjustments can be made to separate storage from compute for example we know that ebs volume can be attached to multiple vms if you use special file system some amazon file system in this case you can attach ebs volume to multiple ec2 instances and this is this thing feels already less separation you can have i think they have snapshotshots. I'm not sure about full flash branching and so on. Yeah, maybe if you have snapshot and you create a new best volume, you already pay for tubious volumes. And this is not as Aurora or neon have they

have ability to have thin clone. And you don't pay, it doesn't increase the storage bill twice when you do it. In this case, probably it does. I mean, in the case of this Elastic file system, Amazon, I don't remember. This file system which supports attaching EBS volume to multiple instances. So money is very important aspect here, for sure. But how come Postgres can support any of these things? I don't see, in the nearest years, I don't see it at all. The only way I see it happening, they even said in the medium term.

So I guess it depends what you mean by that. The only way I see it happening is if some of these providers, like your Neons, get a lot of support for contributing what they want to do maybe as hooks or maybe maybe making things extensible rather than it being in core so it might be i don't know if that's the same thing kind of in practice but that's the only way i could see some things happening i can't see the current postgres core devs prioritizing this as a feature partly as you said because there's so many other things competing for attention. And actually, it's not that many people

in the grand scheme of things contributing to core. They've got relatively limited bandwidth and lots of good, important things to be working on already. Right. So this answers the last question, but let's talk about, again, let's talk about this. So what do we think about those products? I can tell you what some of my friends, fellow founders, other people, why they're picking Aurora, for example.

I'm not sure it's always the wisest decision for smaller businesses, but they like the appeal of auto scale up without downtime, without a lot of operation. Is it really without downtime? Because yesterday I tried Neon and I wrote them. I had the PgBench running on autoscaling Neon instance and every

time it scales I see errors. So maybe it's not polished yet. But this is hard. But also I don't understand why we need to separate autoscale again. This breaks my mind. Again, I am't understand why we need to separate total scale again. This breaks my mind.

Again, I am an old schooler. It's funny, but considering EBS volume and EC2 instance and cloud, it's already an old school approach. Ten years ago, it was quite a new thing compared to on-premises setups. So I have EC2 instance, I have a BAS volume, and I can quickly provision another C2 instance. Imagine if I have pgBouncer. All connections go through pgBouncer. I have a C2 instance. What if I issue pause to connections and quickly reattach my BAS volume to a new instance and then resume and it has already more CPU and RAM yeah

but I think you're skipping over having the operational confidence to do that and the experience to know that you can even do that but that's not typically how people are self-managing Postgres and it's not typically how I'm trying to ask,

why do we need to cut Postgres guts? I mean, to replace legs and so on. I mean, there's a common saying like, Aurora, Neon, they replace the bottom half of Postgres, storage-related half. They rewrite it. Why do we need to rewrite it? Good question. So I think there's a few

things i think you mentioned cost and there's an interesting argument for i think people see aurora at least i'm not i may and neon actually this is one of their main selling points there's bit more pay as you go so the cost increases slowly as you grow instead of if you're on let's say rds you go up in jumps right you go from a smaller instance to a medium one to a larger one those aren't necessarily well I can see you shaking your head

again I can build managed Postgres service for example Superbase might go this route I don't know we can run it on larger metal instances in AWS and then we can use Firecracker to

provision micro VMs and then we have fine control about how much. We can run we can even run single VM in multiple containers and use quotas for containers in terms of CPU and RAM. Maybe even in Kubernetes.

So you're saying these things are possible, but that's not what people are choosing between. They're choosing between Amazon RDS, regular Postgres, and Amazon Aurora Postgres. Right. And when you're looking at that... RDS users can choose, right. Yeah, so this is the choice in front of people

that want a managed service and i think you could argue i think with with some merit that aurora is even more managed so they'll do even more even more things are possible for you maybe with a few more limitations aurora is definitely has aurora has definitely has some good things that could go to RDS as well. They could go to open source area, for example, plan management extension. But obviously AWS decided to keep it only for Aurora for competitive reasons. So this is clear to me.

So this extension doesn't deal with Aurora special features. It deals with regular... You know it very well, right? Query plan management. Right. This is a good thing. I wish it was open source. So for anybody that doesn't know, I think it's a feature that's come from enterprise databases like your oracles of this world. And if you've ever had a query plan flip for example data's grown in size and it's flipped from quite an efficient plan to one that's no

longer efficient because it thought well because that plan now has a lower cost basically aurora has a feature that allows you to avoid those flips with certain rules around how much lower the cost needs to be before you change from an approved plan, that kind of thing. So it's a very, very interesting feature. Which doesn't require separation of storage at all because it's only about the planner behavior. Yeah. But my point is that people aren't choosing

a separate computer storage. They're choosing the product. Just because of decisions of amazon business guys that's yeah right so let's but but let's say neon for example i think that is much more clear-cut people are excited but i think because you can start by paying zero there's a free tier and you can start by paying very little. First of all, I think a new example shows how open source can get all benefits that non-open source guys like you, by the way, try to hide from us. You understand? Here we go.

Go on. So, I mean, Aurora is positioned as a replacement. For those who migrate from Oracle, Aurora is like, it's positioned like as a replacement. Like, it's for those who migrate from Oracle. This is alternative. And we know Amazon itself migrated from Oracle to, as I understand, Aurora Postgres. And it's good, like, great. And this is like serious Postgres position. But now Neon is making open source

version of the same thing. And my general thought those managed services guys who like RDS first of all, they say, Okay, we automate backups, or UI everything. Wait a few years, open source versions of it will be maybe even better. And we already saw like Stargres, for example, good UI and so on. And it's open nature. So you can, we discussed it in previous episodes, self managed. And what I'm trying to say, like you can, if you don't go open, it's a mistake. Well, yeah, this is really interesting as a commercial discussion, but I think there's also an interesting thing as a customer. If you choose Aurora Postgres, for example, and start relying on the query plan feature,

what are your options for migrating? You're kind of stuck there until there is an open source alternative. That's one of the features. And eventually, the community will have this feature as well. Somebody will implement this plan manager in open fashion. I think you're right. I think you're 100% right.

And I think Oracle, for example, are losing market share to Postgres, but Oracle still makes so much money per year. It's not like they're going to zero really quickly. So it's an interesting discussion. I think there's a lot of ethics and morality at play. Like there's a question of doing, you can be open and have bad morals.

You can be closed source and have good morals. I think we know where we both think Oracle are and where we think some of these newer providers are. It's not fully clear yet for new providers because they're super young yet. And for database, like a couple of years is nothing. Yeah. And I think it is a bit different if you're a platform versus if you are a like a GUI tool. So I've mentioned this to you previously, but the reason my tools are closed source is because I don't see another business model for it.

I can't. There's not much support for it. And the UI is most of the value. It's not, there's no service that adds a lot of value. It's it just, it's just a UI. So if we made it open source, anyone could host it for free and there'd be no need to pay us ever. It will give a, it will give warranty that it won't die. For example, if you or new new company or if my company is closed our tools, like if we don't open source them fully, it will be hard for people to

continue using them. And this applies to you. This applies to me as well. By the way, I promised last time I'm going fully open. Everything will be open. I'm still thinking like, yes, we will do it. We just released 3.4 database lab engine, DB lab engine now it's called. And 3.5 will be fully Apache 2.0, including everything we try to keep only for paid customers. No, we will have everything open. And I suggest Neon to think about this approach as well, because they keep some pieces not open, you know, and this is for sure, strategic move and so on. So control plane automation, not fully open. And this is a difficult decision if you have a lot of money raised, right? Yeah. Well, even if you don't have a lot of money raised, if your ambition is to build a business around it versus your ambition is to build, you know, there were people in the past that have been in, but there were people in the past that have built databases

because it's academically interesting to them. And if that's your goal, making it as open source as possible makes a lot of sense but if your goal is to build a business it also makes sense to prioritize commercials like i do on like it's interesting to think about in this in this area i recommend listening michael stonebreaker's speech when he received touring reward right so yeah he compared riding a bicycle uphill downhill and he has a lot of experience in both purely academic products and like academic research projects and also building companies for profit companies so this is interesting area but if you want your thing to live longer, you need to make it open. I think we moved away from the main topic. Very much so. Let me quickly go through a few other things that

I think people might want to read up on or watch. You did a good interview, I thought, with somebody from Neon. It was Starskelvich. Yep. On Postgres TV. I'll link that up in the show notes. And another one with somebody from Cloud Spanner about their Postgres compatibility. Partial support of Postgres syntax. Which is actually something I did want to cover here, that a lot of these systems are Postgres compatible, which, I mean, it makes sense, right? Because they've literally taken out the storage layer.

You probably can't say any of them are Postgres. None of them are Postgres. Well, yeah. But if you read Neon's landing page, you'd be very forgiven for thinking it was, right? Like it says serverless Postgres. Everything is Postgres, Postgres, Postgres. So what I meant is if you're looking into these systems

and considering using them, check what they mean by Postgres compatibility. Will what you want to use work? Are the extensions you want to use available? Is the syntax you want to use available? What kind of wait events they provide? Because Aurora has its own specific wait events. By the way, I still consider wait event documentation

of RDS and Aurora RDS as the best. But have you seen for Postgres 17 probably already, the dictionary of weight events will be provided as a system view, like select star from PG weight events or something, and you see with explanation, list of events with explanation. So it's good. Not only in documentation, but right in your Postgres. Good.

Well, hopefully in the documentation too, but if not... Documentation has it, but it's not as well explained as RDS documentation. Oh, got it. Okay. It's very short, like this wait event and wait event type and just one sentence and that's it. But RDS has a whole page with practical pieces of advice. Yeah, I remember now. And I remember not realizing that the first time I saw the page because I didn't realize you could click on each.

Yeah, well, it's a UI issue, UX issue in RDS docs. But I mean, if it has different wait events, it means for advanced users, it means managing will be different, right? Slightly different. So it's not Postgres, it's deviation. And this is what I mean by do your research, check what you want.

Just don't assume that if you're using Postgres already or if you've used it in the past and you want to use certain features, just don't assume they'll you're using Postgres already or if you've used it in the past and you want to use certain features, don't assume they'll be available in these providers. Check. And then the last thing I did want to bring up, which is in this area, was the feature timescale announced. So timescale, I'd argue, is Postgres. Bottomless. But yes, but on timescale cloud specifically, not, I mean, it kind of makes sense that it would only be available in the cloud option they allow you to transparently or sorry without without changing how you query the data

move certain chunks which is so well partitions but they call them chunks to object storage on s3 and the arguments they give for it are reducing cost so that's an interesting maybe that's like the the true root argument is that here in theory it should be cheaper yeah here well of course if you first first of all if you s3 is like virtually infinite ebs volume is always limited. Even it's like, I don't know, I don't remember last new limits, 64 terabytes or maybe already more than 100. I don't manage databases more than a few dozens of terabytes. So I'm not that experienced.

I don't have 100 terabyte Postgres. Still bigger than most, but yeah. I think it will happen in the next few years, actually, but not yet. So offloading data to S3 is very good. Actually, ZFS project had an issue in there, OpenZFS for Linux. They had an issue discussing this feature to transparently move data to s3. And they developed it, I mean, Delphix developed it, but they decided not to open source it

yet, unfortunately. But I think this is exactly where there is some potential for postgres maybe to support something to move some partitions to object storage. This is an interesting idea. Or to offload them to different machines. You need to think about consistency checks here, right, and so on. This is an opportunity for Postgres itself to develop some universal solution which will work with various clouds and maybe even on-premise

and will help you to have this bottomless feature. This is a good example because it doesn't require to redesign half of Postgres, but it requires some redesign at least because if you moved it and then suddenly, you know, S3, if you don't mark some checkboxes, it's very reliable in terms of it doesn't lose data but it's not very highly available as abs volumes especially like regional and so on are so you they might be not available and you will see like some corruption errors like i postgres cannot read some partitions and so on and this should be handled properly properly if you move data to S3.

Yeah, I agree. I'm just struggling to understand how Postgres could control. Bear in mind, that's quite provider-specific. I just see it's an easier problem to solve and beneficial for a wider audience, not only AWS users. Yeah, yeah. You can install S3 as a mini project. You can install it yourself. I mean,

to build some data center and so on, cloud repatriants, right? So those who migrated back from clouds to, like, we know, like Basecamp, right? Yep. So...

By the way, I'm not sure they moved the S3 back. I think that might still be... But I think this wave is obviously starting to grow for those who want to optimize budgets. And in this case, if you know, in Postgres, you can mark some old partitions to be stored out of your hot disk, which is limited and quite expensive and so on.

Maybe even to disk. Already now you can move partitions to colder disk if it's on the same server it's all a very old approach and you can

automate it probably and so on there is some opportunity here and you can achieve this bottomless feature I want a petabyte

database maybe I don't want to erase data I mean on timescale you've got the double benefit of it supporting compression

as well. But just for completeness, I guess the main trade-off then is read performance. So you've moved it to S3 for cheaper storage, but if you do read it in future, presumably

your queries are going to be quite a lot slower. Of course, right. It's quite a lot slower. Of course, right. It's quite a lot slower. And in case of, again, if to remember ZFS, if this feature someday lands into ZFS, ZFS has different types of cache. One is regular ARC, A-R-C. It's like file cache locally on machine. But there is also ARC2, which allows you to,

for example, HIP Analytics had very good article like five years ago or so. So they wanted very good performance. They wanted to use local ephemeral NVMe disks. I think it was time when EBS volumes were not NVMe based, not Nitro architecture. Now they are. But that time, actually, local disks will be always faster, definitely. But they are ephemeral. If you restart, you may lose data. So what they did, they used large EBS volumes, slower but large and reliable.

So if restart happens, you don't lose data. And using ZFS Arc 2, they put cache on local ephemeral NVMe disks. Smaller, very fast, and also ephemeral, meaning that you can lose them, but you can rebuild this cache. Transparently, automatically, ZFS will do it for you. And this also sounds for me like separation of cloud and compute. Imagine if this happens, if ZFS will start working in the same way with object storage like S3, infinite storage in S3 and local caches using Arc2.

Very good feature. And you don't need to rebuild Postgres to benefit from it because Postgres relies on underlying file cache as well so it's working on upper level very interesting so let's maybe try to wrap it up and what are the main benefits for

users which this separation claims to provide first is serverless, faster change of... Is serverless autoscaling or no? So I struggle with definitions, to be honest with you. My understanding is there's a couple of features of serverless that people love. One is scale to zero.

So if you're not using it, you pay nothing. This doesn't require any changes. You just need the proxy, like maybe smarter PgBouncer, which will start your node. And you need probably Firecracker to start node faster. So that's it. I don't mean serverless Postgres, but I meant serverless anything. That's my understanding. But scale to zero doesn't require separation. I mean, it's already separated again.

Like, I don't understand. Sorry. Like, I don't understand scale to zero part. Regardless, I think it's clear. If we want to go into any more detail in any of these, we're going to need to invite somebody else on to talk to or learn a lot more about it ourselves.

And I hope that's answered the question for the person who requested this somewhat. That's, that's what we know. And, and I also think if you're a general user of Postgres and you don't have any problems, you don't need to worry about any of this.

Like there's, I don't see any reasons why we can't work on regular Postgres. The reason we don't know that much about this is because most people don't need this. Let's list all the things. So this is 1 to 0. Good. Understood. I'm not convinced we need to rebuild half of Postgres.

I think it's achievable with regular Postgres. It's just a matter of how fast you can shut down. Shut down is not important, but how fast you can start your compute part. And this is the main part. Postgres does need changes. Maybe I'm missing something. Again, my skepticism is not final. I keep my mind open.

I try to understand. Second is this bottomless feature we just discussed. So limits are much higher you're not limited by single ebs volume anymore because you use s3 like a lot of stuff there good and again i just described maybe there are chances to achieve this as timescale did actually already you with you don't need to again to rewrite a lot of postgres. Maybe we will have it for more cases, but it's a good feature. I think. Third is probably database branching, which both, well, Neon has branching, Aurora has, they have only thin cloning. Because what was the difference is simple thin cloning plus ability to commit or to make a snapshot

and then to claim now you can start new clones from that point. This is already branching. And Aurora has only thin cloning, as I understand. Neon has full-fledged branching API, not really close to Git, but very full-fledged branching API, not really close to Git, but very full-fledged. So my big concern about their implementation is that they provide budget benefits only for storage part. So if I run 10 clones, I pay for 10 compute instances. And this means that they close doors for good CI testing both products. I want same bill. I mean, we discussed it. I want same bill for my CI CD pipelines. If

I run many of them, I don't want to have big O of n bill where n is number of pipelines. I want big O of one constant price. And this is what we achieve with ZFS with DbLab. So again, maybe, and we don't replace Postgres. Postgres is the same. We just replace file system. I think if you want a constant bill, serverless is not for you. I think that's fair to say.

Right, but what sounds appealing is scale to zero. And I'm exploring this to add this to DatabaseLab. I see how to do it. So we probably will do it because if somebody is not working, especially for physical approach, I just learned that unlike wall G, pgbcrest has incremental delta restore. So you can sleep for many days, then wake up and restore physically only delta of physical pitch data changes this is great so you don't need to replay walls it's very slow if you accumulate three days of walls replay will

take some time for heavy loaded system if we talk about to apply delta physical backup that's great so so i mean we we can achieve very good behavior here of course new and the characteristics are very good but again system is quite new so and maybe fourth and final one auto auto scaling to both directions not to just zero but between like by the way do you know why neon compute units are limited by seven only no i don't know yeah that's interesting question seven doesn't seem big enough for me i think they've started i think with new systems it's really sensible to start aimed at startups because if you like the difficulty with databases a lot of people choose it based on reliability based on you know if we're

talking about the storage layer of postgres that's arguably its best feature you know we we choose our database because we want it to be reliable and battle tested and it's got decades of track record there so if you're a new system with a different underlying storage mechanism you're probably best off aiming your service at people who can take a bit of a risk because they've got nothing to look that they're a new startup they they are enticed by oh it's going to be free at first or it's going to be really cheap as we scale so i think it's a really smart strategy to go small first i think super base have done the same but as they get more serious as those startups grow they can

move they can gradually appeal to larger companies i think it makes sense yeah but equally i don't know seven's an odd number right like what is i would have expected to be a multiple two i have no idea they're very smart people running it so i imagine it's a very good reason i have no idea okay but i want to ask about this autoscaling, you as a specialist in query planning. What happens if, first of all, I hope they will fix it. I'm sure they will fix it. I never tried it for Aurora serverless. When autoscale happens, my PGBench shouldn't be disconnected, shouldn't experience errors. And the same with my application, right? So it should be seamless. I think this can

be achieved. And that's great. I mean, consider this is solved. But I wonder what's happening with query planner and its settings. Because if we jump from one, for example, from one CPU and some RAM to much bigger instance, planner settings should be adjusted. We have bigger cache, right? This I don't understand. I think Neon is special, and I'm sure they think about it. I just don't understand how I can manage my performance settings, database configuration, when it's quite unpredictable

in terms of how big instance is at any given point of time. What do you think about this problem? Yeah, I guess it's stored alongside the... I guess it knows when you're connecting to it what it has available and therefore what its settings are. I have no idea if they control that for you or if it's configurable on a... Can you set various limits as to what...

Can you set your, let's say work mem can you say when when i'm when i'm down when i'm provisioned to a small instance set it to this and when i'm on a medium instance set it to that a high number and said when i'm enlarged can i set it to an even higher number my guess is they haven't got to it yet but i actually have no idea i would like to have more control and predictability than unpredictable changes. And I actually know what maybe Aurora implemented this plan management feature, exactly because with auto scaling and serverless approach, we need to freeze plans, maybe. Who knows?

We're guessing I have no idea. But if you know if you've been involved in that that'd be really interesting to hear from you yeah you mean me or listeners uh anybody listening i'm joking so maybe it's time to wrap up i think we just started this discussion it's very interesting i'm far from understanding it fully so i appreciate if people share their thoughts and we can continue another time but it's very interesting I'm just an old schooler and I rely on things I know how they work

though I don't know how they work fully always so yeah I mean Postgres itself it's still it's so huge system you never know how it works fully but I do my best so yeah please share this episode with others as usual. And we need support as usual to subscribe in YouTube on all podcast platforms and the likes, comments, we appreciate everything. It helps us grow and see also signals that it's interesting

to continue. Nice one. And thanks thanks for this suggestion we've had a few more suggestions as well so we appreciate those coming in thank you nice one thank you bye