Postgres FM - pgBackRest

Episode Date: July 3, 2026

Nik and Michael are joined by David Steele to talk all things pgBackRest. Here are some links to things they mentioned: David Steele https://postgres.fm/people/david-steelepgBackRest https://pgbackrest.orgpg_basebackup https://www.postgresql.org/docs/current/app-pgbasebackup.htmlBarman https://pgbarman.orgpgmoneta https://github.com/pgmoneta/pgmonetaWAL-G https://github.com/wal-g/wal-gReturn pg_control from pg_backup_stop (patch proposal) https://www.postgresql.org/message-id/flat/86436ff9-eb98-4c8a-825e-3bcae022107b%40pgbackrest.org#e36e2ad2f410d1a9a1152e36aa66bb48Add StorageReadMulti for prefetched multi-file/range reads from object stores (pgBackRest change that needs review) https://github.com/pgbackrest/pgbackrest/pull/2783pg_hardstorage https://github.com/cybertec-postgresql/pg_hardstorageWhy the cycle of open-source sustainability needs to be virtuous (blog post by Gabriele Bartolini) https://www.gabrielebartolini.it/articles/2026/04/why-the-cycle-of-open-source-sustainability-needs-to-be-virtuousARIN https://www.arin.netStefan Fercot https://pgstef.github.io/aboutpgBackRest sponsors at the time of recording: AWS, Supabase, pgEdge, Tiger Data, Percona, Eon, Xata, Dalibo, Data Egret~~~00:00 – Intro & pgBackRest origin story03:07 – Could pgBackRest live in Postgres core? Why it hasn't happened11:37 – Primary vs. standby backups & corruption safety20:36 – Measuring RPO & RTO in practice26:20 – Checksum performance & S3 storage class tips30:02 – Standby replay bottleneck35:12 – Log shipping, streaming replication, and sharding44:56 – Maintenance reality & the sponsorship crisis47:41 – What's next: repo-to-repo backup & RPO-zero streaming56:00 – Incremental backup1:05:01 – Reliability philosophy1:10:05 – Second maintainer, community growth & wrap-up~~~What did you like or not like? What should we discuss next time? Let us know via a YouTube comment, on social media, or by commenting on our Google doc!~~~Postgres FM is produced by:Michael Christofides, founder of pgMustardNikolay Samokhvalov, founder of Postgres.aiWith credit to:Jessie Draws for the elephant artwork

Hello and welcome to Postgres FM, a weekly show about all things Postgres Quirle. I am Michael, founder of PG Mustard and I'm joined as always by Nick, founder of Postgresor. Hey Nick. Hi, Michael. And today we have with us, David Steele, who is a significant contributor to Postgresquire and the creator and maintainer of both PG Backrest, which we'll be talking about today and PG Audit. Welcome, David. Thank you very much. Good to be here, Nick and Michael.

It's a pleasure to have you. Right, to get started, I wondered if you could give us a little. little bit of history, a little bit of the origin story, perhaps, of PG Backrest. Sure. That's easy enough. Actually, PG Backrest was born at the Dublin Conference in 2013 of conversations that Stephen Frost and Cynthia Scheng and I and Magnus, Hager, and some others were having. At that time, we were working on a fairly large database for the time, around 50T. That doesn't sound so big these days, but in 2013, that was a pretty significant database. You know, basically we needed to make backups of course, as you do, and just the available tools were not up to the task.

It was just simply too large, and we really need to be able to do incrementals, and there was nothing that would do incrementals, plus compression at the same time, and et cetera, et cetera. So as you do in open source, we decided that we would build our own thing. I remember originally it was going to be a pretty simple project, write it in pearl, keep it simple, just be one file, so it'd be easy to copy it. around and distribute and et cetera. Yeah, that didn't last very long. Obviously, it grew pretty quickly. So I built it. I built the initial software that was usable in about 40 hours and to basically solve our initial problem and then just kept building on it. As we had problems and bugs and other things, I would build on it and build on it. Convinced the company I was working for, Resonate, which is an ad tech company, to open source it. And then when I left there,

I kept noodling around at it. I took some time off and I just kept working on it. the restore functionality working well and then that's when I got hired into Crunchy. And at first they weren't super interested in it but Steven was. So I got a little bit of time to work on it and then eventually I got more time to work on it and then eventually we hired Cynthia to come and work on it with me and that went on for a while and then finally we decided to migrate it to sea and we did the whole sea migration which was two years. So that was painful because we didn't we basically didn't write any new features for two years. We fixed bugs only. And we could only write a new feature if it lived entirely in the

C code. And over time, it was more and more possible for that to happen, but it was a tough migration. Two test suites, two of it, it was just, I didn't want to think about it. It was not a lot of fun. And that's basically the project as it is today. Now it's, of course, we're rolling along. It's a C project. We have features based on user demand, based on chasing performance. is always the big thing, like how do we move a bunch of data quickly? How do we do it efficiently? How do we keep the repo as small as possible? Block incremental backups.

The list goes on and on. I have a question here about history. Sure. Maybe you remember those discussions, should PostGus have, remember times of Sloney and then Londester? And those discussions should PostGus have replication inside? Right.

And there are many such discussions. Same for auto failover and so on. And for the case of replication, the idea to have it in core won. For the case of auto fell over, it lost. Still, auto fell over is outside. I'm very curious what you think about idea to have full-fledged backup solution inside core.

And why is it's not happening? One interesting historical fact is, and one of the reasons why some of the other committers were involved in really early planning. And part of the reason for the migration to see was the idea was that we could actually maybe make PG backresty core solution for backup. Now, this is not something that was endorsed by core broadly, or it was just discussions that I had with some committers because they were interested in having a more comprehensive solution in core. Obviously, they wanted to be written in the C. And so that was one of the reasons

that drove the adoption of C. As for actually having, having something like PG backrest and core, it's a little tricky because the PG backrest project moves a lot faster than Core does. So to have it on the yearly cycle would be, let's say we had put PG backrest into core like from the beginning, maybe as soon as it was migrated to sea. It would not be nearly as far along as it is now. At the time we were doing 12 releases a year, we went down to six and now we're currently at four, which I think is just about the right tempo for a project of this type. But being able to release features four times a year,

gets new stuff out there, get people trying it, get people testing it, that kind of stuff. I think it's pretty important. And as PG backriss gets more stable, maybe it would be more appropriate. But at the same time, the project has diverged significantly from, you know, we use a lot of the same concepts as post-us, mem contexts, and error handling looks a lot similar and et cetera, et cetera,

but it's all pretty different. to PG backups grew its own way. So it'd be very difficult to get in there. I think the idea is PG-based backup was going to be that tool. Getting incremental backups into PG-based backup was a big step, but it's still not a complete tool. You have to have tooling on top of PG-based backup for it to be at all usable, especially for incremental.

Because reconstructing all those incrementals going and fetching them and uncompressing them and getting them all ready for the PG- combined backup tool to run, that's a simple. significant amount of work. And obviously it doesn't do anything with wall archiving, expiration. There's just the list of all the things that you need to do. And it's a pretty big list. And it's intimidating, honestly. You know how core goes, right? It's intimidating to even contemplate getting something like that into core. So I think it would be a good idea. But also think about storage drivers. Support for like PG-based backup supports Pazix. right but most backup tools support S3 GCS Azure SFTP

etc because those are the tools that people are actually using so all that would have to go into core as well all the storage drivers people would have to maybe not everything could go it's possible PostGIS is very extensible if just the core thing would go to the core but expose some interfaces particular drivers stay outside and in theory that's what we done a bit with PG-based backup, but the amount of stuff that needs to be done by the outside tool is huge. Now, there are quite a few tools that are based on PG-based backup to do their page-level-block-level incremental. Bar-Man obviously is the most well-known one. I think PG-Munate is one of

them. There's a newer one that I saw recently that's also using it. So you can certainly build a tool around PG-based backup, and I think the idea would be that you'd extend and extend PG-based backup and then people would take that feature out of their tool. And eventually maybe PG Base backup would be a thing that does everything. But at the pace that it's actually evolving, I would expect that to reach feature parity with something like Backrest or WallG or Barman in approximately 30 years. I'm exaggerating a little bit, but if I look at the actual progress that Base Backup has made over the years, that's where we are. Since Bach Incremental, nothing's been done with that even though there are pretty huge performance implications for restores.

Any very large database to restore with that page incremental format is you basically need, let's say you've got a database that's a terabyte, you need at least two terabytes to do the restore, minimum, and it depends on how many incremental as you have, so you can need two, three terabytes to do the restore. All the files need to be copied down regardless of whether those blocks are used or not. Everything needs to be uncompressed and then fed into combined by. which then rewrites everything to a different location. So from a scalability standpoint, it has a pretty serious problem, and we're two versions on from it being introduced,

and no one has actually even thought about actually addressing any of those issues. I don't want to harp on this too much, but the main point is getting stuff into core is hard. It takes me years. I've been working on a very small change for a backup just to mark PG control, Mark PG control when a backup label is required. So you do a backup label, a backup label is required to do the recovery. If the user deletes the backup label, they can end up with corruption, silent corruption.

It's quite annoying, actually. And I've seen people who know Postgres really well, hackers do this and not understand what happened. I saw it many times already. In various teams, it's annoying, yeah. So two years ago, I introduced a small patch for Postgres to just put a flag in PG control to market when we actually need, sorry, on backup label. So if you start Postgres and backup label's not there, it will just stop and say, no, you must have backup label, please provide.

That was two years ago. It goes to review occasionally, but really nothing there. In PGACRUS, we actually implemented that about three years ago. What we do, this is a bit of a hack, but it works pretty well. What we do is we overwrite the last checkpoint in PG control. we write in the hex value DEAD. So if we get a report from a user, because it'll come up a Postgres,

it'll say unable to find checkpoint DEAD. And that's actually not a valid checkpoint at all because it's under the first wall segment limit. So Postgres will market is invalid. It'll throw an error. And then when we get a report from the user, we immediately know, hey, you try to start this, you delete a backup label.

And they're like, yeah, I delete a backup label. And boom, there you go. But I've been trying to get that, that, I wrote that patch two years ago, and I'm still hoping to get that into Postgres. So that's like the speed at which Postgres can operate sometimes, especially for people who, like me, who aren't really committers. So can you imagine trying to get something the size of PG backrest or complexity of PG backrest into Postgres? Yeah, we should highlight this, put the link to this patch and maybe drive some attention to it because some people who participate in PostGast Hiking listen to us and may be. Maybe I agree it's super annoying. And right now all the other tools they need to, basically, if you create full copy, full backup on a replica, you are responsible for placing this backup label yourself.

And it's also how restore works in Postgres. I fixed recently. And this fix was very quick because it was obvious. It was just problem in code and Postgres in Restore path. How Postgres is working with backup label and also signal files. it's if you look how people use it, they see some error in logs and then, okay, I will just delete some file, right, to get going. And this is what's happening all the time, right? But also I noticed PGB crest also requires, maybe because of what this mechanism you explain, well, when making backup on a replica, it requires connection to primary.

And this surprised me a lot recently when I started the code, right? Yeah, that's, that actually doesn't have anything to do with that specifically. the reason why we did that is you get a better backup that way. Better backup. Not all the backups are equal. You get a better backup. First of all, there are some things about backup from standby. I don't necessarily want to get into that because it's pretty esoteric, but there's some things about backups and standby that I don't 100% trust. But I do believe it works, but the reason why a primary backup is better, though,

is you get a couple things. You get stats from the primary, which is actually pretty nice. If you do a backup from standby and you restore, you've got stats from, say, a standby. And if you're actually doing analysis of statistics, it's pretty shocking to suddenly have, like all of your index patterns, scan pattern, read patterns, full change, everything.

So I'm not talking about, sorry, table statistics, like basically, like scan statistics, maybe I'm using the wrong word. I think I understand number of six scans on a specific... Yeah, I think they're both stats, But I'm not talking about the plan or statistics. I'm talking about the actual usage stats. Another thing is, if you're doing a backup on the standby,

you can't actually finish the backup and verify that all the wall has reached the archive. You can, but you might have to wait a day for that to happen or whatever. Like, we feel it's really important to make sure that this backup... When we mark the backup as done, we want it to be done. Not hoping that someday this wall archive is going to arrive. we want it to be done at that moment. Right. So that's pretty important.

Do you mean the backup is like self-sufficient, right? It positively affects our T.O., right? Like recovery time objective. Look, it reduces the time you need to recover. Well, at the very least, we want to know that the backup can be recovered to consistency. Right. Now, if you want to do point in time recovery, that's going to be after the backup finishes, you'll continue while archiving.

and you need to monitor that as well. But at the point where the backup finishes, we want that to definitely be true. Other things like getting actual logs from the primary are more interesting than getting logs from a standby. Although I don't really recommend that people put logs in their PG data directory at all. It's actually quite common. Hold on one second, one second.

Too many things here. First of all, when you say logs, it's wall files, right? No, sorry, in this case, I do mean logs. So, like, basically textual logs that are generated by Postgres, and a lot of people will put those in PG data. Okay, okay, okay. I didn't get it. Why do we care about logs when I'm making backup?

Ah, because we don't want them to be backed up, right? We do, if you have them in, so if people are putting their logs in PG data, and they expect them to be from the primary, it can be surprising when they're logs from a standby. Or somebody's saying, so they're not getting the information that they expect to get. Maybe they have auditing turned on in the primary and not on the standby. So you recover the backup, and now all your audit records are gone, et cetera. So now I'm not a proponent of putting logs in PG data on the primary, but there are people who do this and they want to preserve those logs.

That's another reason. So there's a whole raft of reasons, but basically, so what we do is we copy everything that's replicated from the standby. Oh, the other reason why we really wanted to do this, although honestly we've never implemented it, is once you've done this, you can actually parallelize a backup across all available standbys. Because you're coordinating everything on the primary.

You wait for the standard. the stand by is to reach the checkpoint where the backup started, which we already do, of course. We do a bunch of consistency checks. And then you would be able to parallelize your backup across all the available standbys and really supercharge it. And it's not even that hard to do. We just haven't, there hasn't been demand for it. And there's always a million things to write.

You know how it is. So we just haven't gotten to it yet. But that was another big reason to do it that way because it allows us to parallelize backup in a way that otherwise wouldn't be possible. if you're just backing up from a single standby. There is a dilemma here where, first of all, making full backup on primary, especially, if it's full, like, not incremental, full.

It's a huge stress for disks. Yeah, you don't want that. Yeah, yeah. So usually people upload it to some standby, and then the problem is also like it's which standby, because it's also stress. If it's a read replica, it's stress for those reads as well. So distributing totally makes sense,

but there is also, like, since you talk and focus a lot on corruption, and consistency, I was always curious if we, for example, have allocated standby only for backups, which I think Crunchy Bridge had an issue. We had customer there. Crunchy Bridge had an issue when they made full backups on primary. I was super shocked because, like, how come? It was fixed. And I think full backups went to H.A. Replica, which is for H.A. It's allocated. It doesn't receive reads. But then I'm curious, if some corruption happens, it might happen only. There are many kinds of corruption, right?

I can think about, at least theoretically, some corruption might be only on one node, but not on others. And if you always make backups from only one node, then you don't notice this corruption propagates to all backups. So some rotation at least is good. But combining and pluralization, it's also interesting, right? I'm just thinking when SQL server, for example, there is a mechanism to self-heal, if, for example, primary notices, some pages are corrupted that can grab them from replicas, which was implemented long ago. And this is cool technology.

And I'm curious about this dilemma. We want also distribute load not to hurt our user traffic, but also what's happening with corruption. And this is whole, this is Pandora box of topics. That'd be an interesting idea, actually, too. If you were doing backups from multiple standbys, you could compare and contrast. If something, if there was corruption on one standby,

you could say, hey, why don't I try grabbing this file from the other standby? instead. Fail it, send it back to the main process, which would reissue that job on a different standby. That's a pretty cool idea. You talk about corruption, like data checks on this level of corruption, but there may be a higher level of corruption, like index level, like logical level, foreign keys, like a lot of stuff. Sure. And your corruption scenarios on primaries and standbys are quite different too. Because the way things are being written out on a primary is actually fairly different to the way it's being written out on a standby. Also, the standby is essentially a single-threaded operation.

Let's not go out instead of all. Obviously, it's terrible for performance from the standpoint of any issues there might be with locking and parallelism and other things on a primary. The standby is not going to see those bugs, probably. And so there are whole classes of bugs that can happen on a primary that could cause corruption that aren't going to happen on a standby. So from that perspective, it might actually be safer to do backups on a standby as well

just because it's a simpler path. The data goes through a much simpler path to get the disk on the standby than it does on the primary as a rule, unless you really have one writer on the primary, which is not the way things tend to run these days, not for interesting databases at least. So yeah. Yeah.

And as you say, the corruption we're detecting is only checksum corruption. There are some projects out there where people who have combined PG backrest with Amcheck. So basically when it does recovery, like basically you call this command, it automatically integrates PG backrest with some sanity checks afterwards and does ANCHEC and some other things to check a higher level of consistency than just the page check sums. Yeah, when you do recovery, it's a good point usually. This is what we usually do. Not all recovery attempts, but some of them, percentage of them should also check index health. This is really useful in test recovery. So hopefully you're testing your recovery right and that's a great time to do am check because then you can have confidence in your

emergency harrison fire recovery that it actually does work because you practiced it you ran amchuk on all the practice runs so at that point you should feel pretty pretty confident to run the production restore without having to take the penalty of running amchick at that time if you've been properly testing i agree with everything but worked all you cannot do it on all runs because full-fledged Amchek takes a lot of time. Yeah, that's why I say like it's really for tester storage, right? So you're testing things and it's tricky, right? Okay, I guess you can say Amchuk takes two days, so we're going to do this once a week. And then we'll do more recovery testing, but only run Amchuk once a week. And I think

that's a perfectly valid way to go about it, to think about it. Absolutely. I'm also very curious, did you think about some more observability, capabilities in PGBCrest to understand that. My final goal is, for example, I have 100 clusters. I just know, I just want to know actual RPO RTO all the time, measured somehow, during those restore tests, during the backup process itself. And just to have some high-level view, this is what's happening. That's it.

But now you need to build a lot to have that. RPO, you can measure using the information that PG backrest provides in the info command because it'll tell you what the most recently, how up-to-date you are on your wall segments, right? So that's going to give you your RPO. So I'm one segment behind, 10 segments behind, five segments behind. So at least you know what your recovery point is based on the information that you get. RTO is a much trickier scenario, though. RPO is also tricky because you think about, you.

like technical guy and I'm also technical guy. We think about bytes. Business guys think about seconds, the data loss, like how many hours of data we lost, right? And this is to translate somehow. It's absolutely fair and you're not going to be able to get that that timing exactly from just you could use, for instance, our repo LS command to go and grab information about the wall and find out how much wall you're generating per second. And it's not going to give you exactly the time but it's going to be awfully close right so you could estimate you're like okay we're generating five wall per minute or two wall behind and therefore our recovery point is this many bites behind and this many seconds behind so you'd actually be able to estimate

that reasonably well using tools just that are available on pg backrest although the actually estimating the time of the wall you would have to roll your own for that because that would have to be something you do for repo. So the nice thing about RepoLS is it's a command we give you and it works on any repository, whether you're on S3, GCS, SFTP, POSIX. So you can use that tool to get information about the repository that's not included in our JSON info and it will work in any environment on any repository, on any storage. So you don't have to muck about with S3, AWS CLI over here and the Azure CLI over here

and whatever the GCS equivalent is and et cetera, et cetera. And then, okay, so you could sort that out, I think, with the tools that are available with PG backgros, but RTO is still a thing that needs to be tested. We can't tell you RTO at backup time. And since Postgres is famously slow for recovery, maybe not slow, but certainly things don't get ridden as fast as they get written on the primary. That much we can agree on. Recovery actually, it goes pretty well. And Postgres has gotten pretty good at things. PG backrest prefetches, wall segments, all the scents.

All this combined actually gives you a pretty good performance overall. But again, you're going to have to test it to know how long it takes you to do recovery. Fun little example, there have been, and this hasn't happened one time, but the first time it happened was in early days of PG backrest, because our thing has always been high volume performance, etc. So there are a number of people over the years who have used PG backrest because they can't use replication, because replication simply will not keep up with their load ever. So what they do is they basically continuously take and restore backups.

In that way, they can measure the RTO pretty well, actually, because they're constantly doing these recoveries. And they get to a point where they, and then we'll see how long it takes to get to the point where they were when they did the restore, right? Because they can never actually catch up. But they know how long it's going to take. If the primary fails, how long it will be until the standby is up to date

using all from the archive. downtime will be this. A dime time will be this. And they know it exactly because they do it every single day. And this is a use case that I've seen multiple times now. People have brought it up with me. It's a really interesting use case. So they are bumping into this 100% single CPU situation problem of innate process on replicas.

So they, it means they'll leave without replicas. With these replicas, they cannot use it because they are lagging, basically, right? Yeah, they're pretty far behind. So they really are just there to, I mean, obviously, you can make up scenarios where a replica like this could be used. Let's say you're doing reporting and you know your replica lag is six hours. Then you know by 8 a.m., you can start running reports. Right, that kind of thing. So you can game it, but you're right.

It severely reduces the usability of the replica in that scenario. But they just need something because they can't start from, they're just trying to minimize the time it takes for them to get to running again. Because if they start from a restorer of their 50T backup, that's going to be however long that takes. PG backers is pretty fast, but 50 terabytes is 50 terabytes. And then they start recovery. One hour. I showed an article 37 terabytes per hour. We can do it one hour, but it's local and we meet or local in the media.

It's very different. Exactly. As usual, the backup is, you know, I've gotten a lot of flak over time about using, we do a lot of checks. summing and we use pretty heavy duty check sums and people are like oh those check sums are so expensive how can you stand how expensive there are they're not expensive once you actually start pushing stuff if you're writing to your local SSD then yeah the check sum overhead is 5% in PG backrest I've measured it we have tests for this so yeah it looks like a lot but then you actually start pushing stuff out to

S3 and poof it just disappears you don't even see it anymore it's it's how Yeah, the trick is getting this kind of volume in a realistic environment. People don't store their backups locally. At least they shouldn't be. That's the message we try to get. I should repeat the tests with S3. I'm very curious, the level of paralyzation we can get. Of course, we need, the machine itself should be where we recover.

It should have local MVMs. I'm very curious. We have a, yeah, sorry. Let me just address that really quick because we do have a good, interesting optimization coming hopefully in the next release, if I can get it reviewed by July 10, that's our feature freeze. It basically does, basically it's adding prefetch for object stores and something I call, I'm sure other people do this, but I haven't really figured out what it should be called. So I've been calling it readover, hoping someone will come up with a better name.

But a lot of times we're reading through, say, a bundle, and we need n number of files or n minus m number of files. And so we're basically starting a bunch of new reads. So we'll grab these three files, and then there's a file we're skipping. So we'll start a new read, grab four files, and then we have to skip two files. So we'll start a new read and then do... So what we're doing is actually we have this thing now called readover, which is also under review, and it will skip over a configurable block of bytes. So if we're reading through the file and it's configured by default to 64K.

So if we have this one small file that's breaking up the read that we're doing, doing, we'll just read it and throw it away and then continue. And I picked 64 bytes because based on my testing, that was, it's cheaper to read 64 bytes pretty much in all scenarios, no matter what your storage type is, whether it's reduced redundancy or call it the longer term storages and stuff. I can't think right now. And so 64K is basically always a win. Even if you're paying for egress, which you will be in some cases, you're still going

to pay less if you just read the six years. and from a latency standpoint, it's always a win. Because starting a new read is just expensive in terms of time. And there's actually a cost to starting to read as well. Once you've actually got a file open your reading, in normal estuary storage, you're not paying for that egress. But if you actually start a new request, you do pay for that new request. Anyway, some interesting features that would make the tests that you were talking about

very interesting to run. It would be a very different scenario than PG backgrs today. Longer storm to storage glacier, right? Yeah, no, I'm not thinking a glacier. I'm thinking of the one in between. I actually set all my buckets to transition to it automatically after 30 days. So just transition to that. You can still read it instantly, but it costs.

If you go and read it, it costs. And there was a, people have been wanting to add this as a feature where you can go into the storage class immediately for all backups and wall. Yeah. Standard I.A. I.A., thank you. Infrequent access. That's the one.

And it's actually a bad idea to write IA initially because in a healthy repo you're actually reading the repo. And as soon as you read one backup out of that repo, you've destroyed all the advantages of IA. It's all gone. The idea of IA is, I've got an older backup from six months ago that I'm holding for compliance, and now suddenly I need to recover it. Great. You do that once every two years. Now you're using IA correctly. But if you're writing all your backups into IA initially, you'll just end up spending more money.

The only scenario it works out is where you write and never ever do any recovery or any wall recovery ever. Which is just not a use case to me. Back to these cases you are familiar with. I'm very curious how much of wall data was written per second on those cases where it was like the problem. They cannot afford replicas with near zero lag. right yeah some hundreds of megabytes per second I suspect it can be it really depends on what you're doing let's say you are the simplest possible case is you're replaying sequential rights on an unindex table right that's your simplest possible case that'll run like gangbusters and that'll

definitely go into hundreds of megabytes or gigabytes maybe not per second but hundreds of megabytes per second, I think, becomes pretty reasonable at that point. There's still some, still quite a bit of exchange to get the wall segments, move them, rename them, do this kind of stuff. But it's quite fast. But as soon as you start doing interesting things, like say, updating indexes, then your right volume is going to go down significantly. Then it's all going to be I.O. latency. Right? So the CPU will drop below 100. You'll go into de-weight, and you're going to be looking at latency. So if your CPU actually is 100,

you're probably doing really simple rights, generally speaking, and you can maintain a really high throughput if you're doing that. If you've got complicated index rights and stuff like that, your CPU will generally not be 100 anymore, and you'll be into I.O8 instead. So I played around with this in a bunch of different scenarios, trying to figure out ways to help Postgres and maybe gain Postgres,

and certainly, Certain scenarios are just slow. It makes total sense because if it's like simple, it's I-O-bound and if on primary we have hundreds of course doing this work, but on the replica we call on the one doing this, like catching up, trying to catch up to do the same work basically, logically, right? It's terrible. This is, yeah. Yeah, that makes sense.

Simple work. Yeah. And unfortunately, all the cool work that Andres at all had been doing on ASync I.O., Tom and etc. It's not going to help that scenario very much on the standby. It's great on the primary, but it just makes a situation where the primary can write even more data than the standby can ever possibly keep up with, again, because single-threaded. So those async operations are only going to buy you so much. Direct I.O. and stuff obviously will help. So there are improvements being made in the single-threaded recovery. So make no mistake. I'm not saying there aren't because there

definitely are, but they're not keeping pace with the improvements that are being made on the primary side, which can be an order of magnitude greater than what's happening on the standby. Yeah, exactly. I used, I think, Intel machines and I, in this experiment where I achieved 30-something terabytes per hour with PG backrest, and I compared it to PG-based backup, and I expected two, 300 megabytes per second as usual. This is like how, it's, but since it was PostGus 18, I saw one gigabyte per second. I was super surprised. It's actually gotten better. I remember the article, yeah, and that jives with what I've been seeing in terms of overall I have improvements in PostGus. So they're impacting everything. Make no mistake, but it's obviously

that recovery bottleneck is still there. And there's not much that PG backgress can do about that except just make sure that Postgres always has the wall. As soon as Postgres requests a wall segment, we've got it right there waiting for it. When we got record. recommendations on how to set up your storage so that we can do a move into the PG-Wall directory, so it's as fast as it can possibly be. We're not doing a copy. We don't f-sync things. We don't, et cetera, et cetera. So we're trying to run everything as fast as possible, but there are limits. This makes total sense, and it's great insight. Like, improving things around, you might

highlight some difficult problems, and they might become even more acute, right? So everything improved, so you have better performance, much faster you can achieve the problem with lagging replication. Yeah, there was definitely a time before PG backrest where and other parallel wall implementations that were getting the wall was the bottleneck. And Postgres kind of merrily sailed along and did its thing, and it was just wall that was the issue. Now it's usually Postgres. Another place this expresses is if you're doing, like people have noticed and started pointing out to me, although I already knew it, is that if you're doing log shipping, you can do recovery much faster than if you are doing replication. Because log shipping, compressed segments, which are prefetched and decompressed asynchronously and then moved into the PG wall directory is quite a bit faster than streaming uncompressed wall segments over the network from the primary, basically every day of the week, if you have high volume.

So that really brings into question the idea of, I brought up a standby and I want to do recovery. How do I do the best things where I'm going to go and log ship as long as I can because that's faster and also puts less load on the primary and then only switch over to getting data from the primary when I have to. And at one point, do you switch back and stuff like that?

And Postgres doesn't currently manage. that very well and the rules are a little bit arcane. So some archivers, we haven't really done this in PG backrest yet, but there's certain things you can do to basically, you can artificially give an error to Postgres to make it switch back over to replication. Do you get what I'm saying? So let's say we get to pretty much the end. You're moving very fast. So you're talking about replica where both restore command and primary Conan 4 are configured, right? Yes. And then you'll say Postgres has a very strict precedence rule. I think it searches local pG wall directory first, then it uses streaming application,

and only then it uses restore command, if I'm not mistaken. I believe that's the way it works, yeah. And then you say you have a way to full PostGus to achieve what you want, and what you want is to get walls from archive because they are compressed and you can paralyze it at restore command level and only switch to primary conunfoam to streaming replication in some cases. Right? And then you have a with some error, this I don't get already. I can imagine throwing an error from a store command.

I cannot imagine anything at primary coin for that's a problem. The situation is, let's say you are, you've been doing log shipping because that's the fast thing to do. So you're getting wall segments. Now, Postgres, if it's basically decided on log shipping, isn't going to switch away from that until something happens. The idea is like the actual, the wall get routine, when it knows it's at the end of what it has. So let's say post-guess, so you get to the end and now the next wall segment

arrives two minutes later. Now you've essentially built in two minutes of lag, because post-cust will patiently wait two minutes for that next wall segment to arrive, as long as it doesn't get an error. So in this situation, what you can do is actually inject an error into the archive get to say, I know I'm at the end of the archive stream in the repo, because I can see it. I'll throw an error to force Post-Gust back to replication. Yeah, streaming replication. I think I got it wrong. I think Restore command has precedence over streaming

communication, maybe. I think it does. I always forget it. But if the recipe you describe implies that. But it's something to see that. And a lot of times, even if streaming replication were prioritized, let's say you recover, restored a backup, and you need to get wall from three days ago to start doing recovery,

there's a very good chance that's not going to be on the primary. So you're going to go over to log shipping at that point anyway. But the other thing you would want to do is let's say you start lagging too badly on the primary. Ideally, Postgres would go back to log shipping because it's quite a lot faster. Historically, that wasn't true, but with modern archivers, like something you'd find in PG backrest or Walgy, we can do that a lot faster than replication. So in theory, if you're lagging behind enough on replication, you switch back to log shipping, catch up, then go back to replication in order to have as little lag as possible.

So the question is, how do you coordinate? On the PG backrest side, we can gain it to force PG backrest to switch to streaming rep, but we can't really get it to go back to log shipping. Now, these are discussions that have been had. I don't know if we've, I can't remember if we've had them on hackers, but it's something we've certainly talked about at conferences. Yeah, that's an interesting idea. It would be great to have more control here and more flexible.

configuration. Some kind of, sorry. We have huge delay because we are on very different parts of the world, right? Three very different parts of the world. We are very geographically distributed, that's for sure. So, yeah, so on the one side, the archiver would have to tell Postgres that it's caught up. So rather than throwing a nasty error, we could say, we could have a code to return value two or something to say, hey, I'm at the end of the wall in the repository.

Or just signal a leg. Signal the leg. This is my leg, that's it. And then the mechanism will decide which precedents, adjust precedents, based on knowledge about legs. Or that. And honestly, if you're going to do that, the way to do that would be inside the archive library interface. And you're familiar with the archive library, right? Or archive libraries. We don't have one for PG backrest yet because it's not needed for performance in PG backrest. And it doesn't really provide any other benefits. So we haven't really done it yet, although now that several versions of Postgres support it,

I'm looking at probably adding that this year or next year at the latest. Because then maybe we could push some of these ideas into Postgres as saying, hey, let's flip back and forth between Archive Get and streaming replication based on what's, and passing stats back and forth inside the archive library would be easy. Right? You don't need any fancy return values or JSON or something. It would just be an API call that you would use with the archive library. in Postgres. Postgres 15 when it was introduced.

Postgres 15. I wanted to clarify very clearly that we talk about fetching walls and streaming, but after that, Postgres needs to replay, and then we have this innate process, 100% CPU. This cannot be solved anyhow, right? So if you manage to... Usually, the problem is there.

This is what I'm trying to say. It depends on the system, I think, but even with a simple, regular streaming replications, which is no compression, nothing. I usually don't see any problems with receiving wall and just putting it to file with wall receiver. The problem is usually only need process. This is my observations on multiple systems. And that's true. A lot of it depends on your network. If you're running things in multiple availability zones and stuff, or across cloud providers or whatever, that situation can

definitely change. Yeah, you're right. But certainly if you've got primary and standby locally, but the network, if the network network speeds these days, you're right. Streaming replication should not be a problem in that environment in this day and age. My observations are with good network and single region. I agree with you. If you reduce network complexity, yes.

That sounds all very clever and like obviously needed at super high scale. We've talked to a lot of people and it seems like there's quite a few projects in the space of sharding these days. And it feels like the people that go the sharded route, sidestep this problem because each of the shards just has lower volume, like by definition. So I'm wondering if it's just like an orthogonal thing to PG backrest, like sharding can happen and each one can be backed up. Does it affect you in any way? Or are you thinking on those lines at all?

From a PG backrest standpoint, no, because it's really from our, whether it's, um, so the, the things in this arena that I know support PG backrest are like green plum and, um, multi-grass. And in both those situations, it's a very straightforward PG-backrest scenario. We're just doing recovering, they give us a point in time to recover to, and that's up for them to figure out how to get the shards back and sink, and then we just give them what they asked for. Multigress looks enough like post-grass that PG-backrest just is cool with it. On the Green Plub side, there's a company that has created a fork that specifically supports green plum that will read its control files and do that. And we've never brought that.

into core mostly because I don't want to get into green plum and testing green plum. And also, annoyingly, the oldest version of green plum is still based on post-gust 9.4, which we expired a couple of years ago. So it's a little bit painful to have. So that company is actually basically supporting now, not only do they have their fork, but they're also supporting versions of post-crest that we don't support anymore so that they can still support green plum six, which is based on 9.4. And maybe that's something we could bring into core postgres, but it's a resource issue.

We just simply don't have the resources for it. And it's just easier to focus on open source postgres. And when drive people towards the idea that if you make your stuff look like Postgres, it'll just work. If you muck about with PG control, like the parts of, so we even have a feature where you can add stuff to PG control as long as you add it to the end. And we'll automatically figure out. So you tell us your Postcode 16. we'll automatically figure out where the checksum lives based on the size of your PG control

and all that kind of stuff and work all that out. But the contract is that the part of PG control that was part of that version of Postgres must be the same. Because we read all over PG control. It's extremely important for our operation, so you can't move things around, you're just allowed to add to the end.

So we do make concessions to forks. Things like it works with EDB, their enterprise server, and other things that have new control versions and stuff like that, as long as you kind of know what you're doing. We call it a maintainer feature. So the idea is if you shouldn't be mucking around with this feature, if you are not a maintainer of a fork and deploying PG backrest for that particular fork or documenting how PG backwrest should be used with that fork or something. If you have to use these options on a regular cluster,

then you've probably done something wrong. You shouldn't be doing that. But obviously I can't control what people do. That's part of the fun of it, right? Yeah. Talking of maintenance, is it a good time to transition into how you have been maintaining it over the years, kind of what the situation was and it's changed recently? So I'm wondering if you wanted to share a bit of that. Yeah, maintenance has always been a big part of PG backrest.

One thing is we have a very comprehensive test suite for a project to the size. We test on five different architectures, we test on a variety of distros, we have 100% unit test coverage, we have integration tests, we have, we test our dot code or test code, or we test everything. And just keeping that test speed up to date is a bit of a challenge. It's not the core code. The core code doesn't really, they don't, the C code doesn't break because REL10 comes out. What breaks is our tests and the documentation and all these sorts of things. So it's a non-trivial task just to keep all that going. And then you have things like, say, Circe CI just going away on a month's notice. So we, we,

We need to migrate off a CIRCI. Luckily, we're already on GitHub actions, so we're able to do that fairly easily. When I say maintenance, I also include bug fixes that come in. So we get a pretty regular stream of bug reports, some of bugs or some are not. Most of them these days tend to be pretty weird edge cases. Enough people are using PG backrest that really obvious bugs

get caught pretty quickly. If we release something and there's bugs in it, I get reports the next day almost. bug fixes are maintenance. The documentation is maintenance. Interacting with the community and answering questions and feature requests and other things like that's all maintenance. So that's actually a pretty big part of the job. And after I left, after I did not transition to Snowflake, so Snowflake bought country, I did not transition. I wasn't happy with their terms. So I decided just to take some time and travel. I kept maintaining PG backrest, but at that point I was really just maintaining it. which is fine, but it really got to the point of thinking about really restarting development on PG Backrest again. I was like, I think I might need a job.

You know? And so I started trying to get, I actually put sponsorship links, and I've been including sponsorship with every release since last summer. So it was around for basically a whole year, not getting a lot of traction, of course, and then I started spending more time on it. And then I got to the point where I realized maybe this whole sponsorship, thing isn't going to work out, and that that got us to the recent crisis, where I announced that I wasn't going to work on PG Backrest anymore, and suddenly, I got sponsorship.

And when I say suddenly, it really was suddenly. Within three weeks, pretty much everything was worked out, and I was able to make an announcement that the project was going to continue to be maintained, and et cetera. But not just maintained. The key is adding new features. Right now, I've been doing, on PG Audit and PG Backriss, I've been doing a lot of cleanup work, because they've been sitting for a especially PG audit, wasn't really getting a whole lot of love. So I've been working on that, but I'll be done with that at the end of this week, basically. And then it's time just to dive into big new features again, which I'm really excited about. I've got a lot of ideas.

The sponsors have a lot of ideas, as you can imagine. The great thing is the ideas that the sponsors have are directly align with big ticket items that have been on our list for a while. They're not asking for outlandish stuff. They're asking for repo to repo backup, for instance, is a big ask. And that's been on the list for a long time, because that's a pretty important thing to be able to do. Streaming, doing streaming wall replication. So you can have RPO zero if you want without having a standby. So the idea is you have RPO zero without a standby.

PG-R-R-D-C wall, right? Yeah, although we would work directly at the protocol level. And in fact, in fact, we might, I'm actually thinking, given that we had this archive library thing and given that PGVACRUS can just live in Postgres, I'm not even sure if we need to deal with the protocol level. We can just sit there as a companion to Postgres and stream wall out. Isn't that expensive to sit as Postgres? Like, I don't get it. What exactly the idea? So basically, instead of a replication slot, we would just be keeping us.

with the current right wall location. Where are we currently writing in wall? That's the point that we should currently be targeting. And although we would still need a replicate, if we really wanted say synchronous replication RPO zero stuff, we probably still need a replication slot for that. The other thought we had is that we could actually game, we could create a replication slot but not actually use it and just update the row. So basically do our own background archive and then update the row in Postgres to our current position what we've actually pushed out to storage. Because it, and if I do this, I want to do it right. In a lot of cases, the things that are doing the wall receiver, they're not actually, let's say your eventual destination is S3.

They're not actually writing packets off to S3. They're storing the stuff locally on whatever host the wall receiver is running on. And then when they get a full wall segment, they're pushing that to S3. So what I would want to do here is actually have a couple of parameters that decide what your acceptable lag is. Let's say you're willing to have three seconds of lag, right, or so many bytes. And we can measure that. In this situation, we would be able to measure either one of those very easily, right?

So we would say, okay, we will, if we get to that point where we're going to hit that lag, we'll actually write a chunk of wall out to S3 and store it. And the wall archive get routine would actually be written so that it would be able to to, later these chunks would be assembled, of course, reassembled into a wall segment to keep things kind of tidy. But the wall get routine would actually be able to understand. If you get to the end of the wall stream and all you've got is these chunks, we'll be able to read them out and reconstruct them and send them post-cust. So you actually have, because I feel like the implementation is out there right now, yeah, you're getting the wall off of the primary, which is good.

Don't get me wrong. That's a pretty important thing. But I want to get the wall all the way to the repository all the way to the final storage. So if everything goes away, this is, and all you've got is the S3 bucket, then you still have the whatever you set that O2, sorry, then you would, you would have that unless we fall behind or other things happen. Obviously there's scenarios, but I think it could be a lot more efficient because then we could figure out, hey, they're generating a wall at a ridiculous rate. We actually need to switch over to just compressing and pushing whole wall segments and then come back to chunking up portions of this wall segment. And with the replication protocol, you don't really have that option because you've just

got this fire hose that's sending you data. And what you really want to do is say, no, this is too much. Also, it's a lot slower than just compressing and shipping. So we're like, so no, we're going to go back to doing whole segments and we get caught up on the whole segments. We'll start doing the most recent segment in chunks back and forth. And I think the best way to do that that I'm thinking, I'm always an outside the box kind of guy, is actually in an archive library, sitting next to Postgres and just asking Postgres, where are you, where are you, you know, kind of thing. And we can actually monitor, you know, check stuff on disk. So we'd be able to see, oh, we've just gotten a new wall segment, so everything in the old wall segment is our game,

et cetera, et cetera. So there's plenty of heuristics we can use to improve this. So that's the way I'm leaning right now and that gives us the archive library and this wall receiver RPO feature at the same time. RPO control. This is better RPO control, basically. Yeah, I'm sorry, I meant to say RP0. So if we could actually gain this so that we can update the not do streaming replication, but update the row in Postgres for replication, then we could actually do synchronous replication this way. And we could basically, you know, do synchronous replication to S3.

If someone wanted such a thing, yikes, it would be slow. I'm very curious, what is your opinion about? Barman was the first tool which used only streaming replication, right? I remember this. So what is your opinion on using streaming replication for backups? And this new cybertech tool released last week, PG hard storage. I quickly checked they don't use PG receive wall as well. they just implement protocol, they work with protocol and pretend to be reapply.

Yeah, that's what most people have done. I think barmen might still be using PG receivable, but most most other implementations have gone straight to the protocol. The protocol's not that complicated. Yeah. To be honest. Like whole uppercase. If you got LibPQ, once you've got that setup, if you're using, especially if you're using lib PQ directly, it's actually fairly trivial. The replication protocol is very simple, which it should be. There's nothing wrong simplicity. I'm all about it. I think ultimately it's a big bottleneck making backups through the streaming replication. So for very large databases, it's just going to be a pretty big bottleneck.

Obviously, someday we can add parallelism, do all these things and et cetera. There's a lot of... Compression. Well, they got compression, so that's good. Although, I think... Is it the compression server side or is it client side? I think it might still be client side. No, it's server side.

Okay. I can't remember. Anyway, that's actually a pretty solvable problem. But you've got basically a scale of now you need to back up something really large and you're pushing everything through this little pipe. BGBasebe Cup has compression since postgast 15 and you can, yeah, right? Or no. It does.

I just can't remember which side it happens on, whether it happens on the server side or the client's, I think it is. ... I believe it is, yeah. But it's still single thread. That's one bottleneck taking care of because you know. because you're not pushing uncompressed data over the network, which you definitely don't want to do.

But it's still single-threaded. And that's a huge limitation on the backup side, obviously. And then on the recovery side, we still have this problem of all the data needs to be recovered before anything can be done if you're doing block incremental. If you're not doing page incremental, then you can make that a bit more efficient.

You can stream the tar file from S3 and decompress it as you go and do various things. But if you're doing page incremental, which I think pretty much everyone wants to do, that becomes a huge bottleneck on the restore side. I'm sorry, like we never had so compressed. And I know we already are way over time, but it's so compressed.

You compress so much knowledge, and I like it so much. But can you explain, like elaborate page level versus block level and why you think it's important? Sorry, that's my confusion because what was introduced in PG-17, the page-level incremental with the wall summarized and everything. In PG-backrest, we call that block-level increment. And the reason why we call it block reddles, we don't always operate at the page, at page size.

We operate at block size. And then we actually combine blocks into what we call a super block for compression. So let's say I want to get, let's say I want to get one block out. I might have to retrieve three compressed blocks or four compressed blocks, depending on the super block size, to actually get that. But the important thing is we can, let's say we've got a one gigabyte file and we need to recover one block. we can go recover three blocks to do that instead of a thousand blocks, however big, however many blocks are in a one geek file, right? I think it's usually 1,200 at our largest block size. So we can go recover three blocks instead of 1,200 blocks, even though we have to recover three blocks just to write one block. So the main thing is we're able to go, let's say you've got a standby or a primary that's stale for some reason. You don't know why.

So you're going to do a Delta restore. So in a Delta restore, we go and we look at every file, and for the block level stuff, we chop it up into pieces, and then we do a hash on each piece, and then we do a hash for the entire file. If the whole file hash matches what's in the backup manifest, then we're done. Move on.

Next file. If the whole file manifest doesn't match, then we can go recover what we call the block map, which is a map that tells us where all the blocks live in all the backups, you might have 10 incremental backups since your last full, and now we need to go figure out where are the blocks that we need located, and go grab those blocks. And in the other implementations that have been done, you can't do that.

You basically have to go recover everything, and some of them will have good things like, well, as it's reading it, it'll know that it doesn't need the block and it'll just throw it away. That kind of readover thing I was talking about before. So it'll just throw away the blocks it doesn't need. But it's still reading them out sequentially. We're actually able to go random access and pull out just the blocks you need. And it's extremely powerful because we have that Delta Restore concept

where we can actually just recover part of a cluster. And that's where the power of those checksums that we spent, in theory, 5% of our time, but actually it's really much less than 1%. That's where that all comes in. On the PG-Basebackup side right now, you basically have to recover all the full and all the incrementals, decompress everything, all at the same time, present that to PG-based back, that will then rewrite that into another, sorry, PG- Combined Backup, which will then rewrite that into another directory. So at a minimum, to do recovery, you're looking at double

your database size and actually more. And you don't really know what that more is, unless you've stored it, it's up to you to actually store that information to find out what is the total number of bytes that I'm going to need to even pull down the data? On the PG background side, when we're doing a recovery, nothing ever hits disk except in PG data. So we're just reconstructing those files inside PG data. There's no spooling. There's no spooling during backup. There's no spooling during archive push. There is spilling during archive get because we actually pre-fetched wall files and we store them so that we can hand them over to Postco. So we spool on that side, but for the most part, we're zero disk operation. And to be really efficient, that's the way you need to be looking.

And right now, the design and base backup is exactly not that. A lot of disk I.O. They've done some tricks with if you're on ZFS or other file systems that can do copy and write tricks and do other fun things to try to minimize the number of writes that you're going to do. I don't know if it actually minimized the space that you need on some file systems. that might be true. Thank you for explanation. It was great. The whole idea was the feasibility of PG-based backup, using the streaming replication

as a backup tool. And I think, yeah, you can do it, but when you're working at scale, it starts to introduce a lot of limitations. And from day one, Gabrielle Bartolini read a pretty good thing after I announced that I wasn't working on PG Backrest about his and my fundamental disagreement about how this tool should work. And his idea is that the copy thing should be owned by Postgres. So Postgres should do all the file copying. And my idea was that no, the copy should be built into the tool. Because then we can do, the sky's the limit. We can do anything we want. And I think the performance that you can get from

PG-Backress speaks to the value of having that complexity. And it is, it's a lot of complexity. Don't get me wrong to have all that built into PG backups, but the payoff is performance. And also reliability. Like we, we have checksums that we can check on everything. We know when the data that's coming back is good. That's one thing that kind of peeves me is that if you, like let's say you were doing PG combined backup, it doesn't actually verify that the data is correct. So if you want to verify, you actually have to run PG-Verify backup as a separate step before you run the combined backup. And in my mind, those should always be combined into a single operation.

You're looking at the data, you're verifying it, you're writing it out. And the cost for that, again, oh, check sums are expensive. Yeah, but if you're streaming the data directly from S3 and check summing it and writing it out as it comes in, the check summing cost disappears. You don't see it anymore. If you copy all the data from S3 locally, decompress it, and then start running it, any check sums on it, it looks really painful because you weren't able to, the idea for me is let's game all the latency that we can, the storage latency that we can. So while we're waiting

for the storage to give us something, we'll send it the next request to S3, and while it's thinking, sending this data, we check some. We're done with that block. The next block is ready for us. We pull that in, we start checks something. We've asynchronously set the next request to S3. So we've already got more data coming. In the next version of Postgres, if everything goes well, we'll also have pre-fetched. So we won't be asynchronously fetching one block. We'll be asynchronously fetching by default four.

And just pulling that data in as fast as we can, but even as fast as you can get stuff over the network, that cost of checksumming disappears, but it gives you power. And Backrest is the only thing that has something equivalent to Delta Restore, and all of that is powered by the checksums. That's where all that comes from. And then we had the block level Delta Restore as well,

which is powered by the block level hashes, which are actually Xx hash, not Shaw 1. Because the blocks are a maximum of 88K. Although you configure that, you can also configure how much of the XX hash you're going to use for a particular block size. But we did a lot of analysis on this and looking at file systems and how many bits they were using to protect, blah, blah, blah, et cetera.

And basically came up with checksum sizes that were appropriate, for each block size that we support. The whole, the way we do block incremental is actually a pretty complicated topic. I did a talk on it once and it was a complete disaster because it was just too much. You go too deep, you're not getting 32 bytes, you said, right? Or or truncating hashes you said, right? Oh yeah, so the 32 bits. So the, so we do is we generate a 120-bit hash and then we use whatever, however many bits out of that, well, bites actually, that we want. The minimum number of bytes would use for an 8K block is six bytes, which is actually a lot. That's 50% more than would be used for 4K pages on ZFS, for instance, or butterFS. So we go a

little bit crazy on the checksums because we just, we want everything to be right. And then when you're doing block incremental, we actually have two levels of checksums. So we use the block checksums to reconstruct the file. And then we also check sum the entire file. And the Shaw 1 hash of the entire file plus all the block checksums have to match. And the chances of that getting a collision, like a piece of data that actually satisfies the conditions, all the block checksums match and the Shaw 1 checksum of the whole file matches. We're talking heat death of the universe probability here, right? It's just simply not going to happen. You're going to have Discorruption a billion times before anything like this ever fails, as far as my math works

up. It's a lot of interesting topics, I feel. And I like the direction you are thinking, especially RPO control. This is a super cool thing because I see the demand because PostGIS has been used in critical systems and people just don't know what's happening. there, like how, what's happening in the case of disaster, right? Like, it's hard to answer simple questions to business leaders. Yeah, it's not one size fits all, right?

You guys are familiar with Aaron, right? The internet number registry. I know the people who run the database over there, and they use PG backrest. I work with them for years in another company. And they're all RPO zero stuff, right? But the right volume is stupidly low. So they've got one primary, which receives. 100 rights an hour on a really busy day.

And then they replicate that. And then you've got the whole internet reading to figure out what blocks are allocated to who at any given point and where they should be routed, that kind of thing. So the read volume is pretty big. The right volume is extremely tiny. So they do synchronous replication because they can't lose anything. And they have low enough right volume that can get away with it. So they would be a great candidate for PG-Backrest RPCZero solution that allows them to synchronously write to S3.

I bet they'd love it. But for most people, that kind of solution just isn't going to work. People ask me how I can work on this software year after year. And the reason is because the problems are actually really interesting and challenging and extraordinarily complex in terms of the solutions that we have to come up with to solve these problems. It's, it's, P.G. Backgris is a relatively small project, but it's dense. The stuff that we do is... Let me compress a little bit. You mentioned Aryan.

It's American Registry for Internet numbers, right? Right. That's the one area. So it's basically an important piece of Internet, right? Very important piece of the internet. Yeah, very important piece. It's interesting that you consider like volumes are not huge and, yeah, streaming replication and so on. Yeah.

It's interesting, interesting and challenging. But I have cases where it's like just not answered, RPRC are not answered, and extremely important cases. Like important companies also like they simply don't know. So if you build this, I think it will be good. It will be useful, helpful, and so on. It's a tricky thing, though, to some extent when you put those tools in people's hands, because of course there's some level in the hierarchy that's going to say always RP zero. Of course, we have to have synchronous replication.

We cannot afford data loss. We need to use the synchronous application and so on. And also we cannot. We don't have split brains. You mentioned Gabriel Bartolini, right? I also have my opinion about Cloud Native PG and solutions he chose in this tool. Yeah, but like at high level we are great. But if you look at inside, you see data loss is possible, split brains are possible, almost everywhere.

And it's really extremely hard to achieve. And I like so much you focus on these topics, these topics like RPO control and especially corruption and check sums everywhere. It's great. My whole job in life is to protect the data, protect everyone's data. And that's what I think about all the time. Like how can PGVACros be the most reliable thing, even if it's not the fastest or I think it's generally the fastest, but if it's not the most feature rich, there are a couple of features that other programs. have that we don't have, although we're working on that. But everything we introduce has to be performant and ultimately absolutely reliable. People trust the software to protect their data. Now,

they have an obligation too. They need to, if I get people at conferences that will come to me and say, you save my job. PG backers saved my job. And I'm like, I really appreciate you saying that, but actually you saved your job because you set up backup. You tested restores. And when the big day happened, you were ready. Yes. That's the restores. Because they'll tell me, oh, yeah, we set up weekly restore tests like you recommended. And I'm like, okay, great, I can recommend these things, but a lot of people don't do it. So if you actually go and do the stuff that the backup people recommend,

kudos to you, you deserve all the credit for saving your job, not me. But don't get me wrong, I still like hearing the stories. It's great to know that PG Backrest is useful and valued, of course. Untested backups are Shredinger backups. They should not be considered proper backups. Yeah, backups must be. I can't remember where I found that, but I put that on one of my very early talks, Schrodinger's backup. It wasn't me.

I wasn't original to me. But I can't remember I saw it. But as soon as I saw it, I was like... It's an obvious idea, honestly. Multiple people might invent it. Anyway, way over time, I enjoyed so much talking to you. Final question, what, like we discussed a lot of technical detail, like very, very advanced, I must say. I think it's okay if some people don't get everything because we definitely dived into multiple areas very deep.

We also touched the situation about sponsorship. We haven't touched the situation about maintainers. What would help you to have a second big maintainer? A second big maintainer would be good for a couple of reasons. One, to help me write the big features. These days, I'm the only one writing big features and I'm just one person. And the other thing is review. So review is a big bottleneck for me.

I can actually produce quite a lot of code. But it's got to be gone over carefully by somebody. I'm using LLMs now for review as well. So I'll ask Cloud code review, anything I write before I send it to any person, just to catch the really obvious stuff. And these days, the not so obvious stuff, it's getting pretty good. But I also need someone to bounce ideas off of. Another maintainer would be good for that.

Someone who I could reliably chat with and be like, So I was thinking about X, Y, and Z, and I just, you know how it is when you vocalize something, it becomes, sometimes you don't even need to get input from them. Just by explaining the problem to them, you, oh, yeah, I know what I need to do here. It's so obvious to me now. So that's why I really need another maintainer. We're working on that right now. That should be, I don't want to make any announcements or say anything regarded it, but this is something that's actively being worked on to get another, maybe not full-time maintainer. but very active, very involved maintainer.

That would be great. And to some extent, it's hard to know how big we could scale this project. It's an interesting, how many people are interested? Also, how big are the problems we're solving? Can two people handle everything? Probably, especially with tools these days helping. We're getting more contributions from the community. Those are coming in a pretty regular stream now, so I don't have to write everything.

I don't have to find all the bugs. I don't have to fix all the bugs. I don't have to do little authentication tweaks for S3 or blah, blah, blah. Most of that is coming from outside contributors, and I just review it. And if it's reasonable, commit it. So we're building community that way as well. But if we can get one more person who's really looking at it regularly, that would be great. And that would, of course, be in addition to Stefan, with Data EGret, who actually already spends quite a bit of time on PGVACROS as well, review and testing.

etc. Cool. Great. Thank you. I don't have any more questions. I have, but I will. Yeah. Post all them.

They're super technical again. David, thank you so much. Thanks for joining us, but also thanks for all the maintenance you've done over the years. Oh, absolutely. It's definitely been my pleasure. Yeah. Congrats on getting all the sponsors.

And thanks to them as well for keeping it going. Yeah. Like, kudos to the sponsors. They may have made this possible. I'm just, I'm back to work. I'm back to what I love doing. and everyone gets to benefit. So I think this has worked out really well.

It's a really cool open source story. Yeah, it is absolutely good. A great open source story. Thank you. Have a great week. Yeah, thank you very much for having me. It was a lot of fun. Bye.