Postgres FM - hot_standby_feedback

Episode Date: September 11, 2026

Nik and Michael discuss hot_standby_feedback — what it controls, whether and when read replicas should be changed to on, and other settings you can change to help too. Here are some links ...to things they mentioned: hot_standby_feedback https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-HOT-STANDBY-FEEDBACKhot_standby_feedback: experiments on PostgreSQL 17 (by Nik) https://console.postgres.ai/shared/brief/6018f69d-8975-48e3-8259-a72c77105572Best practices for Amazon RDS PostgreSQL replication (blog post) https://aws.amazon.com/blogs/database/best-practices-for-amazon-rds-postgresql-replication/log_autovacuum_min_duration https://www.postgresql.org/docs/current/runtime-config-logging.html#GUC-LOG-AUTOVACUUM-MIN-DURATIONmax_standby_streaming_delay https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-STANDBY-STREAMING-DELAYWAIT FOR LSN https://www.postgresql.org/docs/19/sql-wait-for.htmltransaction_timeout https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-TRANSACTION-TIMEOUTHot Standby Feedback should default to on in 9.3+ https://www.postgresql.org/message-id/flat/20121130190227.GJ3957%40awork2.anarazel.de~~~What did you like or not like? What should we discuss next time? Let us know via a YouTube comment, on social media, or by commenting on our Google doc!~~~Postgres FM is produced by:Michael Christofides, founder of pgMustardNikolay Samokhvalov, founder of Postgres.aiWith credit to:Jessie Draws for the elephant artwork

Transcript
Discussion (0)
Starting point is 00:00:00 Hello, hello, this is Posgous FM. My name is Nick, PostGos AI, and as usual with me, Michael Gimastered. Hi, Michael. Hello, Nick. I propose to discuss hosting by feedback, and that's interesting because I have four customers with this problem right now, different platforms.
Starting point is 00:00:19 Oh, wow. On different platforms, right? Yeah, on Superbase, on RDS, and something else. It's very interesting how problems sometimes are very, very silent until suddenly they come from multiple directions right? I don't know what's it's happening right but it's just happening
Starting point is 00:00:37 all four are they all suffering from the same direction yeah okay interesting so all of them are complaining about read replicas lagging and it's all because platforms most platforms
Starting point is 00:00:53 keep hosened by feedback off by default like post Bs itself has it off by default while my honest opinion, like strong opinion, I would say, is that you cannot have read replicas with off. For general OLTPs with many users, and I say this after a lot of experience,
Starting point is 00:01:19 I've built three social networks with each one of them achieved many million users and one plus Dow daily activity. users. And we needed to scale reads, of course, right? So we needed read replicas. The first one I built in 2006. And also I helped many companies who also needed to scale reads.
Starting point is 00:01:43 My honest opinion, you cannot have read replica with hosted by feedback off, just because it will start lagging. But this is how it is. Like you, all people need to rediscover it because defaults are, this is our favorite topic, right? defaults are terrible in many aspects. Well, yeah, I'm actually not sure.
Starting point is 00:02:05 I fully agree on this one yet. Maybe I will by the end. You are not alone. Yeah, I'm not sure. It's a really difficult one, I think, this one, in terms of, like, risks versus, yeah, when things buy you. But maybe let's go back to basics.
Starting point is 00:02:22 Like, what's the... Yeah, yeah, yeah. So we should discuss, and I just threw it. at the very beginning to set my goal. I'm going to discuss it in detail and explain my way of thinking.
Starting point is 00:02:36 But just separately, I'm just wondering, are there people who really scaled reads on really large web or mobile apps and head it off? Because I don't understand how it can be possible unless you are
Starting point is 00:02:51 okay, your application is okay with licking replicas. Sometimes it's so. If you serve traffic to some users and some stale information like lagging some seconds or maybe minutes is fine I can imagine that you basically show some news for example or blog posts or something but in general modern applications
Starting point is 00:03:13 like social media applications e-commerce and so on where you want to scale reads to asynchronous replicas of course we talk about asynchronous replicas here then we do need host and buffing back being to be on Anyway, let's start from the basics. We had several episodes where we covered basics already. I just wanted to have a quick recap. First of all, my favorite thing,
Starting point is 00:03:38 if some people don't realize, there are hidden columns, Xmin X-Max. X-Mex is transaction ID which gave birth to this TAPL. TAPL is a raw version, right? And there is a concept of X-MIN Horizon, and we had a separate episode about it, which defines global for whole database,
Starting point is 00:04:00 global horizon, which says, this is the oldest transaction ID that defines data snapshot which is still needed to some transactions, some clients, right? And it means,
Starting point is 00:04:15 and this is global, a single value, basically. There are many values coming from different sources, but horizon is absolute minimum of all of them. And it defines the garbage collection behavior, vacuum behavior, right? Because when delete or update or canceled insert happens,
Starting point is 00:04:36 dead tuples are produced, right? And the job is not done fully. The remaining of the job is garbage collection. Dead tuple must be deleted later. And if this debt topple belongs to a transaction which is newer then our X-Men Horizon vacuum cannot delete it. And this is simple logic, it's global. Some people think it's maybe per table,
Starting point is 00:04:59 but it's not as global. Yeah. And this is all very simple on the single node. If you're only worried about having a primary database or maybe only an H.A. replica, no reads going on elsewhere. It's nice and simple, right? Like, we just hold on to the very versions
Starting point is 00:05:18 that might be needed by the oldest theory or whatever the process might need it. And then once that's finished, we do all the cleanup. Or vacuum does it clean up. And here we need to dive into some details. So yeah, let's first think about only the primary
Starting point is 00:05:36 and single node situation where usually people start. You have, there are five reasons, five sources in our monitoring covers them. And we discussed it in the Xmin Horizon episode. Five reasons to block X-Men Horizon.
Starting point is 00:05:52 like if all transactions are fast, they finish promptly. No replication is used neither logical nor physical and you don't use repaired transactions. In this case, Ex-Mitt Horizon flows naturally, and you can calculate
Starting point is 00:06:10 age of XMMMEC horizon, meaning that how many transaction ideas, real transaction ideas, passed since this horizon and ideally it's very low. 1000 is quite low. 10,000 is, it's quite low. One million, it's already
Starting point is 00:06:24 noticeable, right? And we all know that one billion is already basically half of capacity we have. Because if the whole capacity is like 2.1 billion roughly right? We have integer 4 byte
Starting point is 00:06:40 integer transaction IDs and half of that space is our future, half of is our past and we need freezing vacuum one of its drops. It has multiple jobs. One of its jobs is to freeze TAPL
Starting point is 00:06:55 saying that don't look at this X-Men, it looks like from future, but it's from the past, right? And if we don't freeze during 2.1 transaction ID, if our XMN horizon is that old, age of it,
Starting point is 00:07:12 is that old. We can use function age, actually, right? In this case, it means that we're going to be in big trouble and database will be not accessible because transaction ID up around will happen. This is the biggest danger, actually, of X-Men Horizon being blocked for
Starting point is 00:07:28 long time. And in the case of single node, of the primary, everything is relatively simple, but I noticed many people don't realize nuances here. Remember, this year I started saying that the
Starting point is 00:07:43 warning, let's avoid long-running transactions, is quite inaccurate. Right? Remember this? We talked about. And I noticed actually even AI is making a lot of mistakes. For example, with GPT-6 Astra, I decided to renew graphics in Pedersim City and also make it more entertainable and have some bad scenarios to be reproduced and so you could see what happens.
Starting point is 00:08:10 And it wrote a very simple thing and I see it quite often. It said, somebody typed Begin and left transaction open for a day and we're in trouble. But it's not simple. Begin is not a trouble. Because by default, we have read-committed transaction isolation, which means that after each statement, snapshot is free to go.
Starting point is 00:08:35 Basically, we don't block X-Men Horizon. So if we think about transaction at default transaction-resolution level, read-committed, if it consists of multiple statements, whilst each statement lasts, of course we need bad snapshot and we block X-MIN horizon. We need to work with that data
Starting point is 00:08:51 some select is running on some huge table, yeah, it lasts for long, and we need that data. But once this select finishes, even if transaction is still open, that snapshot is not needed, and basically we allow XSynth Horizon to shift, right? And simple begin doesn't hold any snapshot at all, right,
Starting point is 00:09:14 at default isolation level. So, yeah, it's very inaccurate, it and that's why saying, oh, long-riding transactions are dangerous, it's too rough. It's not a precise statement because not every long-running transaction is dangerous. Yes. Some are, of course, but not quite all, right? There are lots of cases where they're not, yeah, sure. Right.
Starting point is 00:09:40 Of course, we don't talk about locks here because there are two topics. Locks and Exmead Horizon blockage, two dangers of long-grinds. transactions. So we need to distinguish read-committed transactions and transactions running at higher isolation level, repeatable read, and serializable.
Starting point is 00:10:03 And repeatable read is pretty common because this is how PG-dump works. So every time you run PJ dump, even if it's not doing, of course it's doing something always, right? But it captures snapshot in the very beginning of transaction and even if you dump a lot of
Starting point is 00:10:19 tables you select from them the pitch dump selects from them since the next table snapshot is still the same as it was defined at the very beginning of transaction it means if you start a transaction at repeatable read that's enough
Starting point is 00:10:34 to be harmful right you just say start transaction or begin isolation level I always forget syntax you don't need even to run any queries inside it it's already harmful it holds
Starting point is 00:10:49 you say harmful it starts to hold the X-Men Horizon right right it affects yeah short holds are not expected right hold the reason it's being held
Starting point is 00:10:59 is for a good reason those if you're doing a a dump you want probably everything and not just consistency but you want the data
Starting point is 00:11:08 at that point in time and if it's changed then you want like you still need those you can't have those roads being cleaned up so you this is how consistent
Starting point is 00:11:17 is defined, right? Yeah, yeah. Because if you read from two tables from different points of time, this is inconsistent dump. So we don't want that. That's why we need to be a computer read. But you say it's harmful.
Starting point is 00:11:30 It just means we're holding on to a bit of extra data, maybe bloating some indexes a bit. Like there's an amount of that we should always expect we're running posts, right? There's a certain amount of it that will always be natural. We try and minimize it and we try not to let it get excessive. but then if that goes on for too long, it gets into that boundary where it is maybe excessive
Starting point is 00:11:56 or causing other issues. Is that what we're trying to say? So harmful, yes, but like it scales with the amount of time or maybe like time multiplied by the amount of churn in that time. Yeah, yeah, yeah. I usually say you need, like, it's really hard to understand ageing number of transactions. although it's more representative metric in this context.
Starting point is 00:12:22 If you say long transaction running, like three hours during busy time at high traffic is not the same as during weekend or low traffic, right? Because it might be even nothing happened during three hours at weekend. Sometimes we see some applications with very acute spikes of traffic, right? But so there is no very clear connection between time. I name it XSIT consumption or XSID growth rate. So how fast real
Starting point is 00:12:52 transaction ID grows? Because this is what defines this Xmin Horizon age. And one of the contributors, by the way, save points, sub-transactions. If you use them a lot in one transaction, you consume many real transaction IDs because every safe point gets
Starting point is 00:13:08 allocation of XIT, real XIT. So this is one of the dangers I identify that safe points lead to higher consumption of XS. So yeah. And yeah, of course, like, if we block a vacuum, also we need to be very accurate with language here
Starting point is 00:13:26 because I realized also when we say vacuum is blocked, some people think it's blocked it like fails. I even saw, so I have a new tool to be released soon called PGBS detector, which is going to, and we started using it internally to identify bullshit in our write-ups. And I saw RDS article from the, 2019 and this was the only article talking about Hotson by feedback from RDS. Nothing else.
Starting point is 00:13:53 So I tried to understand why they keep default off and what's the reasoning behind it. So I saw that in that article, this, like, very confusing statement, like vacuum is blocked and then error, like canceled statement, like it was vacuum statement canceled. No. When we say XMil Horizon blockage leads to... affects vacuum negatively. It's not like vacuum fails or cannot work at all.
Starting point is 00:14:22 It works partially. It cleans up that dead topples which became dead before X-Men Horizon. And it just skips and reports it in the logs, tuples which it cannot clean as dead. Already dead, but cannot be cleaned up yet. I don't remember.
Starting point is 00:14:38 Cannot be removed, cannot be deleted. I don't remember exactly wording. But this is what's useful to see because you can see for each table a vacuum visit. auto vacuum visited. You can see what is the damage for this. You can also run vacuum verbose and verbose will report such tapels which are dead but cannot be deleted because Xmin Horizon. It actually also reports Xmin Horizon. It just prints a transaction ID and
Starting point is 00:15:05 what's the age of it, which is very useful as well. So that's why auto vacuum logging should be enabled and maybe not zero, which means all occurrences of auto vacuum But maybe at least above a few seconds, 10 seconds or one second, if you're concerned about some very often executions, which are very fast. So anyway, auto-viking logs are quite useful in this area. What's next? I think we've not really even discussed what Hotstandby feedback does.
Starting point is 00:15:37 We've talked about the retention of dead doubles while something's happening. but Hot standby feedback comes in because it's something happening not on the primary, right? Like it's something happening on your replica, yeah. It's conflicts. So if you think how replication works,
Starting point is 00:15:56 physical replication, it replaced wall like it was recovery. It was built on top of recovery from crashes, right? And the hotsand by feedback is just constantly replaying wall like it was a recovery. That's why a function to check is it primary or standby
Starting point is 00:16:12 is called PG using recovery, which is confusing, right? And it replaced Wall good, but what if we have some query ongoing selecting from a huge table, for example, reporting, right? It's pretty common. Let's offload heavy reporting queries
Starting point is 00:16:27 to read Repreperance. If it's running there and Wall has instruction to clean up that couples, which on the primary already dead, but we're still reading them. This is one of conflicts, and this is snapshot conflict
Starting point is 00:16:41 of applications. conflicts with a query or query or transaction conflicts with this wall data, right? It cannot be replayed. So depending on the type of physical standby, one of two settings are used
Starting point is 00:16:57 to decide what to so what to do is obvious. Replication is post. It cannot be replayed. That's one option, yeah. Yeah, this is default option. So we wait until this, like until the end of this transaction.
Starting point is 00:17:12 And then we'll continue replaying wall. Yeah. We wait until X-Men horizon shifts. If it shifts, we can replay it. So you can see it manually in produced activity, back-end X-min column. And also it's report, yeah, so the first is back-li-backed soon. So, back-end X-min is for each backend, you can see this local horizon for each back-end. And the minimum of them is our horizon for this node.
Starting point is 00:17:42 right? It waits. It waits not forever. There is a setting, I believe, it's called Max Host, Max standby, Max streaming standby delay. Yes. There's Max standby streaming delay and Max standby archive delay.
Starting point is 00:18:00 I will never remember all Googles. So there are two, one for streaming, one for wall replay replicas which are consuming walls from archive, like a three archive or something according to restore command but more common because they have smaller legs is streaming replication which connects to the
Starting point is 00:18:19 primary unless we use cascade replication and the streams walls and applies them and this is when we use max standby streaming delay. By default I think it's 30 seconds it waits maximum 30 seconds after this
Starting point is 00:18:36 Postgrease says replication is more important and just cancels the query. And the first thing happens, people don't like queries being cancelled, right? Or this important reports. Especially sometimes long ones. Like if it's a long reporting query,
Starting point is 00:18:54 right, you've already waited 10 minutes for it. And that's very visible because reporting usually is close to decision makers. So they start complaining very fast. Like, why report was not delayed? Okay, we have conflicts. Canceled, okay. Let's increase this to three minutes.
Starting point is 00:19:10 minutes, 30 minutes, three hours a day. And it's okay. Yeah. At the extreme you can set minus one. And so replication will wait until, yeah. And this whole time, I think it's really important to stress, this whole time, anything newer than that conflict is not getting replayed on the replica. So no, any new inserts, any updates, any deletes, nothing's coming through.
Starting point is 00:19:40 Yeah, it's global. Like, if the Horizon is global, replication is just a single process, actually. You can see a startup process in Top or PS. And this is what applies changes. By the way, changes might be received already because replication has stages. They can be flushed to local PG-Wall directory,
Starting point is 00:19:59 but they just cannot be applied because somebody needs very old data, which we already want to clean up. All right. So, So what happens next? Reports work. For example, we set three hours,
Starting point is 00:20:15 all reports last no more than one hour. We are good. But then, usually what happens, some users started to complain. Stale data. And if your load balancer is not smart enough, if you have a smart load balancer,
Starting point is 00:20:32 by the way, you mentioned PostGGS-19 has wait for a LSD feature, right? We probably should not talk about it. I have a lot to talk about this. But smart load balancing understands that some replica is lagging and stops using it, just because we don't want to deliver
Starting point is 00:20:50 stale data to our clients. But if you have pretty basic load balancing and some read queries, red transactions went to read replica, it takes some time because usually who suffers is not decision makers, like management of the company wanting reports.
Starting point is 00:21:08 but some users. And they also, usually, not every user is willing to waste energy to report problems, right? It takes time, usually. That's a problem. It takes time and also, you have to rule out other issues first, right?
Starting point is 00:21:22 You have to make sure it's not on your side. Like, there's, just because something seems like it's not working, you don't automatically assume it's a service. Yeah. And what I'm describing is a very common situation, which I went through in 2006 or seven and just for,
Starting point is 00:21:38 thought it's solved, but due to defaults, especially defaults on managed post-GuS platforms which keep hots and buffback off. This is what's happening. So people increase max standby streaming delay, and then they just say, okay, we have replication legs, and they start thinking, why? Because it's not obvious, why? It's not really not obvious, because, yeah, it's just legs and so on. So should we switch
Starting point is 00:22:08 Should we move to then why we now have Hot standby feedback and what it does? So Hots and by feedback reports XMIN horizon observed on replica to the primary So the primary can involve it in calculation of the final XMISO use by vacuum
Starting point is 00:22:28 deciding what can be cleaned safely what still cannot be cleaned So if you had one hour quick on the primary which led to bloat sometimes, right? Because XMith Horizon, again, global it was blocked. And then you offloaded to read replica. Without HOSS&BitFatBek, you have legs up to one hour because your query is a one hour.
Starting point is 00:22:54 And it's terrible leg, right? Or when you switch to HOSM by Feedback to On, you have the problem, like, XMithRISN being reported, and you have the similar situation as it's a was executed on the primary itself, right? And I think the critical thing to mention is the chance of you then getting conflicts are massively reduced because the primary won't have cleaned up those dead tuples
Starting point is 00:23:22 and therefore won't send those conflicts through to the replica until it's finished what it was doing. So it avoids that case we were just talking about in most... That's why leg doesn't happen. So conflict leads to the... lack. Conflict leads to two things. First leg of replication and when it's too much lag, then
Starting point is 00:23:42 canceling the source of the problem query. Yeah. Right. So, one block Xminor Horizon progress. So you're right. I liked that you used to reduce number of conflicts because it doesn't eliminate him. For example, we talked about the
Starting point is 00:23:58 conflict when we need to clean up but Tupper is still needed there. This is a snapshot conflict. But there might be also conflict when we change schema, we need access exclusive lock. And the mistake is to keep this access exclusive lock for long.
Starting point is 00:24:14 And on the primary, it leads to many problems. We know, like some selects even will wait. Accesses exclusive lock blocks even selects. Schema change. So if you open transaction, added a column very briefly, and then started to read something from somewhere.
Starting point is 00:24:30 You keep the lock until very end of transaction as we discussed many times, lock cannot be released midway. It released on at the end. If you do something else this lock is held and nobody can work with that table. But on this replica also if you keep this lock on the primary,
Starting point is 00:24:47 wall comes saying lock and again a leg will happen. And Hotson Bitha doesn't solve this at all. This should be clear. So yeah, you're a very good warning. I liked it. It's not solving fully. Even more. I learned it only recently.
Starting point is 00:25:04 So if you usually use Standard with replicas these days are used streaming replication plus replication slots. Replication slots were created later than streaming replication. It's like additional level. And you can use streaming replication without slots. Usually people think replication slots are protection from walls being deleted from the primary. Replicate legs too much.
Starting point is 00:25:29 It cannot converge anymore, right? It's solved usually if you have good backups in S3, object storage, you can configure, restore command on replica, and it will just take walls from there, no problem. So slots can be like mitigated. Also slots are convenient for observability, but what I also learned, imagine we have streaming replication without slot.
Starting point is 00:25:50 Slots have X-min horizon reporting, right in PG-D replication slots. You can see it, X-min column. But if you don't have slot, streaming replication, it works all good. But then some network issue, replica disconnected, On the primary, for example, vacuum deleted tuples,
Starting point is 00:26:08 the reconnection happened, vacuum already deleted apples, and wall is delivered that tuples are deleted. And despite Hots and Byfit back being on, we have a situation like it was off. This was an interesting finding for me last week when I dived into this topic deeper, using our new tool for experiments,
Starting point is 00:26:27 reproduced a lot of failure scenarios, including this one. So it's interesting. It means that slots are useful in many different cases also. like the census, right? So, yeah, Hots and Beth feedback on drastically reduce
Starting point is 00:26:41 a chance that you will have a leg on standby. And this is what we want for read the replicas because we want them to be up to date as much as possible. But it also does one other thing that I think is quite important. It should also drastically reduce the number of cancellations you get if you have these long-running queries, let's say longer than 30 seconds or whatever you set that timing to. it should reduce those getting cancelled because the conflicts aren't going through
Starting point is 00:27:10 because you're not getting conflicts in the first place, right? Consolation is a result of delay, which is a result of conflict. So conflict, exactly, yeah. Delay or replication lag, and then cancellation when it's too high. The value of lag is too high. So that's why I think RIDERDARICS should have hosted by feedback on and reasoning that it will be worse because they will lead to bloat. This is just how Postgres works.
Starting point is 00:27:40 Usually people start with all queries going to primary, then they offload queries to Replicate. There's just a mistake to expect that those queries won't affect vacuum behavior. That's it. Of course, you can live with off. Actually, two points here. First point is quite interesting.
Starting point is 00:27:59 So we want to minimize them, of long-running transactions which block X-min horizon progress, and how can we do it, we can set transaction time out, both on primary and all read replicas. For example, if we set it to three hours, that's it, no more three hours of damage. But as we discussed, damage is relative. Three hours, we don't know how many transactions happened, right?
Starting point is 00:28:28 I think, actually, it would make sense, it might make sense to have all timeouts we have transaction and timeout we have statement time out and idle in transaction session time out you know transaction timeout limits whole transaction which is good
Starting point is 00:28:42 but I think it might make sense either to have all these three measured in not in milliseconds or seconds or but in a transaction count imagine transaction time out but count of transactions right or adjust hosting by feedback so it would be not like
Starting point is 00:29:01 on and all, but some threshold after which we cancel. Yeah, so there is definitely room for improvement here, and if you look, my AI told me other database systems ship settings for replicas. PostGIS ships two extremes
Starting point is 00:29:17 on and off. But with transaction timeout, at least measured in seconds, right? It's like indirect, but it's already good enough in many cases, which was implemented at PostGGGGIS 17. So all PostGist it's not available.
Starting point is 00:29:32 But it's good enough. My recommendation, on and three hours transaction timeout, so nobody could leave transaction open which blocks XMHRIZN progress for a very long time. But you need to do it
Starting point is 00:29:45 on the primary as well, right? Because who knows what happens there? It's no difference here. I think the big difference, I think the big difference is people not offloading just like OLDP traffic to a replica.
Starting point is 00:30:00 I think it's, people think that I've got a replica, I can send my reporting or analytics queries there, or I can give a data team access to that, and it doesn't matter. It protects the primary. And I think the key learning from this setting is that's not true. Like, you have to be, it can still affect the primary, and therefore you have to factor that in almost like an architectural level. Should you even be running those reporting queries there if it on an OTP, like an important OTP system, or should you not do that?
Starting point is 00:30:33 Like, I think it raises those important questions. Yeah. For reporting replicas, maybe you should keep it off, but the consequences are like very often replication legs. And I even say that it makes like basically this replication replica like single user. Like you take, run a very long query, all other users like suffer and cannot use it anymore. say, oh, it's very old data, we need fresh data.
Starting point is 00:31:01 So, like, you become very, like, expensive user. But maybe it's fine in some cases. Like, maybe it's better to do that and then catch up. I don't know, but it's quite expensive. Yeah, it's quite expensive to have a whole node for reports, which, and this node is lagging and catches up. I don't know, like, it's... But some people do, like, shipping it to, like, Click House.
Starting point is 00:31:24 We did whole episodes on this, didn't me? And I think there's some interesting... alternatives. But this is not Rezreplicate, it's an analytical replica, right? Or if you have... Yeah. If you have different storage, there are some
Starting point is 00:31:36 not Postgres, but some alternative to Postgres which stores data differently and executes queries much faster. In this case, maybe Hots and Wife Back On is still good because negative effect to vacuum will be low, right? I can imagine in some cases where off is good,
Starting point is 00:31:52 for example, if it's a delayed replica. Some people keep eight hours, 12 hours, delayed replica, which replaced wall with delay 10, 12 hours, to be able to very quickly restore pointed time recovery. I think this recipe is quite outdated because now we have snapshots, and even with lazy load, it's quite fast to provision multi-terabyte databases from cloud snapshots.
Starting point is 00:32:18 But if you use delayed replica, of course, you don't want hosted by feedback. I actually think it's not possible to use it there, because hosted by feedback works only with streaming replication, right? So with Replicas which work using Restore command replaying walls from archive or from someplace hosted by feedback cannot be applied there
Starting point is 00:32:40 and you cannot see them in replication slots so these replicas are invisible and this is how delayed replicas should work and in this case you will be dealing with the same mechanics but defined by a max standby not streaming
Starting point is 00:32:58 but what the other alternative archive archive delay might stand by archive delay so I think this is this covers quite well this topic I think PostGar has opportunity to improve things
Starting point is 00:33:13 drastically for better control I wish like I had a capability to define the damage not in seconds but precise like maximum like 100,000 transaction IDs. Ideas.
Starting point is 00:33:28 Yeah, and if it happens, if somebody exceeds it, rather I want to cancel that, that one. And for reporting,
Starting point is 00:33:35 the most standard approach for large databases, I agree with you, it remains in other database system and you need to create some pipeline
Starting point is 00:33:45 like a logical replication pipeline to ship data to Clickhouse or anything else like a small flag. Yeah. Yeah.
Starting point is 00:33:52 We could monitor for that. Like, you can do that You can build that yourself, right, if you monitor for things holding back the X-Men Horizon, and you can check the agent in transaction IDs. So that's already possible today, right? But we just have to do it ourselves rather than a configuration parameter.
Starting point is 00:34:13 Yeah. So, yeah, you are talking about some automation, which will be outside post-guards, but it will monitor all five reasons for fixed-min horizon being blocked, identify them and then cancel them and then fight them. It exceeds some, yeah. Yeah. You're like, you're talking about self-driving, it is.
Starting point is 00:34:32 But it should be a post-guess in my opinion. This is just belongs in post-base. This logic is too basic, too fundamental. It's possible to implement outside. Definitely. I remember actually many times we implemented like something in PG-Kron or regular Kron like canceling long-running transactions which are like offensive.
Starting point is 00:34:52 Looking at a back-and-examines. on. Yeah, we did it actually. And this was like duct taping, right? Yeah, sure. This is not, but I'm talking like, I just feel, I feel the need in Postgres itself could be improved. Maybe some, some listener should propose some patches. I think it would be tough to, I think it's, I think this is a really tough one because it's a genuine trade-off. Like, I don't feel like there's a right solution and it's going to depend on like what you're using your, replica for and maybe there's like a case that 95% of people want and like therefore we should just change it but it feels to me like some people do prefer the trade-off of having it on and some people prefer the trade-off of having it off and it's maybe tricky going to be hard to find something everybody's actually happy with yeah actually I think for yeah this is yeah I think which to leave the choice on the user who provisions read replica
Starting point is 00:35:54 it's also a good idea, right? To make... Okay, so yes. By the way, I've read, I saw many hackers proposed to switch default in Postgres itself to own. There was such opinion.
Starting point is 00:36:04 Very strong one, multiple, very well-known hackers proposed it. But then it was discussion, it was a position to it. By the way, at that time, transaction timeout didn't exist. So maybe now it should be reconsidered.
Starting point is 00:36:17 Yeah, it was before Postgastricht. And the reason I remember, which hits in my mind brightly. It was named that problems with legs and conflicts cancelled,
Starting point is 00:36:30 like legs and cancelled queries on replica. Yes. Very obvious to use. And it's better to discover what's happening, understand and then make decision based on your
Starting point is 00:36:40 situation and experience rather than you don't notice or blowed. Bloat on the primary building up slowly. But the same time, the same problem exists on the primary itself
Starting point is 00:36:54 absolutely the same, right? And I think you can make the opposite argument. Bloat is quieter. Like, the reason you don't notice bloat is because maybe it's not as bad. Yeah, we should control bloat anyway. We should control bloat anyway, right? And on the primary it happens exactly in the same mechanism. So why should we make different approach,
Starting point is 00:37:16 apply different approach to replica? And blood control is the whole topic anyway, right? If you want to improve that experience, that should be done on the primary first. And if we care about that from a post-quest tuning perspective, we should probably be changing the defaults of various auto-vacuum settings first.
Starting point is 00:37:33 You know what? I like how I'm pulling you into discussion and hackers. You should think and write your opinion next time. Yeah, I'll try and get braver. Yeah, it's a rabbit hole, I know. Yeah. Anyway, this is great
Starting point is 00:37:50 and helpful. And I think I agree with you that it feels like the vast majority of OLTP replicas where you're just offloading a huge number of really quick queries. Hot Samba PhiBic on makes so much more sense to me than off. But I can't agree that all read replicas should have it off. Like I do think there are these people using it. Probably still it's a good architectural choice as these analytics replicas. And I think you're right that it makes sense to keep those off.
Starting point is 00:38:21 and it's nice that you can do it on a per replica basis. It doesn't have to be the same for all of your replicas, which is cool. Yeah. Good. Awesome. Anything else? No, just don't let your replica to like too often. Nice one, Nick. Thanks so much. See you next time.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.