diff --git a/docs/cdc.rst b/docs/cdc.rst new file mode 100644 index 000000000..cad1b06d9 --- /dev/null +++ b/docs/cdc.rst @@ -0,0 +1,299 @@ +.. _cdc_continuity: + +Change Data Capture (CDC) Continuity +===================================== + +pg_auto_failover manages **physical** replication slots for its own +primary/standby topology: that part of the design has always been there, +and is covered in :ref:`fault_tolerance`. This page is about **logical** +replication slots, the kind used by Change Data Capture (CDC) tools such as +`Debezium `_, ``pg_logical``, or a hand-rolled +consumer built on ``pgoutput`` / ``wal2json``. + +Historically, a logical replication slot was tied to the single Postgres +instance that created it: if that instance was a pg_auto_failover primary +and a failover promoted one of its standbys, the slot simply didn't exist +on the new primary, and the CDC consumer had to be recreated from scratch +(losing its position, and usually requiring a full resync of the +downstream system). This was discussed at length in `issue #400`__ and +`issue #1030`__, back when Postgres itself had no mechanism to keep a +logical slot in sync on a standby. + +__ https://github.com/hapostgres/pg_auto_failover/issues/400 +__ https://github.com/hapostgres/pg_auto_failover/issues/1030 + +Postgres 17 changed this by adding **failover-enabled logical replication +slots**: a slot created with ``failover => true`` gets its position +mirrored to physical standbys by a dedicated background worker (the +*slot-sync worker*), gated by the ``synchronized_standby_slots`` setting on +the primary. When a standby holding such a synced slot gets promoted, +Postgres lets it be used for logical decoding immediately, with its +existing position intact -- no resync, no special handling required from +whoever is managing the promotion. (The slot's ``synced`` column stays +``true`` as a permanent record of its provenance; it is not reset on +promotion, and isn't the thing that governs whether the slot is usable.) + +pg_auto_failover takes advantage of this, automatically, on every node +running Postgres 17 or more recent: + + - **`dbname` in `primary_conninfo`**: the slot-sync worker refuses to + start at all unless the standby's ``primary_conninfo`` carries a + ``dbname`` keyword. pg_auto_failover has included it since the PG17/18/19 + compatibility work landed. + + - **`sync_replication_slots` and `hot_standby_feedback`**: both are part + of the default settings pg_auto_failover writes to every 17+ node's + configuration, so the slot-sync worker is running on every standby + without any manual setup. (``hot_standby_feedback`` was already a + pg_auto_failover default before 17, for unrelated reasons; 17 only adds + ``sync_replication_slots``.) + + - **`synchronized_standby_slots` on the primary**, computed by the monitor + and kept in sync automatically as the formation's topology changes. This + is the setting that makes the *continuity* guarantee hold: it lists the + physical replication slot names of every current failover candidate + (every node with a non-zero candidate priority), so a failover-enabled + logical slot is only allowed to advance once every node that might get + promoted has already replayed that far. Whichever standby ends up + promoted, it is guaranteed to already have everything the logical slot + had confirmed. You can see the computed value at any time with:: + + pg_autoctl show settings + + which includes a ``synchronized_standby_slots`` row next to the + existing ``synchronous_standby_names`` one. + + - **Cleanup of stale slots after an interrupted demotion**: a node that + was briefly primary before being demoted could otherwise be left + holding on to its old, no-longer-synced failover slots, which blocks the + slot-sync worker from ever starting on it (``same name slot ... already + exists on the standby``). pg_auto_failover now drops those leftover + slots as part of the same cleanup step that already resets physical slot + state after a demotion. + +What you still need to do yourself +----------------------------------- + +pg_auto_failover does not create, drop, or otherwise manage logical +replication slots: that remains entirely your CDC tool's responsibility, +exactly as it would be on a Postgres cluster not managed by +pg_auto_failover. Concretely: + + 1. ``wal_level`` is ``logical`` by default on every node pg_auto_failover + creates (see :ref:`wal_level_default` below) -- there is nothing to do + here for a new cluster. If you're turning CDC on for an + *already-running* cluster that was created with the default, or with + ``--wal-level replica``, see :ref:`wal_level_rollout` for how to change + it safely: ``wal_level`` is ``PGC_POSTMASTER`` (Postgres won't let you + change it without a full server restart), and **every** node needs it, + not just the current primary -- any node could be promoted later, and a + node's own ``wal_level`` governs whether it's allowed to decode + logically at all, independently of what the primary that originally + wrote the WAL had configured. + + 2. Create your logical replication slot(s) with ``failover => true``, for + example:: + + SELECT pg_create_logical_replication_slot( + 'debezium', 'pgoutput', false, false, true); + + or, if you're using ``CREATE SUBSCRIPTION`` on a logical subscriber:: + + CREATE SUBSCRIPTION ... + WITH (failover = true, create_slot = true, slot_name = '...'); + + 3. Point your CDC consumer at the formation the way you would point any + client application: through the primary. If you use pg_auto_failover's + own application-facing connection string machinery, a failover already + redirects regular clients transparently; the same connection logic + applies to your CDC consumer's connection to the (new) primary. + +.. _wal_level_default: + +Defaults and opting out +------------------------ + +Every node pg_auto_failover creates (``pg_autoctl create postgres``, +``create monitor``, ``create coordinator``, ``create worker``) sets +``wal_level = logical`` by default -- the cost (extra WAL volume, catalog +``xmin`` retention pressure) is paid whether or not you ever create a +slot, so if you know you'll never need CDC on a given cluster, you can +opt back out at creation time:: + + pg_autoctl create postgres --wal-level replica ... + +The flag only takes effect at node creation, before Postgres's first +start: ``wal_level`` can't be changed by editing the config and reloading +on a running node (see :ref:`wal_level_rollout` for that case). There is +no supported way to go the other direction (``replica`` back up to +``logical``) except through that same rollout procedure, applied to a +cluster that opted out at creation time. + +.. _wal_level_rollout: + +Turning on ``wal_level = logical`` on an already-running cluster +------------------------------------------------------------------- + +Because ``wal_level`` is ``PGC_POSTMASTER``, there's no way around a full +Postgres restart on every node -- but pg_auto_failover already has the +primitives to do that fleet-wide with zero *unplanned* downtime, using +exactly the same maintenance-mode mechanism documented in +:ref:`upgrading_from_previous_versions`, which solves the identical +problem -- "a change needs a restart on every node, without triggering a +spurious failover" -- for a different cause (a pg_auto_failover version +upgrade rather than a GUC change). The one planned switchover below costs +the same single, short write-unavailability window as any manual +switchover with this tool, nothing more. + +For a formation with one primary and one or more standbys, using +:ref:`pg_autoctl_restart_postgres` (a single command that enters +maintenance, restarts Postgres, and leaves maintenance again, since +nothing needs to happen while Postgres is down for a plain configuration +change -- see :ref:`restart_postgres` for more worked examples of this +command, including one that does need something to run while Postgres is +down): + + 1. For each **standby**, one at a time: + + a. Apply the change, either directly in ``postgresql.conf`` or via + ``ALTER SYSTEM SET wal_level = 'logical';`` (both are safe: + pg_auto_failover's own managed settings live in a separate included + file, so this never collides with anything it writes). + b. Restart Postgres through maintenance mode: this enables maintenance + (safe: the monitor switches the primary to asynchronous replication + and the standby is no longer a promotion candidate while in this + state), restarts Postgres so it comes back up with the new + ``wal_level``, then disables maintenance again so the standby + rejoins:: + + pg_autoctl restart postgres --maintenance --pgdata + + 2. Once **every standby** has been upgraded, apply the same configuration + change to the primary, then restart it with ``--allow-failover`` added:: + + pg_autoctl restart postgres --maintenance --allow-failover --pgdata + + ``--allow-failover`` is required here: entering maintenance on a + primary means promoting a standby first, the same promotion path as a + normal switchover. This is the one and only planned + write-unavailability window in the whole procedure. Once the restart + completes, the old primary rejoins as a regular standby, already + running with the new ``wal_level``. + +At the end, every node has ``wal_level = logical`` and the formation is +back to its original primary/standby layout (or not, if you'd rather +leave the promoted standby as the new primary -- nothing above requires +failing back). + +Limitations +----------- + + - This only applies to Postgres 17 and more recent, on every node of the + formation. Mixed-version formations (e.g. during a rolling major-version + upgrade) get no continuity guarantee until every node involved in a + potential promotion is on 17+. + + - ``synchronized_standby_slots`` only ever lists nodes with a non-zero + candidate priority. A node with ``candidate_priority = 0`` can still be + promoted manually (a forced failover), and in that case the usual + Postgres trade-off applies: a logical slot's consumer may need to + tolerate replaying already-seen changes, or, in the worst case, be + recreated. + + - Citus formations are not a primary focus of this feature: Citus already + sets ``wal_level = logical`` for its own metadata synchronization needs, + but CDC continuity for Citus worker/coordinator logical slots has not + been specifically tested. See :ref:`cdc_continuity_citus` below for why + Citus needs more than this feature to give CDC consumers a coherent + single stream, and is intentionally out of scope here. + + - **No "N of M" quorum knob.** Among the candidates it lists, + ``synchronized_standby_slots`` requires *all* of them to confirm before + a failover-enabled logical slot advances -- Postgres provides no + quorum knob here, unlike ``synchronous_standby_names``'s ``ANY N (...)`` + syntax. This does not mean a lagging or unreachable candidate + permanently stalls decoding, though: pg_auto_failover only lists a + candidate while it's currently a converged secondary, so one that + falls behind or stops reporting in (pushed back to ``catchingup`` by + the monitor) drops out of the list on its own, and the slot keeps + advancing against whichever candidates are still caught up -- it + reappears automatically once that node catches back up and reports + secondary again. What's still missing is an explicit operator-facing + knob to require only some of several simultaneously-healthy candidates + to confirm; to exclude a node from the list permanently, regardless of + its replication state, lower its candidate priority to ``0`` (at the + cost of no longer being an automatic failover candidate). + + - **A node flip-flopping back to primary can lose its own + originally-created slot.** If the *same* node that originally created a + ``failover => true`` slot is demoted, then later promoted back to + primary a second time, Postgres's slot-sync worker can get stuck + indefinitely with ``could not synchronize replication slot "...": + Synchronization could lead to data loss`` (comparing the node's own, + already-more-advanced local WAL/catalog position against what the + *other* node's copy of the slot had reached) -- a temporary placeholder + slot ends up active rather than a real persistent one, and does not + survive the second promotion. This is a Postgres-level position + reconciliation limitation specific to round-tripping back to the exact + original creator, not something pg_auto_failover's demotion cleanup + causes or can resolve on its own; it was found while testing this + feature (see ``tests/tap/specs/cdc_continuity.pgaf``) and does not + affect the common case of failing over to a *different* node. If you + hit it, drop and recreate the slot after the second promotion. + +.. _cdc_continuity_citus: + +Citus: a single merged CDC stream is future work +------------------------------------------------- + +Everything above gives CDC continuity **per Postgres node** (coordinator or +worker). It does not solve the problem of giving a CDC consumer a single, +correctly-ordered stream for a *distributed* table whose rows are sharded +across several Citus worker groups, each independently managed by +pg_auto_failover and each with its own ``wal_level=logical``/failover-slot +setup. + +Today, decoding a sharded table means running a separate consumer per +worker and reconciling downstream -- Citus's own guidance for Debezium-style +CDC is exactly that: one connector per worker node, leaving ordering and +cross-shard transaction reassembly to the consumer. A proper merged stream +would need, roughly: + + - **Topology awareness**: tracking ``pg_dist_shard`` / ``pg_dist_placement`` + (or the ``citus_shards`` view) to know which worker streams currently + back a given distributed table, including across shard rebalancing. + + - **Transaction reassembly**: a cross-shard write is split into one local + transaction per worker it touches, usually committed via two-phase + commit under the hood. The per-worker fragments need to be correlated + back into a single logical transaction (grouping by the 2PC global + transaction id used in ``PREPARE TRANSACTION`` / ``COMMIT PREPARED``) + before being handed to the consumer. + + - **Cross-worker ordering**: once transactions are reassembled, they still + need a deterministic global order and a way to tell a consumer "nothing + older than this point remains outstanding on any worker" -- the + watermark / resolved-timestamp pattern used by other distributed + change-feed systems (buffer each worker's stream, advance a merged + watermark only once every worker has confirmed past it, then emit in + order up to that watermark). + + - **Renumbering**: assigning a single monotonic global sequence/LSN to the + reassembled, reordered stream so a downstream consumer gets stable, + dedupable offsets instead of per-worker LSNs that reset independently + on each worker's own failover. + + - **Failover resilience of the merge layer itself**: each worker's own + slot already survives its own failover (this feature); the merge + component additionally needs its own durable checkpoint -- one position + per worker stream -- so it can resume correctly after its own restart, + independently of any single worker's HA event. + +This is a distinct, substantial piece of work -- effectively a standalone +Citus-aware CDC proxy/gateway sitting in front of per-worker logical +streams, not an extension of pg_auto_failover's monitor/keeper -- and is +intentionally left for a separate effort rather than bundled into this one. + +See also :ref:`configuration` for the full list of settings pg_auto_failover +manages, and :ref:`fault_tolerance` for how physical replication slots are +used internally. diff --git a/docs/faq.rst b/docs/faq.rst index 4e1dcd3d5..32c515279 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -279,3 +279,40 @@ following commands:: $ pg_autoctl disable monitor $ pg_autoctl enable monitor + +Can I use logical replication / CDC tools (Debezium, etc) with pg_auto_failover? +--------------------------------------------------------------------------------- + +Yes, as long as every node in the formation runs **Postgres 17 or more +recent**. See :ref:`cdc_continuity` for the full picture; the short version: + + - pg_auto_failover doesn't create or manage your logical replication + slots itself -- that stays your CDC tool's job, same as with any other + Postgres cluster. + + - What pg_auto_failover does manage, automatically, on Postgres 17+: the + ``synchronized_standby_slots`` setting on the primary (kept in sync with + the current failover candidates), the ``dbname`` Postgres needs in + ``primary_conninfo`` to run its slot-sync worker, and ``sync_replication_slots`` + / ``hot_standby_feedback`` on every standby. Together, these are what let + a logical slot created with ``failover => true`` keep working after a + promotion, instead of being silently left behind on the old primary. + + - ``wal_level = logical`` is the default on every node pg_auto_failover + creates (see :ref:`wal_level_default`) -- you only need to set it + yourself when opting back in on a cluster created with + ``--wal-level replica``, or one created before this default existed + (see :ref:`wal_level_rollout`). You still need to create your slot(s) + with ``failover => true`` yourself: pg_auto_failover manages the + infrastructure that keeps them working across a promotion, not slot + creation itself. + +On Postgres versions before 17, Postgres itself has no mechanism to keep a +logical slot's position correct on a standby without losing unconsumed +changes, so there is nothing pg_auto_failover can safely do there: a +logical slot on the primary does not survive a failover, the same as +without pg_auto_failover. This was the situation discussed in `issue #400`__ +and `issue #1030`__ before Postgres 17 added slot synchronization. + +__ https://github.com/hapostgres/pg_auto_failover/issues/400 +__ https://github.com/hapostgres/pg_auto_failover/issues/1030 diff --git a/docs/fault-tolerance.rst b/docs/fault-tolerance.rst index ac9da4f87..76d168a6b 100644 --- a/docs/fault-tolerance.rst +++ b/docs/fault-tolerance.rst @@ -1,3 +1,5 @@ +.. _fault_tolerance: + Failover and Fault Tolerance ============================ diff --git a/docs/index.rst b/docs/index.rst index 5f7f8440e..edb2eb624 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -55,6 +55,7 @@ __ https://github.com/hapostgres/pg_auto_failover architecture-multi-standby failover-state-machine fault-tolerance + cdc security .. toctree:: diff --git a/docs/operations.rst b/docs/operations.rst index bf79f60ab..9251c467f 100644 --- a/docs/operations.rst +++ b/docs/operations.rst @@ -1,3 +1,5 @@ +.. _operations: + Operating pg_auto_failover ========================== @@ -194,6 +196,8 @@ monitor. And when the upgrade is done we can use ``pg_autoctl show state`` on the monitor to see that eveything is as expected. +.. _upgrading_from_previous_versions: + Upgrading from previous pg_auto_failover versions ------------------------------------------------- diff --git a/docs/ref/configuration.rst b/docs/ref/configuration.rst index cf554823b..e20c639b1 100644 --- a/docs/ref/configuration.rst +++ b/docs/ref/configuration.rst @@ -229,6 +229,13 @@ Name of the PostgreSQL replication slot used in the streaming replication setup automatically deployed by pg_auto_failover. Replication slots can't be renamed in PostgreSQL. +On Postgres 17 and more recent, pg_auto_failover also maintains the +``synchronized_standby_slots`` Postgres setting on the primary, computed +from the formation's current failover candidates: this is not a +pg_auto_failover configuration key of its own (there is nothing to set), +but it is what lets a failover-enabled logical replication slot survive a +promotion. See :ref:`cdc_continuity` for the full picture. + **replication.maximum_backup_rate** When pg_auto_failover (re-)builds a standby node using the ``pg_basebackup`` diff --git a/docs/ref/pg_autoctl.rst b/docs/ref/pg_autoctl.rst index 86b143f48..a814dbb35 100644 --- a/docs/ref/pg_autoctl.rst +++ b/docs/ref/pg_autoctl.rst @@ -52,6 +52,7 @@ pg_autoctl provides the following commands:: state Prints monitor's state of nodes in a given formation and group settings Print replication settings for a formation from the monitor standby-names Prints synchronous_standby_names for a given group + synchronized-standby-slots Prints synchronized_standby_slots for a given group (Postgres 17+ CDC continuity) timeline Show the timeline history known to the monitor for a group, and each node's position against it file List pg_autoctl internal files (config, state, pid) systemd Print systemd service file for this node diff --git a/docs/ref/pg_autoctl_show.rst b/docs/ref/pg_autoctl_show.rst index 896b94342..215f6eb37 100644 --- a/docs/ref/pg_autoctl_show.rst +++ b/docs/ref/pg_autoctl_show.rst @@ -13,6 +13,7 @@ pg_autoctl show - Show pg_auto_failover information pg_autoctl_show_state pg_autoctl_show_settings pg_autoctl_show_standby_names + pg_autoctl_show_synchronized_standby_slots pg_autoctl_show_timeline pg_autoctl_show_file pg_autoctl_show_systemd diff --git a/docs/ref/pg_autoctl_show_synchronized_standby_slots.rst b/docs/ref/pg_autoctl_show_synchronized_standby_slots.rst new file mode 100644 index 000000000..55e3516a5 --- /dev/null +++ b/docs/ref/pg_autoctl_show_synchronized_standby_slots.rst @@ -0,0 +1,100 @@ +.. _pg_autoctl_show_synchronized_standby_slots: + +pg_autoctl show synchronized-standby-slots +=========================================== + +pg_autoctl show synchronized-standby-slots - Prints synchronized_standby_slots for a given group (Postgres 17+ CDC continuity) + +Synopsis +-------- + +This command prints the current value for the Postgres 17+ +``synchronized_standby_slots`` setting for the primary Postgres server of +the target group (default ``0``) in the target formation (default +``default``), as computed by the monitor: the comma-separated physical +replication slot names of the group's current failover candidates. See +:ref:`cdc_continuity` for the full picture of what this setting is for:: + + usage: pg_autoctl show synchronized-standby-slots [ --pgdata ] --formation --group + + --pgdata path to data directory + --monitor pg_auto_failover Monitor Postgres URL + --formation formation to query, defaults to 'default' + --group group to query formation, defaults to all + --json output data in the JSON format + +Options +------- + +--pgdata + + Location of the Postgres node being managed locally. Defaults to the + environment variable ``PGDATA``. Use ``--monitor`` to connect to a monitor + from anywhere, rather than the monitor URI used by a local Postgres node + managed with ``pg_autoctl``. + +--monitor + + Postgres URI used to connect to the monitor. Must use the ``autoctl_node`` + username and target the ``pg_auto_failover`` database name. It is possible + to show the Postgres URI from the monitor node using the command + :ref:`pg_autoctl_show_uri`. + + Defaults to the value of the environment variable ``PG_AUTOCTL_MONITOR``. + +--formation + + Show the current ``synchronized_standby_slots`` value for the given + formation. Defaults to the ``default`` formation. + +--group + + Show the current ``synchronized_standby_slots`` value for the given group + in the given formation. Defaults to group ``0``. + +--json + + Output a JSON formatted data instead of a table formatted list. + +Environment +----------- + +PGDATA + + Postgres directory location. Can be used instead of the ``--pgdata`` + option. + +PG_AUTOCTL_MONITOR + + Postgres URI to connect to the monitor node, can be used instead of the + ``--monitor`` option. + +XDG_CONFIG_HOME + + The pg_autoctl command stores its configuration files in the standard + place XDG_CONFIG_HOME. See the `XDG Base Directory Specification`__. + + __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html + +XDG_DATA_HOME + + The pg_autoctl command stores its internal states files in the standard + place XDG_DATA_HOME, which defaults to ``~/.local/share``. See the `XDG + Base Directory Specification`__. + + __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html + +Examples +-------- + +:: + + $ pg_autoctl show synchronized-standby-slots + 'pgautofailover_standby_2,pgautofailover_standby_3' + + $ pg_autoctl show synchronized-standby-slots --json + { + "formation": "default", + "group": 0, + "synchronized_standby_slots": "pgautofailover_standby_2,pgautofailover_standby_3" + } diff --git a/docs/ref/pg_autoctl_watch.rst b/docs/ref/pg_autoctl_watch.rst index 2a18b940b..816d18afd 100644 --- a/docs/ref/pg_autoctl_watch.rst +++ b/docs/ref/pg_autoctl_watch.rst @@ -84,6 +84,16 @@ currently selected formation, the formation replication setting :ref:`number_sync_standbys`, and then in the right most position the current time. +Whenever the group has at least one failover candidate, the header also +shows a "CDC Slots" count: how many physical slot names the +monitor-computed ``synchronized_standby_slots`` setting currently lists for +that group (see :ref:`cdc_continuity`). This is the same list regardless of +whether you actually have a failover-enabled logical replication slot in +use -- it reflects the topology, not live slot activity. Watching this +number drop during a failover is a quick way to notice a candidate dropping +out of the list; see :ref:`pg_autoctl_show_synchronized_standby_slots` for +the full slot name list. + The second section displays one line per node, and each line contains a list of columns that describe the current state for the node. This list can includes the following columns, and which columns are part of the output diff --git a/src/bin/common/pgsetup.c b/src/bin/common/pgsetup.c index 175f02072..ede8c4377 100644 --- a/src/bin/common/pgsetup.c +++ b/src/bin/common/pgsetup.c @@ -1996,6 +1996,31 @@ pg_setup_standby_slot_supported(PostgresSetup *pgSetup, int logLevel) } +/* + * pg_setup_synchronized_standby_slots_supported returns true when the + * target Postgres instance represented in pgSetup is Postgres 17 or more + * recent, the first version that implements the "synchronized_standby_slots" + * GUC and the pg_sync_replication_slots() slot-sync worker that keeps + * failover-enabled logical replication slots usable across a physical + * standby promotion. Before 17, Postgres has no mechanism to keep a logical + * slot's position correct on a standby without losing unconsumed changes, + * so pg_auto_failover's CDC-continuity support only applies to 17+. + */ +bool +pg_setup_synchronized_standby_slots_supported(PostgresSetup *pgSetup) +{ + int pg_version = 0; + + if (!parse_pg_version_string(pgSetup->pg_version, &pg_version)) + { + /* errors have already been logged */ + return false; + } + + return pg_version >= 1700; +} + + /* * pgsetup_parse_hba_level parses a string that represents an HBAEditLevel * value. diff --git a/src/bin/common/pgsetup.h b/src/bin/common/pgsetup.h index f8afcfa18..ae1db488c 100644 --- a/src/bin/common/pgsetup.h +++ b/src/bin/common/pgsetup.h @@ -233,6 +233,7 @@ typedef struct pg_setup NodeReplicationSettings settings; /* node replication settings */ SSLOptions ssl; /* ssl options */ char citusClusterName[NAMEDATALEN]; /* citus.cluster_name */ + char walLevel[NAMEDATALEN]; /* wal_level, defaults to "logical" */ } PostgresSetup; #define IS_EMPTY_STRING_BUFFER(strbuf) (strbuf[0] == '\0') @@ -277,6 +278,7 @@ SSLMode pgsetup_parse_sslmode(const char *sslMode); char * pgsetup_sslmode_to_string(SSLMode sslMode); bool pg_setup_standby_slot_supported(PostgresSetup *pgSetup, int logLevel); +bool pg_setup_synchronized_standby_slots_supported(PostgresSetup *pgSetup); HBAEditLevel pgsetup_parse_hba_level(const char *level); char * pgsetup_hba_level_to_string(HBAEditLevel hbaLevel); diff --git a/src/bin/common/pgsql.c b/src/bin/common/pgsql.c index 3807f41d2..f28d06dc4 100644 --- a/src/bin/common/pgsql.c +++ b/src/bin/common/pgsql.c @@ -1489,6 +1489,37 @@ pgsql_set_synchronous_standby_names(PGSQL *pgsql, } +/* + * pgsql_set_synchronized_standby_slots sets the Postgres 17+ + * synchronized_standby_slots GUC on the local Postgres to the value + * computed on the pg_auto_failover monitor: the comma-separated list of + * physical replication slot names of the current failover candidates. This + * is what makes a failover-enabled logical replication slot wait for every + * promotable standby before its confirmed position advances, so the slot + * stays usable by a CDC consumer across a promotion. + */ +bool +pgsql_set_synchronized_standby_slots(PGSQL *pgsql, + char *synchronized_standby_slots) +{ + char quoted[BUFSIZE] = { 0 }; + GUC setting = { "synchronized_standby_slots", quoted }; + + if (sformat(quoted, BUFSIZE, "'%s'", synchronized_standby_slots) >= BUFSIZE) + { + log_error("Failed to apply the synchronized_standby_slots value \"%s\": " + "pg_autoctl supports values up to %d bytes and this one " + "requires %lu bytes", + synchronized_standby_slots, + BUFSIZE, + (unsigned long) strlen(synchronized_standby_slots)); + return false; + } + + return pgsql_alter_system_set(pgsql, setting); +} + + /* * pgsql_replication_slot_maintain advances the current confirmed position of * the given replication slot up to the given LSN position, create the @@ -1590,6 +1621,37 @@ pgsql_drop_replication_slot(PGSQL *pgsql, const char *slotName) } +/* + * pgsql_drop_stale_failover_slots drops every logical replication slot that + * still has failover enabled but isn't a synced copy (failover AND NOT + * synced): a slot in exactly that state on a node that is becoming a + * standby again is necessarily a stale leftover from when this node was + * primary, not something the slot-sync worker created. Left in place, it + * blocks that worker from starting at all ("same name slot ... already + * exists on the standby"), so the standby never gets a working synced copy + * (see issue #1188: this happens whenever demotion is interrupted, e.g. by + * demote_timeout, so the normal demotion cleanup below doesn't get a chance + * to run first). + * + * The failover/synced columns only exist from Postgres 17 onward (the + * version that introduces failover-enabled logical slots and the slot-sync + * worker in the first place), so this must only ever be called when + * pg_setup_synchronized_standby_slots_supported() is true. + */ +bool +pgsql_drop_stale_failover_slots(PGSQL *pgsql) +{ + char *sql = + "SELECT pg_drop_replication_slot(slot_name) " + " FROM pg_replication_slots " + " WHERE failover AND NOT synced AND NOT active"; + + log_trace("pgsql_drop_stale_failover_slots"); + + return pgsql_execute_with_params(pgsql, sql, 0, NULL, NULL, NULL, NULL); +} + + /* * BuildNodesArrayValues build the SQL expression to use in a FROM clause to * represent the list of other standby nodes from the given nodeArray. diff --git a/src/bin/common/pgsql.h b/src/bin/common/pgsql.h index f27c6f6e0..b7b004dec 100644 --- a/src/bin/common/pgsql.h +++ b/src/bin/common/pgsql.h @@ -375,9 +375,12 @@ bool pgsql_replication_slot_exists(PGSQL *pgsql, const char *slotName, bool *slotExists); bool pgsql_create_replication_slot(PGSQL *pgsql, const char *slotName); bool pgsql_drop_replication_slot(PGSQL *pgsql, const char *slotName); +bool pgsql_drop_stale_failover_slots(PGSQL *pgsql); bool postgres_sprintf_replicationSlotName(int64_t nodeId, char *slotName, int size); bool pgsql_set_synchronous_standby_names(PGSQL *pgsql, char *synchronous_standby_names); +bool pgsql_set_synchronized_standby_slots(PGSQL *pgsql, + char *synchronized_standby_slots); bool pgsql_replication_slot_create_and_drop(PGSQL *pgsql, NodeAddressArray *nodeArray); bool pgsql_replication_slot_maintain(PGSQL *pgsql, NodeAddressArray *nodeArray); diff --git a/src/bin/pg_autoctl/cli_common.c b/src/bin/pg_autoctl/cli_common.c index 709504b7c..80f1634f3 100644 --- a/src/bin/pg_autoctl/cli_common.c +++ b/src/bin/pg_autoctl/cli_common.c @@ -241,6 +241,22 @@ cli_common_keeper_getopts(int argc, char **argv, break; } + case 'K': + { + /* { "wal-level", required_argument, NULL, 'K' }, */ + if (strcmp(optarg, "logical") != 0 && strcmp(optarg, "replica") != 0) + { + errors++; + log_error("Invalid value for --wal-level: \"%s\", " + "expected \"logical\" (the default) or \"replica\"", + optarg); + } + + strlcpy(LocalOptionConfig.pgSetup.walLevel, optarg, NAMEDATALEN); + log_trace("--wal-level %s", LocalOptionConfig.pgSetup.walLevel); + break; + } + case 'S': { /* { "skip-pg-hba", required_argument, NULL, 'S' }, */ diff --git a/src/bin/pg_autoctl/cli_common.h b/src/bin/pg_autoctl/cli_common.h index 78b7b051c..5892ec575 100644 --- a/src/bin/pg_autoctl/cli_common.h +++ b/src/bin/pg_autoctl/cli_common.h @@ -141,6 +141,7 @@ extern CommandLine show_state_command; extern CommandLine show_settings_command; extern CommandLine show_file_command; extern CommandLine show_standby_names_command; +extern CommandLine show_synchronized_standby_slots_command; extern CommandLine show_timeline_command; /* cli_watch.c */ diff --git a/src/bin/pg_autoctl/cli_create_node.c b/src/bin/pg_autoctl/cli_create_node.c index 39b3f4aab..c2a8de303 100644 --- a/src/bin/pg_autoctl/cli_create_node.c +++ b/src/bin/pg_autoctl/cli_create_node.c @@ -102,7 +102,8 @@ CommandLine create_postgres_command = KEEPER_CLI_SSL_OPTIONS " --candidate-priority priority of the node to be promoted to become primary\n" " --replication-quorum true if node participates in write quorum\n" - " --maximum-backup-rate maximum transfer rate of data transferred from the server during initial sync\n", + " --maximum-backup-rate maximum transfer rate of data transferred from the server during initial sync\n" + " --wal-level 'logical' (the default, needed for CDC) or 'replica'\n", cli_create_postgres_getopts, cli_create_postgres); @@ -344,6 +345,7 @@ cli_create_postgres_getopts(int argc, char **argv) { "replication-quorum", required_argument, NULL, 'r' }, { "maximum-backup-rate", required_argument, NULL, 'R' }, { "region", required_argument, NULL, 'G' }, + { "wal-level", required_argument, NULL, 'K' }, { "run", no_argument, NULL, 'x' }, { "no-ssl", no_argument, NULL, 'N' }, { "ssl-self-signed", no_argument, NULL, 's' }, @@ -357,7 +359,7 @@ cli_create_postgres_getopts(int argc, char **argv) int optind = cli_create_node_getopts(argc, argv, long_options, - "C:D:H:p:l:U:A:SLd:a:n:f:m:MI:W:w:RGVvqhP:r:xsN", + "C:D:H:p:l:U:A:SLd:a:n:f:m:MI:W:w:RGK:VvqhP:r:xsN", &options); /* publish our option parsing in the global variable */ diff --git a/src/bin/pg_autoctl/cli_root.c b/src/bin/pg_autoctl/cli_root.c index d1c16a587..1d4ba0731 100644 --- a/src/bin/pg_autoctl/cli_root.c +++ b/src/bin/pg_autoctl/cli_root.c @@ -46,6 +46,7 @@ CommandLine *show_subcommands_with_debug[] = { &show_state_command, &show_settings_command, &show_standby_names_command, + &show_synchronized_standby_slots_command, &show_timeline_command, &show_file_command, &systemd_cat_service_file_command, @@ -63,6 +64,7 @@ CommandLine *show_subcommands[] = { &show_state_command, &show_settings_command, &show_standby_names_command, + &show_synchronized_standby_slots_command, &show_timeline_command, &show_file_command, &systemd_cat_service_file_command, diff --git a/src/bin/pg_autoctl/cli_show.c b/src/bin/pg_autoctl/cli_show.c index 0cf0c4b33..9d8645b1e 100644 --- a/src/bin/pg_autoctl/cli_show.c +++ b/src/bin/pg_autoctl/cli_show.c @@ -46,6 +46,8 @@ static void cli_show_events(int argc, char **argv); static int cli_show_standby_names_getopts(int argc, char **argv); static void cli_show_standby_names(int argc, char **argv); +static void cli_show_synchronized_standby_slots(int argc, char **argv); + static int cli_show_timeline_getopts(int argc, char **argv); static void cli_show_timeline(int argc, char **argv); @@ -128,6 +130,19 @@ CommandLine show_standby_names_command = cli_show_standby_names_getopts, cli_show_standby_names); +CommandLine show_synchronized_standby_slots_command = + make_command("synchronized-standby-slots", + "Prints synchronized_standby_slots for a given group " + "(Postgres 17+ CDC continuity)", + " [ --pgdata ] --formation --group", + " --pgdata path to data directory \n" + " --monitor show the monitor uri\n" + " --formation formation to query, defaults to 'default'\n" + " --group group to query formation, defaults to all\n" + " --json output data in the JSON format\n", + cli_show_standby_names_getopts, + cli_show_synchronized_standby_slots); + CommandLine show_timeline_command = make_command("timeline", "Show the timeline history known to the monitor for a group, " @@ -876,29 +891,39 @@ cli_show_standby_names_getopts(int argc, char **argv) /* - * cli_show_standby_names prints the synchronous_standby_names setting value - * for a given group (in a known formation). + * cli_show_monitor_computed_setting fetches a group-wide, monitor-computed + * setting value (synchronous_standby_names, synchronized_standby_slots, + * ...) via fetchFunction and prints it, in JSON or plain text depending on + * outputJSON. settingName is both the JSON key and what's named in the + * error message, so callers reusing this pass the same string Postgres + * itself uses for the GUC. + * + * Shared by cli_show_standby_names() and cli_show_synchronized_standby_slots() + * below, which otherwise only differ in which monitor function to call and + * which setting name to print under. */ static void -cli_show_standby_names(int argc, char **argv) +cli_show_monitor_computed_setting(KeeperConfig *config, + bool (*fetchFunction)(Monitor *monitor, + char *formation, + int groupId, + char *buffer, + int size), + const char *settingName) { - KeeperConfig config = keeperOptions; Monitor monitor = { 0 }; - char synchronous_standby_names[BUFSIZE] = { 0 }; + char settingValue[BUFSIZE] = { 0 }; - (void) cli_monitor_init_from_option_or_config(&monitor, &config); + (void) cli_monitor_init_from_option_or_config(&monitor, config); - (void) cli_set_groupId(&monitor, &config); + (void) cli_set_groupId(&monitor, config); - if (!monitor_synchronous_standby_names( - &monitor, - config.formation, - config.groupId, - synchronous_standby_names, - BUFSIZE)) + if (!(*fetchFunction)(&monitor, config->formation, config->groupId, + settingValue, BUFSIZE)) { - log_fatal("Failed to get the synchronous_standby_names setting value " - " from the monitor, see above for details"); + log_fatal("Failed to get the %s setting value " + " from the monitor, see above for details", + settingName); exit(EXIT_CODE_MONITOR); } @@ -907,22 +932,53 @@ cli_show_standby_names(int argc, char **argv) JSON_Value *js = json_value_init_object(); JSON_Object *jsObj = json_value_get_object(js); - json_object_set_string(jsObj, "formation", config.formation); - json_object_set_number(jsObj, "group", (double) config.groupId); - json_object_set_string(jsObj, - "synchronous_standby_names", - synchronous_standby_names); + json_object_set_string(jsObj, "formation", config->formation); + json_object_set_number(jsObj, "group", (double) config->groupId); + json_object_set_string(jsObj, settingName, settingValue); (void) cli_pprint_json(js); } else { - /* current synchronous_standby_names might be an empty string */ - (void) fformat(stdout, "'%s'\n", synchronous_standby_names); + /* current setting value might be an empty string */ + (void) fformat(stdout, "'%s'\n", settingValue); } } +/* + * cli_show_standby_names prints the synchronous_standby_names setting value + * for a given group (in a known formation). + */ +static void +cli_show_standby_names(int argc, char **argv) +{ + KeeperConfig config = keeperOptions; + + (void) cli_show_monitor_computed_setting(&config, + monitor_synchronous_standby_names, + "synchronous_standby_names"); +} + + +/* + * cli_show_synchronized_standby_slots prints the Postgres 17+ + * synchronized_standby_slots setting value for a given group (in a known + * formation): the comma-separated physical slot names of the group's + * current failover candidates, which is what lets a failover-enabled + * logical replication slot survive a promotion. See docs/cdc.rst. + */ +static void +cli_show_synchronized_standby_slots(int argc, char **argv) +{ + KeeperConfig config = keeperOptions; + + (void) cli_show_monitor_computed_setting(&config, + monitor_synchronized_standby_slots, + "synchronized_standby_slots"); +} + + /* * cli_show_timeline_getopts parses the command line options for the * command `pg_autoctl show timeline`. diff --git a/src/bin/pg_autoctl/fsm_transition.c b/src/bin/pg_autoctl/fsm_transition.c index 106762a8a..96526aec9 100644 --- a/src/bin/pg_autoctl/fsm_transition.c +++ b/src/bin/pg_autoctl/fsm_transition.c @@ -632,6 +632,54 @@ fsm_enable_sync_rep(Keeper *keeper) } +/* + * fsm_fetch_monitor_setting fetches a group-wide, monitor-computed setting + * value (synchronous_standby_names, synchronized_standby_slots, ...) into + * the given buffer via monitorFetchFunction, or falls back to + * defaultValueWhenNoMonitor when the monitor is disabled for this node. + * settingName is only used to word the error message. + * + * Shared by fsm_apply_settings() below for both settings it applies, which + * otherwise only differ in which monitor function to call, which buffer to + * fill, and what to fall back to without a monitor. + */ +static bool +fsm_fetch_monitor_setting(Keeper *keeper, + bool (*monitorFetchFunction)(Monitor *monitor, + char *formation, + int groupId, + char *buffer, + int size), + char *buffer, + size_t bufferSize, + const char *defaultValueWhenNoMonitor, + const char *settingName) +{ + Monitor *monitor = &(keeper->monitor); + KeeperConfig *config = &(keeper->config); + + if (config->monitorDisabled) + { + strlcpy(buffer, defaultValueWhenNoMonitor, bufferSize); + return true; + } + + if (!(*monitorFetchFunction)(monitor, + config->formation, + keeper->state.current_group, + buffer, + bufferSize)) + { + log_error("Failed to get the %s value from the monitor, " + "see above for details", + settingName); + return false; + } + + return true; +} + + /* * fsm_apply_settings is used when a pg_auto_failover setting has changed, such * as number_sync_standbys or node priorities and replication quorum @@ -643,34 +691,46 @@ fsm_enable_sync_rep(Keeper *keeper) bool fsm_apply_settings(Keeper *keeper) { - Monitor *monitor = &(keeper->monitor); - KeeperConfig *config = &(keeper->config); LocalPostgresServer *postgres = &(keeper->postgres); - /* get synchronous_standby_names value from the monitor */ - if (!config->monitorDisabled) + /* get synchronous_standby_names value from the monitor, '*' without one */ + if (!fsm_fetch_monitor_setting(keeper, + monitor_synchronous_standby_names, + postgres->synchronousStandbyNames, + sizeof(postgres->synchronousStandbyNames), + "*", + "synchronous_standby_names")) { - if (!monitor_synchronous_standby_names( - monitor, - config->formation, - keeper->state.current_group, - postgres->synchronousStandbyNames, - sizeof(postgres->synchronousStandbyNames))) - { - log_error("Failed to enable synchronous replication because " - "we failed to get the synchronous_standby_names value " - "from the monitor, see above for details"); - return false; - } + /* errors have already been logged */ + return false; } - else + + if (!primary_set_synchronous_standby_names(postgres)) + { + /* errors have already been logged */ + return false; + } + + /* + * Also fetch and apply the synchronized_standby_slots setting (Postgres + * 17+ CDC continuity): same monitor round-trip pattern as + * synchronous_standby_names above, computed from the group's current + * failover candidates rather than its synchronous replication quorum. + * primary_set_synchronized_standby_slots() is itself a no-op before + * Postgres 17, so we always fetch the value and let it decide. + */ + if (!fsm_fetch_monitor_setting(keeper, + monitor_synchronized_standby_slots, + postgres->synchronizedStandbySlots, + sizeof(postgres->synchronizedStandbySlots), + "", + "synchronized_standby_slots")) { - /* no monitor: use the generic value '*' */ - strlcpy(postgres->synchronousStandbyNames, "*", - sizeof(postgres->synchronousStandbyNames)); + /* errors have already been logged */ + return false; } - return primary_set_synchronous_standby_names(postgres); + return primary_set_synchronized_standby_slots(postgres); } diff --git a/src/bin/pg_autoctl/keeper_config.c b/src/bin/pg_autoctl/keeper_config.c index 7621ff530..918f1390e 100644 --- a/src/bin/pg_autoctl/keeper_config.c +++ b/src/bin/pg_autoctl/keeper_config.c @@ -103,6 +103,10 @@ make_strbuf_option("postgresql", "hba_level", NULL, \ false, MAXPGPATH, config->pgSetup.hbaLevelStr) +#define OPTION_POSTGRESQL_WAL_LEVEL(config) \ + make_strbuf_option("postgresql", "wal_level", "wal-level", \ + false, NAMEDATALEN, config->pgSetup.walLevel) + #define OPTION_SSL_ACTIVE(config) \ make_int_option_default("ssl", "active", NULL, \ false, &(config->pgSetup.ssl.active), 0) @@ -237,6 +241,7 @@ OPTION_POSTGRESQL_LISTEN_ADDRESSES(config), \ OPTION_POSTGRESQL_AUTH_METHOD(config), \ OPTION_POSTGRESQL_HBA_LEVEL(config), \ + OPTION_POSTGRESQL_WAL_LEVEL(config), \ OPTION_SSL_ACTIVE(config), \ OPTION_SSL_MODE(config), \ OPTION_SSL_CA_FILE(config), \ diff --git a/src/bin/pg_autoctl/monitor.c b/src/bin/pg_autoctl/monitor.c index f24c34643..9cc1e6b49 100644 --- a/src/bin/pg_autoctl/monitor.c +++ b/src/bin/pg_autoctl/monitor.c @@ -4002,23 +4002,26 @@ monitor_print_formation_settings_as_json(Monitor *monitor, char *formation) /* - * monitor_synchronous_standby_names returns the value for the Postgres - * parameter "synchronous_standby_names" to use for a given group. The setting - * is computed on the monitor depending on the current values of the formation - * number_sync_standbys and each node's candidate priority and replication - * quorum properties. + * monitor_get_group_setting_from_sql runs the given single-row, single + * text-column SQL query (taking formation and groupId as its two + * parameters, in that order) and copies the result into buffer. settingName + * is only used to word the error messages. + * + * Shared by monitor_synchronous_standby_names() and + * monitor_synchronized_standby_slots() below, which otherwise only differ + * in which monitor SQL function to call and what to name the setting in + * their error messages. */ -bool -monitor_synchronous_standby_names(Monitor *monitor, - char *formation, int groupId, - char *synchronous_standby_names, int size) +static bool +monitor_get_group_setting_from_sql(Monitor *monitor, + const char *sql, + const char *settingName, + char *formation, int groupId, + char *buffer, int size) { PGSQL *pgsql = &monitor->pgsql; SingleValueResultContext context = { { 0 }, PGSQL_RESULT_STRING, false }; - const char *sql = - "select pgautofailover.synchronous_standby_names($1, $2)"; - int paramCount = 2; Oid paramTypes[2] = { TEXTOID, INT4OID }; const char *paramValues[2] = { 0 }; @@ -4031,9 +4034,9 @@ monitor_synchronous_standby_names(Monitor *monitor, paramCount, paramTypes, paramValues, &context, &parseSingleValueResult)) { - log_error("Failed to get the synchronous_standby_names setting value " + log_error("Failed to get the %s setting value " " from the monitor for formation %s and group %d", - formation, groupId); + settingName, formation, groupId); if (context.strVal) { free(context.strVal); @@ -4043,10 +4046,10 @@ monitor_synchronous_standby_names(Monitor *monitor, if (!context.parsedOk) { - log_error("Failed to get the synchronous_standby_names setting value " + log_error("Failed to get the %s setting value " " from the monitor for formation %s and group %d," "see above for details", - formation, groupId); + settingName, formation, groupId); if (context.strVal) { free(context.strVal); @@ -4054,13 +4057,58 @@ monitor_synchronous_standby_names(Monitor *monitor, return false; } - strlcpy(synchronous_standby_names, context.strVal, size); + strlcpy(buffer, context.strVal, size); free(context.strVal); return true; } +/* + * monitor_synchronous_standby_names returns the value for the Postgres + * parameter "synchronous_standby_names" to use for a given group. The setting + * is computed on the monitor depending on the current values of the formation + * number_sync_standbys and each node's candidate priority and replication + * quorum properties. + */ +bool +monitor_synchronous_standby_names(Monitor *monitor, + char *formation, int groupId, + char *synchronous_standby_names, int size) +{ + return monitor_get_group_setting_from_sql( + monitor, + "select pgautofailover.synchronous_standby_names($1, $2)", + "synchronous_standby_names", + formation, groupId, + synchronous_standby_names, size); +} + + +/* + * monitor_synchronized_standby_slots returns the value for the Postgres 17+ + * parameter "synchronized_standby_slots" to use for a given group, computed + * on the monitor from the current failover candidates in that group (see + * pgautofailover.synchronized_standby_slots() and its C implementation, + * synchronized_standby_slots() in node_active_protocol.c, for the details). + * This is what lets a failover-enabled logical replication slot survive a + * promotion: the primary only lets such slots advance once every node that + * might get promoted has replayed that far. + */ +bool +monitor_synchronized_standby_slots(Monitor *monitor, + char *formation, int groupId, + char *synchronized_standby_slots, int size) +{ + return monitor_get_group_setting_from_sql( + monitor, + "select pgautofailover.synchronized_standby_slots($1, $2)", + "synchronized_standby_slots", + formation, groupId, + synchronized_standby_slots, size); +} + + /* * monitor_set_hostname sets the hostname on the monitor, using a simple SQL * update command. diff --git a/src/bin/pg_autoctl/monitor.h b/src/bin/pg_autoctl/monitor.h index 3b0fbe770..d4960def8 100644 --- a/src/bin/pg_autoctl/monitor.h +++ b/src/bin/pg_autoctl/monitor.h @@ -279,6 +279,11 @@ bool monitor_synchronous_standby_names(Monitor *monitor, char *synchronous_standby_names, int size); +bool monitor_synchronized_standby_slots(Monitor *monitor, + char *formation, int groupId, + char *synchronized_standby_slots, + int size); + bool monitor_update_node_metadata(Monitor *monitor, int64_t nodeId, const char *name, diff --git a/src/bin/pg_autoctl/primary_standby.c b/src/bin/pg_autoctl/primary_standby.c index 7510fa132..57491a965 100644 --- a/src/bin/pg_autoctl/primary_standby.c +++ b/src/bin/pg_autoctl/primary_standby.c @@ -47,7 +47,10 @@ static void local_postgres_update_pg_failures_tracking(LocalPostgresServer *post { "port", "5432" }, \ { "max_wal_senders", "12" }, \ { "max_replication_slots", "12" }, \ - { "wal_level", "'replica'" }, \ + { "wal_level", "'logical'" }, /* the default; postgres_add_default_settings() \ + * patches this entry to 'replica' for a \ + * non-Citus node created with \ + * --wal-level replica -- see there. */ \ { "wal_log_hints", "on" }, \ { "wal_sender_timeout", "'30s'" }, \ { "hot_standby_feedback", "on" }, \ @@ -86,6 +89,22 @@ GUC postgres_default_settings_13[] = { { NULL, NULL } }; +/* + * Postgres 17 adds the slot-sync worker (sync_replication_slots) that keeps + * failover-enabled logical replication slots in sync on a physical standby, + * which is what lets a CDC consumer's slot survive promotion. It is only + * useful together with hot_standby_feedback (already on above) and a + * primary_conninfo that includes dbname (handled separately in + * prepare_recovery_settings()); it is a no-op when no failover slot exists, + * so it's safe to enable unconditionally on every 17+ node rather than + * behind a separate opt-in setting. + */ +GUC postgres_default_settings_17[] = { + DEFAULT_GUC_SETTINGS_FOR_PG_AUTO_FAILOVER_13, + { "sync_replication_slots", "on" }, + { NULL, NULL } +}; + GUC citus_default_settings_pre_13[] = { DEFAULT_GUC_SETTINGS_FOR_PG_AUTO_FAILOVER_PRE_13, { "shared_preload_libraries", "'citus,pg_stat_statements'" }, @@ -108,6 +127,18 @@ GUC citus_default_settings_13[] = { { NULL, NULL } }; +GUC citus_default_settings_17[] = { + DEFAULT_GUC_SETTINGS_FOR_PG_AUTO_FAILOVER_13, + { "shared_preload_libraries", "'citus,pg_stat_statements'" }, + { "wal_level", "logical" }, + { "citus.node_conninfo", "'sslmode=prefer'" }, + { "citus.cluster_name", "'default'" }, + { "citus.use_secondary_nodes", "'never'" }, + { "citus.local_hostname", "'localhost'" }, + { "sync_replication_slots", "on" }, + { NULL, NULL } +}; + /* * local_postgres_init_internal initializes an interface for managing a local @@ -705,6 +736,26 @@ primary_drop_all_replication_slots(LocalPostgresServer *postgres) return false; } + /* + * Postgres 17+ CDC continuity (issue #1188): the query above only ever + * touches physical slots matching our own naming pattern, so a logical + * failover slot this node still owns from being primary survives it + * untouched. If that slot is a stale (failover AND NOT synced) leftover + * -- which is exactly what it is, now that we're becoming a standby -- + * it blocks the slot-sync worker from ever starting, so drop it here + * too. This can be reached either after a clean demotion or, via + * fsm_rewind_or_init(), after one interrupted by demote_timeout. + */ + if (pg_setup_synchronized_standby_slots_supported(&(postgres->postgresSetup))) + { + if (!postgres_drop_stale_failover_slots(postgres)) + { + log_error("Failed to drop stale failover replication slots on " + "the local Postgres instance, see above for details"); + return false; + } + } + return true; } @@ -729,6 +780,26 @@ postgres_replication_slot_create_and_drop(LocalPostgresServer *postgres, } +/* + * postgres_drop_stale_failover_slots drops stale (failover AND NOT synced) + * logical replication slots on the local Postgres instance. See + * pgsql_drop_stale_failover_slots() and primary_drop_all_replication_slots() + * (issue #1188) for the rationale. + */ +bool +postgres_drop_stale_failover_slots(LocalPostgresServer *postgres) +{ + PGSQL *pgsql = &(postgres->sqlClient); + + log_trace("postgres_drop_stale_failover_slots"); + + bool result = pgsql_drop_stale_failover_slots(pgsql); + + pgsql_finish(pgsql); + return result; +} + + /* * postgres_replication_slot_advance advances the current confirmed position of * the given replication slot up to the given LSN position. @@ -769,6 +840,39 @@ primary_set_synchronous_standby_names(LocalPostgresServer *postgres) } +/* + * primary_set_synchronized_standby_slots sets the Postgres 17+ + * synchronized_standby_slots GUC on a primary postgres node, so that + * failover-enabled logical replication slots (CDC) survive a promotion. + * + * Postgres versions before 17 don't have this GUC at all, so we skip + * applying it there rather than fail an ALTER SYSTEM SET against an unknown + * parameter name. + */ +bool +primary_set_synchronized_standby_slots(LocalPostgresServer *postgres) +{ + PGSQL *pgsql = &(postgres->sqlClient); + + if (!pg_setup_synchronized_standby_slots_supported(&(postgres->postgresSetup))) + { + log_debug("Skipping synchronized_standby_slots: " + "not supported before Postgres 17"); + return true; + } + + log_info("Setting synchronized_standby_slots to '%s'", + postgres->synchronizedStandbySlots); + + bool result = + pgsql_set_synchronized_standby_slots(pgsql, + postgres->synchronizedStandbySlots); + + pgsql_finish(pgsql); + return result; +} + + /* * primary_disable_synchronous_replication disables synchronous replication * on a primary postgres node. @@ -800,6 +904,8 @@ postgres_add_default_settings(LocalPostgresServer *postgres, PostgresSetup *pgSetup = &(postgres->postgresSetup); char configFilePath[MAXPGPATH]; GUC *default_settings = NULL; + GUC patchedSettings[32] = { 0 }; + char walLevelValue[NAMEDATALEN + 2] = { 0 }; log_trace("postgres_add_default_settings (%s) [%d]", nodeKindToString(postgres->pgKind), @@ -842,6 +948,17 @@ postgres_add_default_settings(LocalPostgresServer *postgres, default_settings = postgres_default_settings_pre_13; } } + else if (pg_setup_synchronized_standby_slots_supported(pgSetup)) + { + if (IS_CITUS_INSTANCE_KIND(postgres->pgKind)) + { + default_settings = citus_default_settings_17; + } + else + { + default_settings = postgres_default_settings_17; + } + } else { if (IS_CITUS_INSTANCE_KIND(postgres->pgKind)) @@ -854,6 +971,47 @@ postgres_add_default_settings(LocalPostgresServer *postgres, } } + /* + * wal_level already defaults to 'logical' in the non-Citus + * default_settings arrays above: CDC continuity (see docs/cdc.rst) + * needs it on any node that might ever become primary, which in + * pg_auto_failover is every node in the group. Citus nodes already + * force 'logical' unconditionally for Citus's own unrelated needs, so + * they're left out of this override entirely. + * + * --wal-level replica at node creation time (pgSetup->walLevel) is the + * one case that needs patching: a non-Citus node opting back out, for + * clusters that know they'll never use CDC and would rather not pay + * wal_level=logical's WAL-volume and catalog xmin-retention cost. The + * static default_settings array is copied into a local, patched array + * here rather than hardcoding the opt-out into it, so there's exactly + * one place that decides the actual value written to postgresql.conf. + */ + if (!IS_CITUS_INSTANCE_KIND(postgres->pgKind) && + !IS_EMPTY_STRING_BUFFER(pgSetup->walLevel) && + strcmp(pgSetup->walLevel, "logical") != 0) + { + int i = 0; + int maxEntries = (int) (sizeof(patchedSettings) / + sizeof(patchedSettings[0])); + + sformat(walLevelValue, sizeof(walLevelValue), "'%s'", pgSetup->walLevel); + + for (i = 0; default_settings[i].name != NULL && i < maxEntries - 1; i++) + { + patchedSettings[i] = default_settings[i]; + + if (strcmp(default_settings[i].name, "wal_level") == 0) + { + patchedSettings[i].value = walLevelValue; + } + } + patchedSettings[i].name = NULL; + patchedSettings[i].value = NULL; + + default_settings = patchedSettings; + } + if (!pg_add_auto_failover_default_settings(pgSetup, hostname, configFilePath, diff --git a/src/bin/pg_autoctl/primary_standby.h b/src/bin/pg_autoctl/primary_standby.h index d3bc06c64..a0365fbfc 100644 --- a/src/bin/pg_autoctl/primary_standby.h +++ b/src/bin/pg_autoctl/primary_standby.h @@ -78,6 +78,7 @@ typedef struct LocalPostgresServer LocalExpectedPostgresStatus expectedPgStatus; char standbyTargetLSN[PG_LSN_MAXLENGTH]; char synchronousStandbyNames[BUFSIZE]; + char synchronizedStandbySlots[BUFSIZE]; PostgresVersionInfo pgVersion; } LocalPostgresServer; @@ -108,8 +109,10 @@ bool primary_drop_replication_slot(LocalPostgresServer *postgres, bool primary_drop_replication_slots(LocalPostgresServer *postgres); bool primary_drop_all_replication_slots(LocalPostgresServer *postgres); bool primary_set_synchronous_standby_names(LocalPostgresServer *postgres); +bool primary_set_synchronized_standby_slots(LocalPostgresServer *postgres); bool postgres_replication_slot_create_and_drop(LocalPostgresServer *postgres, NodeAddressArray *nodeArray); +bool postgres_drop_stale_failover_slots(LocalPostgresServer *postgres); bool postgres_replication_slot_maintain(LocalPostgresServer *postgres, NodeAddressArray *nodeArray); bool primary_disable_synchronous_replication(LocalPostgresServer *postgres); diff --git a/src/bin/pg_autoctl/watch.c b/src/bin/pg_autoctl/watch.c index 9426d107d..72d02d47d 100644 --- a/src/bin/pg_autoctl/watch.c +++ b/src/bin/pg_autoctl/watch.c @@ -277,6 +277,48 @@ cli_watch_update_from_monitor(WatchContext *context) return false; } + /* + * Postgres 17+ CDC continuity (see docs/cdc.rst): count how many + * physical slots the group's synchronized_standby_slots currently + * lists, same group/formation the rest of this dashboard is showing. + * This is purely topology-derived on the monitor side, so it's safe + * (and cheap) to always fetch, whether or not any node is running + * Postgres 17 or holds a failover-enabled logical slot. + */ + { + char synchronizedStandbySlots[BUFSIZE] = { 0 }; + + if (!monitor_synchronized_standby_slots( + monitor, + context->formation, + context->groupId, + synchronizedStandbySlots, + BUFSIZE)) + { + /* errors have already been logged */ + return false; + } + + if (IS_EMPTY_STRING_BUFFER(synchronizedStandbySlots)) + { + context->synchronizedStandbySlotsCount = 0; + } + else + { + /* comma-separated slot names: N commas means N+1 slots */ + int count = 1; + + for (char *p = synchronizedStandbySlots; *p != '\0'; p++) + { + if (*p == ',') + { + ++count; + } + } + context->synchronizedStandbySlotsCount = count; + } + } + if (!monitor_get_last_events(monitor, context->formation, context->groupId, @@ -639,6 +681,33 @@ print_watch_header(WatchContext *context, int r) mvprintw(r, c, "%d", context->number_sync_standbys); attroff(A_BOLD); + { + char nssStr[12] = { 0 }; + + sformat(nssStr, sizeof(nssStr), "%d", context->number_sync_standbys); + c += strlen(nssStr); + } + + /* + * Postgres 17+ CDC continuity (docs/cdc.rst): show how many physical + * slots synchronized_standby_slots currently lists for this group, so + * an operator watching a live failover can see at a glance whether a + * candidate dropping out (count going down) could be stalling logical + * decoding. Zero is the common, uninteresting case (no CDC in use, or + * no promotion candidates), so it isn't worth the row's space unless + * there's something to see. + */ + if (context->synchronizedStandbySlotsCount > 0 && + context->cols > (c + 9 + 14 + 3)) + { + mvprintw(r, c, " - CDC Slots: "); /* that's 14 chars */ + c += 14; + + attron(A_BOLD); + mvprintw(r, c, "%d", context->synchronizedStandbySlotsCount); + attroff(A_BOLD); + } + /* we only use one row */ return 1; } diff --git a/src/bin/pg_autoctl/watch.h b/src/bin/pg_autoctl/watch.h index 0a2e1e748..74dc6bf9d 100644 --- a/src/bin/pg_autoctl/watch.h +++ b/src/bin/pg_autoctl/watch.h @@ -66,6 +66,7 @@ typedef struct WatchContext char formation[NAMEDATALEN]; int groupId; int number_sync_standbys; + int synchronizedStandbySlotsCount; /* Postgres 17+ CDC continuity, see docs/cdc.rst */ /* data to display */ CurrentNodeStateArray nodesArray; diff --git a/src/monitor/Makefile b/src/monitor/Makefile index a86986656..9715751f4 100644 --- a/src/monitor/Makefile +++ b/src/monitor/Makefile @@ -14,7 +14,7 @@ MODULE_big = $(EXTENSION) OBJS = $(patsubst ${SRC_DIR}%.c,%.o,$(wildcard ${SRC_DIR}*.c)) PG_CPPFLAGS = -Wall -Werror -Wno-unused-parameter -Iinclude -I$(libpq_srcdir) -g SHLIB_LINK = $(libpq) -REGRESS = create_extension monitor workers node_active_protocol guard_data_loss fast_forward drop_node stale_primary_report candidate_count_gate lock_and_fetch_migration timeline_fork_detection dummy_update drop_extension upgrade +REGRESS = create_extension monitor workers node_active_protocol guard_data_loss fast_forward drop_node stale_primary_report candidate_count_gate lock_and_fetch_migration timeline_fork_detection synchronized_standby_slots dummy_update drop_extension upgrade ISOLATION = concurrent_remove_node concurrent_remove_standby_and_primary_report concurrent_remove_standby_and_standby_report concurrent_second_primary_death_report concurrent_health_check_and_report concurrent_candidate_priority_and_quorum PG_CONFIG ?= pg_config diff --git a/src/monitor/expected/synchronized_standby_slots.out b/src/monitor/expected/synchronized_standby_slots.out new file mode 100644 index 000000000..81085b373 --- /dev/null +++ b/src/monitor/expected/synchronized_standby_slots.out @@ -0,0 +1,250 @@ +-- Copyright (c) Microsoft Corporation. All rights reserved. +-- Licensed under the PostgreSQL License. +-- +-- Regression test for pgautofailover.synchronized_standby_slots(), the +-- Postgres 17+ CDC-continuity companion to synchronous_standby_names: it +-- lists the physical replication slot names of the group's current +-- failover candidates (AutoFailoverNode.candidatePriority > 0) that are +-- also currently reporting themselves as secondary (reportedState), so +-- that a failover-enabled logical replication slot only advances once +-- every node that might get promoted right now has replayed that far -- +-- excluding candidates that are lagging or otherwise not caught up. +-- +-- Same bootstrap sequence as candidate_count_gate.sql: drive the FSM to +-- primary + secondary + secondary through pgautofailover.node_active(). +-- +-- Unlike candidate_count_gate.sql, register_node()'s result is deliberately +-- NOT printed with "SELECT *": assigned_node_id is a raw sequence value +-- that depends on how many nodes every earlier test in regress_schedule +-- has already registered, which would make this file's expected output +-- fragile to the schedule's exact composition. Only assigned_group_state, +-- which is what this test actually cares about, is printed; node ids are +-- captured silently via \gset for use in node_active() below. +\x on +-- ── formation and node registration ───────────────────────────────────────── +SELECT pgautofailover.create_formation('sss_test', 'pgsql', 'postgres', true, 1); +-[ RECORD 1 ]----+------------------------------ +create_formation | (sss_test,pgsql,postgres,t,1) + +SELECT assigned_group_state + FROM pgautofailover.register_node('sss_test', 'sss_p', 5432, + 'postgres', 'sss_p', 1); +-[ RECORD 1 ]--------+------- +assigned_group_state | single + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'sss_test' AND nodename = 'sss_p' \gset +SELECT assigned_group_state + FROM pgautofailover.register_node('sss_test', 'sss_s1', 5432, + 'postgres', 'sss_s1', 1); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'sss_test' AND nodename = 'sss_s1' \gset +SELECT assigned_group_state + FROM pgautofailover.register_node('sss_test', 'sss_s2', 5432, + 'postgres', 'sss_s2', 1); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'sss_test' AND nodename = 'sss_s2' \gset +-- ── bootstrap: drive the FSM to primary + secondary + secondary ───────────── +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'single'); +-[ RECORD 1 ]--------+------- +assigned_group_state | single + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'single', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+-------- +assigned_group_state | primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+-------- +assigned_group_state | primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +-- p: primary (refresh to pick up second secondary) +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+--------------- +assigned_group_state | apply_settings + +-- Verify bootstrap: p=primary, s1=secondary, s2=secondary. +SELECT nodename, goalstate, reportedstate, candidatepriority + FROM pgautofailover.node + WHERE formationid = 'sss_test' + ORDER BY nodename; +-[ RECORD 1 ]-----+--------------- +nodename | sss_p +goalstate | apply_settings +reportedstate | primary +candidatepriority | 100 +-[ RECORD 2 ]-----+--------------- +nodename | sss_s1 +goalstate | secondary +reportedstate | secondary +candidatepriority | 100 +-[ RECORD 3 ]-----+--------------- +nodename | sss_s2 +goalstate | catchingup +reportedstate | secondary +candidatepriority | 100 + +-- ── both standbys are candidates: both slot names, in candidatePriority order +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); +-[ RECORD 1 ]--------------+---------------------------------------------------- +synchronized_standby_slots | pgautofailover_standby_39,pgautofailover_standby_40 + +-- ── s2 self-reports lagging (catchingup): it drops out despite still being +-- a failover candidate -- this is the lag opt-out: holding the logical +-- slot's progress hostage to a standby that isn't caught up right now +-- would defeat the point of CDC continuity. Note goalstate for s2 was +-- already catchingup before this (an extra standby beyond what +-- number_sync_standbys needs keeps that goal for quorum bookkeeping); +-- what flips here is reportedstate, which is what we actually filter on. +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'sss_test' AND nodename = 'sss_s2'; +-[ RECORD 1 ]-+----------- +nodename | sss_s2 +goalstate | catchingup +reportedstate | catchingup + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); +-[ RECORD 1 ]--------------+-------------------------- +synchronized_standby_slots | pgautofailover_standby_39 + +-- ── s2 catches back up and reports secondary again: it's back ───────────── +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); +-[ RECORD 1 ]--------------+---------------------------------------------------- +synchronized_standby_slots | pgautofailover_standby_39,pgautofailover_standby_40 + +-- ── s2 opts out of the failover quorum: only s1's slot remains ───────────── +SELECT pgautofailover.set_node_candidate_priority('sss_test', 'sss_s2', 0); +-[ RECORD 1 ]---------------+-- +set_node_candidate_priority | t + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); +-[ RECORD 1 ]--------------+-------------------------- +synchronized_standby_slots | pgautofailover_standby_39 + +-- ── no candidates left at all: empty string disables the feature ─────────── +SELECT pgautofailover.set_node_candidate_priority('sss_test', 'sss_s1', 0); +NOTICE: setting candidate priority to zero, preventing automated failover +DETAIL: Group 0 in formation "sss_test" have no failover candidate. +-[ RECORD 1 ]---------------+-- +set_node_candidate_priority | t + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); +-[ RECORD 1 ]--------------+- +synchronized_standby_slots | + +-- ── a single-node group has nothing to synchronize against ───────────────── +SELECT pgautofailover.create_formation('sss_single', 'pgsql', 'postgres', true, 1); +-[ RECORD 1 ]----+-------------------------------- +create_formation | (sss_single,pgsql,postgres,t,1) + +SELECT assigned_group_state + FROM pgautofailover.register_node('sss_single', 'sss_only', 5432, + 'postgres', 'sss_only', 1); +-[ RECORD 1 ]--------+------- +assigned_group_state | single + +SELECT pgautofailover.synchronized_standby_slots('sss_single', 0); +-[ RECORD 1 ]--------------+- +synchronized_standby_slots | + +-- ── an empty group errors out, same as synchronous_standby_names ────────── +SELECT pgautofailover.synchronized_standby_slots('sss_test', 42); +ERROR: no nodes found in group 42 of formation "sss_test" diff --git a/src/monitor/node_active_protocol.c b/src/monitor/node_active_protocol.c index a1e22db01..2704e49e4 100644 --- a/src/monitor/node_active_protocol.c +++ b/src/monitor/node_active_protocol.c @@ -71,6 +71,7 @@ PG_FUNCTION_INFO_V1(set_node_replication_quorum); PG_FUNCTION_INFO_V1(set_node_region); PG_FUNCTION_INFO_V1(report_postgres_version); PG_FUNCTION_INFO_V1(synchronous_standby_names); +PG_FUNCTION_INFO_V1(synchronized_standby_slots); PG_FUNCTION_INFO_V1(testing_lock_formation); PG_FUNCTION_INFO_V1(testing_lock_node_group); PG_FUNCTION_INFO_V1(testing_set_node_health); @@ -2510,6 +2511,37 @@ update_node_metadata(PG_FUNCTION_ARGS) } +/* + * AppendStandbySlotNames appends the pgautofailover_standby_ names + * of every node in nodeList to buf, in list order, joined by separator. + * Shared by synchronous_standby_names() and synchronized_standby_slots() + * below, which otherwise only differ in which node list they build this + * from and how they wrap the result (synchronous_standby_names() wraps it + * in "ANY N (...)" itself, around this call). + */ +static void +AppendStandbySlotNames(StringInfo buf, List *nodeList, const char *separator) +{ + ListCell *nodeCell = NULL; + bool firstNode = true; + + foreach(nodeCell, nodeList) + { + AutoFailoverNode *node = (AutoFailoverNode *) lfirst(nodeCell); + + appendStringInfo(buf, + "%spgautofailover_standby_%lld", + firstNode ? "" : separator, + (long long) node->nodeId); + + if (firstNode) + { + firstNode = false; + } + } +} + + /* * synchronous_standby_names returns the synchronous_standby_names parameter * value for a given Postgres service group in a given formation. @@ -2620,30 +2652,133 @@ synchronous_standby_names(PG_FUNCTION_ARGS) : formation->number_sync_standbys; StringInfo sbnames = makeStringInfo(); - ListCell *nodeCell = NULL; - bool firstNode = true; appendStringInfo(sbnames, "ANY %d (", number_sync_standbys); + AppendStandbySlotNames(sbnames, syncStandbyNodesGroupList, ", "); + appendStringInfoString(sbnames, ")"); - foreach(nodeCell, syncStandbyNodesGroupList) - { - AutoFailoverNode *node = (AutoFailoverNode *) lfirst(nodeCell); + PG_RETURN_TEXT_P(cstring_to_text(sbnames->data)); + } + } +} - appendStringInfo(sbnames, - "%spgautofailover_standby_%lld", - firstNode ? "" : ", ", - (long long) node->nodeId); - if (firstNode) - { - firstNode = false; - } - } - appendStringInfoString(sbnames, ")"); +/* + * synchronized_standby_slots returns the value to use for the Postgres 17+ + * "synchronized_standby_slots" parameter on the primary of a given Postgres + * service group in a given formation. + * + * synchronized_standby_slots lists the physical replication slot names that + * a logical replication slot created with failover => true must wait on + * (via the slot's synced copy catching up) before its confirmed_flush_lsn is + * allowed to advance. This is what lets a CDC consumer's logical slot + * survive a failover: as long as every node that might get promoted has + * already replayed everything the logical slot has confirmed, promoting it + * can never leave the slot pointing past data that node doesn't have. + * + * We compute this the same way as synchronous_standby_names above, but + * listing every current failover *candidate* (AutoFailoverNode. + * candidatePriority > 0) rather than only the nodes currently part of the + * synchronous replication quorum: a node can be promoted whether or not it + * was ever a synchronous standby, so the logical slot must wait on it + * regardless. Nodes with candidatePriority == 0 can only be promoted + * manually (force-failover), which is already an exceptional, operator- + * driven action, so we don't hold back CDC consumers for them. + * + * Among those candidates, only the ones that last reported themselves as + * SECONDARY are listed (AutoFailoverNode.reportedState, not goalState: an + * extra standby beyond what number_sync_standbys needs can sit at goal + * catchingup indefinitely for quorum bookkeeping alone, while genuinely + * being a fully caught-up secondary). A candidate the monitor has pushed + * back to catchingup because it actually fell behind or missed its + * reporting window, or that hasn't reached secondary yet at all + * (wait_standby, join_secondary, ...), is excluded: it remains a failover + * candidate in principle, but it isn't caught up right now, and listing it + * would make the logical slot's progress hostage to that node recovering + * -- the opposite of the opt-out this GUC is meant to give us. + * + * Returns the empty string when there is nothing to wait on (a single node + * group, or no promotion candidates currently reporting secondary), which + * disables the feature: Postgres treats an empty synchronized_standby_slots + * as "no requirement". + */ +Datum +synchronized_standby_slots(PG_FUNCTION_ARGS) +{ + checkPgAutoFailoverVersion(); - PG_RETURN_TEXT_P(cstring_to_text(sbnames->data)); + text *formationIdText = PG_GETARG_TEXT_P(0); + char *formationId = text_to_cstring(formationIdText); + + int32 groupId = PG_GETARG_INT32(1); + + List *nodesGroupList = AutoFailoverNodeGroup(formationId, groupId); + int nodesCount = list_length(nodesGroupList); + + /* + * When there's no nodes registered yet, there's no pg_autoctl process + * that needs the information anyway. Return NULL. + */ + if (nodesCount == 0) + { + ereport(ERROR, + (errcode(ERRCODE_INVALID_OBJECT_DEFINITION), + errmsg("no nodes found in group %d of formation \"%s\"", + groupId, formationId))); + } + + /* a single node has nothing to synchronize against */ + if (nodesCount == 1) + { + PG_RETURN_TEXT_P(cstring_to_text("")); + } + + AutoFailoverNode *primaryNode = GetPrimaryNodeInGroup(formationId, groupId); + + List *standbyNodesGroupList = AutoFailoverOtherNodesList(primaryNode); + List *candidateNodesList = GroupListCandidates(standbyNodesGroupList); + + /* + * Only wait on a candidate that last reported itself as SECONDARY. We + * deliberately check reportedState, not goalState: an extra standby + * beyond what number_sync_standbys needs can sit at goalState + * catchingup indefinitely for quorum bookkeeping reasons alone, while + * genuinely being a fully caught-up secondary (reportedState + * secondary) -- that node belongs in the list. What we want to + * exclude is a candidate the monitor has pushed back to catchingup + * because it actually fell behind or missed its reporting window + * (reportedState catchingup), or one that hasn't reached secondary yet + * at all (wait_standby, join_secondary, ...). Holding a failover- + * enabled logical slot's progress hostage to such a node would defeat + * the point of CDC continuity: the slot would simply stop advancing + * until that node recovers, instead of only waiting on candidates + * actually able to take over right now. + */ + List *readyCandidateNodesList = NIL; + ListCell *candidateCell = NULL; + + foreach(candidateCell, candidateNodesList) + { + AutoFailoverNode *node = (AutoFailoverNode *) lfirst(candidateCell); + + if (node->reportedState == REPLICATION_STATE_SECONDARY) + { + readyCandidateNodesList = lappend(readyCandidateNodesList, node); } } + + if (list_length(readyCandidateNodesList) == 0) + { + PG_RETURN_TEXT_P(cstring_to_text("")); + } + else + { + StringInfo slotNames = makeStringInfo(); + + AppendStandbySlotNames(slotNames, readyCandidateNodesList, ","); + + PG_RETURN_TEXT_P(cstring_to_text(slotNames->data)); + } } diff --git a/src/monitor/pgautofailover--2.2--2.3.sql b/src/monitor/pgautofailover--2.2--2.3.sql index 871e6b83a..e7de407f6 100644 --- a/src/monitor/pgautofailover--2.2--2.3.sql +++ b/src/monitor/pgautofailover--2.2--2.3.sql @@ -689,3 +689,105 @@ with last_events as ) select * from last_events order by eventtime, eventid; $$; + + +-- +-- Postgres 17+ CDC continuity: synchronized_standby_slots(formation, group) +-- computes the Postgres 17+ "synchronized_standby_slots" setting for a +-- group, the same way synchronous_standby_names is computed, but listing +-- every current failover candidate's physical slot name instead of only +-- the synchronous replication quorum. This is what lets a failover-enabled +-- logical replication slot survive a promotion: it only advances once +-- every node that might get promoted has replayed that far. See +-- docs/cdc.rst for the full picture (issue #1188). +-- + +CREATE FUNCTION pgautofailover.synchronized_standby_slots + ( + IN formation_id text default 'default', + IN group_id int default 0 + ) +RETURNS text LANGUAGE C STRICT +AS 'MODULE_PATHNAME', $$synchronized_standby_slots$$; + +comment on function pgautofailover.synchronized_standby_slots(text, int) + is 'get the Postgres 17+ synchronized_standby_slots setting for a given group, used for CDC/logical slot continuity across failover'; + +grant execute on function + pgautofailover.synchronized_standby_slots(text, int) + to autoctl_node; + +-- Re-create formation_settings() to also report synchronized_standby_slots, +-- next to the existing synchronous_standby_names row. +CREATE OR REPLACE FUNCTION pgautofailover.formation_settings + ( + IN formation_id text default 'default', + OUT context text, + OUT group_id int, + OUT node_id bigint, + OUT nodename text, + OUT setting text, + OUT value text + ) +RETURNS SETOF record LANGUAGE SQL STRICT +AS $$ + with groups(formationid, groupid) as + ( + select formationid, groupid + from pgautofailover.node + where formationid = formation_id + group by formationid, groupid + ) + + -- context: formation, number_sync_standbys + select 'formation' as context, + NULL as group_id, NULL as node_id, formationid as nodename, + 'number_sync_standbys' as setting, + cast(number_sync_standbys as text) as value + from pgautofailover.formation + where formationid = formation_id + +union all + + -- context: primary, one entry per group in the formation + select 'primary', groups.groupid, nodes.node_id, nodes.node_name, + 'synchronous_standby_names', + format('''%s''', + pgautofailover.synchronous_standby_names(formationid, groupid)) + from groups, pgautofailover.get_nodes(formationid, groupid) as nodes + where node_is_primary + +union all + + -- context: primary, synchronized_standby_slots (Postgres 17+ CDC continuity) + select 'primary', groups.groupid, nodes.node_id, nodes.node_name, + 'synchronized_standby_slots', + format('''%s''', + pgautofailover.synchronized_standby_slots(formationid, groupid)) + from groups, pgautofailover.get_nodes(formationid, groupid) as nodes + where node_is_primary + +union all + +( + -- context: node, one entry per node in the formation + select 'node', node.groupid, node.nodeid, node.nodename, + 'replication quorum', cast(node.replicationquorum as text) + from pgautofailover.node as node + where node.formationid = formation_id +order by nodeid +) + +union all + +( + select 'node', node.groupid, node.nodeid, node.nodename, + 'candidate priority', cast(node.candidatepriority as text) + from pgautofailover.node as node + where node.formationid = formation_id +order by nodeid +) +$$; + +comment on function pgautofailover.formation_settings(text) + is 'get the current replication settings a formation'; diff --git a/src/monitor/pgautofailover.sql b/src/monitor/pgautofailover.sql index 8545bf38c..f6f0b64d1 100644 --- a/src/monitor/pgautofailover.sql +++ b/src/monitor/pgautofailover.sql @@ -1228,6 +1228,22 @@ grant execute on function to autoctl_node; +create function pgautofailover.synchronized_standby_slots + ( + IN formation_id text default 'default', + IN group_id int default 0 + ) +returns text language C strict +AS 'MODULE_PATHNAME', $$synchronized_standby_slots$$; + +comment on function pgautofailover.synchronized_standby_slots(text, int) + is 'get the Postgres 17+ synchronized_standby_slots setting for a given group, used for CDC/logical slot continuity across failover'; + +grant execute on function + pgautofailover.synchronized_standby_slots(text, int) + to autoctl_node; + + CREATE FUNCTION pgautofailover.formation_settings ( IN formation_id text default 'default', @@ -1266,6 +1282,16 @@ union all from groups, pgautofailover.get_nodes(formationid, groupid) as nodes where node_is_primary +union all + + -- context: primary, synchronized_standby_slots (Postgres 17+ CDC continuity) + select 'primary', groups.groupid, nodes.node_id, nodes.node_name, + 'synchronized_standby_slots', + format('''%s''', + pgautofailover.synchronized_standby_slots(formationid, groupid)) + from groups, pgautofailover.get_nodes(formationid, groupid) as nodes + where node_is_primary + union all ( diff --git a/src/monitor/regress_schedule b/src/monitor/regress_schedule index 4d2ca7bdb..6d188d8ff 100644 --- a/src/monitor/regress_schedule +++ b/src/monitor/regress_schedule @@ -46,6 +46,7 @@ test: lock_and_fetch_migration test: timeline_fork_detection test: failover_candidate_leaves_secondary test: cluster_init_failover_rule_attribution +test: synchronized_standby_slots test: dummy_update test: drop_extension test: upgrade diff --git a/src/monitor/sql/synchronized_standby_slots.sql b/src/monitor/sql/synchronized_standby_slots.sql new file mode 100644 index 000000000..efe32c9e5 --- /dev/null +++ b/src/monitor/sql/synchronized_standby_slots.sql @@ -0,0 +1,182 @@ +-- Copyright (c) Microsoft Corporation. All rights reserved. +-- Licensed under the PostgreSQL License. +-- +-- Regression test for pgautofailover.synchronized_standby_slots(), the +-- Postgres 17+ CDC-continuity companion to synchronous_standby_names: it +-- lists the physical replication slot names of the group's current +-- failover candidates (AutoFailoverNode.candidatePriority > 0) that are +-- also currently reporting themselves as secondary (reportedState), so +-- that a failover-enabled logical replication slot only advances once +-- every node that might get promoted right now has replayed that far -- +-- excluding candidates that are lagging or otherwise not caught up. +-- +-- Same bootstrap sequence as candidate_count_gate.sql: drive the FSM to +-- primary + secondary + secondary through pgautofailover.node_active(). +-- +-- Unlike candidate_count_gate.sql, register_node()'s result is deliberately +-- NOT printed with "SELECT *": assigned_node_id is a raw sequence value +-- that depends on how many nodes every earlier test in regress_schedule +-- has already registered, which would make this file's expected output +-- fragile to the schedule's exact composition. Only assigned_group_state, +-- which is what this test actually cares about, is printed; node ids are +-- captured silently via \gset for use in node_active() below. + +\x on + +-- ── formation and node registration ───────────────────────────────────────── + +SELECT pgautofailover.create_formation('sss_test', 'pgsql', 'postgres', true, 1); + +SELECT assigned_group_state + FROM pgautofailover.register_node('sss_test', 'sss_p', 5432, + 'postgres', 'sss_p', 1); + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'sss_test' AND nodename = 'sss_p' \gset + +SELECT assigned_group_state + FROM pgautofailover.register_node('sss_test', 'sss_s1', 5432, + 'postgres', 'sss_s1', 1); + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'sss_test' AND nodename = 'sss_s1' \gset + +SELECT assigned_group_state + FROM pgautofailover.register_node('sss_test', 'sss_s2', 5432, + 'postgres', 'sss_s2', 1); + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'sss_test' AND nodename = 'sss_s2' \gset + +-- ── bootstrap: drive the FSM to primary + secondary + secondary ───────────── + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'single'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'wait_standby'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'single', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'wait_standby'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'wait_standby'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +-- p: primary (refresh to pick up second secondary) +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); + +-- Verify bootstrap: p=primary, s1=secondary, s2=secondary. +SELECT nodename, goalstate, reportedstate, candidatepriority + FROM pgautofailover.node + WHERE formationid = 'sss_test' + ORDER BY nodename; + +-- ── both standbys are candidates: both slot names, in candidatePriority order + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); + +-- ── s2 self-reports lagging (catchingup): it drops out despite still being +-- a failover candidate -- this is the lag opt-out: holding the logical +-- slot's progress hostage to a standby that isn't caught up right now +-- would defeat the point of CDC continuity. Note goalstate for s2 was +-- already catchingup before this (an extra standby beyond what +-- number_sync_standbys needs keeps that goal for quorum bookkeeping); +-- what flips here is reportedstate, which is what we actually filter on. + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); + +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'sss_test' AND nodename = 'sss_s2'; + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); + +-- ── s2 catches back up and reports secondary again: it's back ───────────── + +SELECT assigned_group_state + FROM pgautofailover.node_active('sss_test', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); + +-- ── s2 opts out of the failover quorum: only s1's slot remains ───────────── + +SELECT pgautofailover.set_node_candidate_priority('sss_test', 'sss_s2', 0); + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); + +-- ── no candidates left at all: empty string disables the feature ─────────── + +SELECT pgautofailover.set_node_candidate_priority('sss_test', 'sss_s1', 0); + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 0); + +-- ── a single-node group has nothing to synchronize against ───────────────── + +SELECT pgautofailover.create_formation('sss_single', 'pgsql', 'postgres', true, 1); + +SELECT assigned_group_state + FROM pgautofailover.register_node('sss_single', 'sss_only', 5432, + 'postgres', 'sss_only', 1); + +SELECT pgautofailover.synchronized_standby_slots('sss_single', 0); + +-- ── an empty group errors out, same as synchronous_standby_names ────────── + +SELECT pgautofailover.synchronized_standby_slots('sss_test', 42); diff --git a/tests/tap/schedule b/tests/tap/schedule index 3b29df16a..b808904f6 100644 --- a/tests/tap/schedule +++ b/tests/tap/schedule @@ -29,6 +29,7 @@ fsm_step_report_advance replication_stall_3dc fast_forward demote_timeout_wait_primary_deadlock +cdc_continuity wait_primary_draining_deadlock timeline_fork_report_lsn_deadlock keeper_fsm_gap_209_wait_maintenance diff --git a/tests/tap/schedules/multi-misc.sch b/tests/tap/schedules/multi-misc.sch index a2439f018..a081943fa 100644 --- a/tests/tap/schedules/multi-misc.sch +++ b/tests/tap/schedules/multi-misc.sch @@ -12,3 +12,9 @@ ensure multi_ifdown drop_node_destroy guard_data_loss + +# PG17+ only (multi-misc only runs on the PG17 matrix leg -- see +# .github/workflows/ci.yml): CDC continuity (synchronized_standby_slots, +# failover-enabled logical slots surviving a promotion) and the #1188 +# stale-failover-slot-after-demote_timeout fix. +cdc_continuity diff --git a/tests/tap/specs/cdc_continuity.pgaf b/tests/tap/specs/cdc_continuity.pgaf new file mode 100644 index 000000000..b2ea25b46 --- /dev/null +++ b/tests/tap/specs/cdc_continuity.pgaf @@ -0,0 +1,142 @@ +# CDC continuity (Postgres 17+): a logical replication slot created with +# failover => true must survive a promotion -- both a clean one (perform +# failover) and one triggered by a primary self-fencing through +# demote_timeout (issue #1188: an interrupted demotion can leave the old +# primary holding stale, non-synced failover slots that permanently block +# the slot-sync worker on the node once it rejoins as a standby). +# +# wal_level defaults to 'logical' on every node pg_auto_failover creates +# (see docs/cdc.rst), so there is nothing to set up here beyond the +# default: test_001 below creating a failover-enabled logical slot right +# after setup{} is itself the regression guard that the default actually +# took effect on both nodes (it would fail outright with a plain Postgres +# error otherwise, same as it would on wal_level='replica'). +# +# Earlier revisions of this spec had to flip wal_level on after the fact, +# and getting that ordering wrong (restarting the *primary* after it +# already had a synchronous standby attached) reliably self-fenced it into +# demote_timeout mid-setup -- exactly the risky pattern +# basic_operation.pgaf's own comments warn about. See +# docs/cdc.rst's wal_level rollout runbook for that procedure, still +# needed for a cluster created before this default, or with +# --wal-level replica. + +cluster { + monitor + formation { + node1 + node2 + } +} + +setup { + wait until primary, secondary timeout 120s +} + +teardown { + compose down +} + +# +# test_001: create a failover-enabled logical slot on the primary, verify +# the monitor-computed synchronized_standby_slots setting names +# the standby's physical slot, and that the slot gets synced. +# + +step test_001_create_failover_slot { + sql node1 { + SELECT slot_name FROM pg_create_logical_replication_slot( + 'cdc_slot', 'pgoutput', false, false, true); + } + expect { cdc_slot } + + sql node1 { SHOW synchronized_standby_slots; } + expect { pgautofailover_standby_2 } +} + +step test_002_slot_gets_synced_to_standby { + # The slot-sync worker polls periodically; give it a few seconds. + sleep 5s + sql node2 { + SELECT slot_name, failover, synced + FROM pg_replication_slots + WHERE slot_name = 'cdc_slot'; + } + expect { cdc_slot|t|t } +} + +# +# test_003: a clean failover must hand the slot to the new primary still +# usable for logical decoding, with no resync required. Note: +# "synced" stays true forever as a provenance marker (this slot +# originated as a synced copy) -- it is not what governs whether +# the slot can be decoded from, and is not reset on promotion. +# + +step test_003_clean_failover_preserves_slot { + perform failover + wait until node2 state is primary + and node1 state is secondary + timeout 90s + + sql node2 { + SELECT slot_name, failover, synced + FROM pg_replication_slots + WHERE slot_name = 'cdc_slot'; + } + expect { cdc_slot|t|t } +} + +# +# test_004: reproduce issue #1188 -- at this point node2 is primary (from +# test_003) holding the real, promoted copy of cdc_slot, and +# node1 is its standby with a synced copy. node2 self-fences +# through demote_timeout (network partition, same technique as +# demote_timeout_wait_primary_deadlock.pgaf), node1 is promoted, +# and node2 rejoins as a standby still holding its own stale +# (failover AND NOT synced) copy of the slot. Before the fix, +# this permanently blocked node2's slot-sync worker, logging +# "same name slot ... already exists on the standby" forever. +# +# This necessarily re-promotes node1 -- cdc_slot's original +# creator -- a second time, which independently runs into a +# separate, narrower Postgres limitation (see "Limitations" in +# docs/cdc.rst: a node round-tripping back to primary can lose +# its own originally-created slot's data, regardless of this +# fix). That's why test_005 below only asserts the #1188 +# regression itself (no blocking name collision, slot-sync +# worker not stuck) rather than re-asserting data continuity +# already covered by test_002/003 -- doing both at once in a +# 2-node formation would conflate two different limitations. +# A 3-node formation could isolate them fully; left as a +# possible follow-up. +# + +step test_004_primary_self_fences_mid_demotion { + network disconnect node2 + wait until node2 assigned-state = demote_timeout timeout 60s + wait until node1 state is wait_primary timeout 60s + + # Let node2's own keeper notice the partition and stop Postgres + # (network_partition_timeout, default 20s) before reconnecting it. + sleep 30s + network connect node2 +} + +step test_005_old_primary_rejoins_without_the_1188_deadlock { + wait until node2 state is secondary + and node1 state is primary + timeout 90s + + # Give the slot-sync worker's retry loop time to either succeed or get + # stuck -- these are absence checks, so they're only meaningful once + # enough time has passed for the problem to have shown up if it exists. + sleep 20s + + # The slot-sync worker must not be stuck behind node2's own leftover + # failover slot (the exact symptom reported in #1188): neither the + # permanent name-collision error nor an endless "would lose data" retry + # loop (a sign the stale slot cleanup ran too late or not at all). + logs node2 not contains "already exists on the standby" + logs node2 not contains "could not synchronize replication slot" +}