Postgres 17+ CDC continuity: synchronized_standby_slots, demotion slot cleanup (fixes #1188)

docs/faq.rst

@@ -279,3 +279,40 @@ following commands::
 
   $ pg_autoctl disable monitor
   $ pg_autoctl enable monitor
+
+Can I use logical replication / CDC tools (Debezium, etc) with pg_auto_failover?
+---------------------------------------------------------------------------------
+
+Yes, as long as every node in the formation runs **Postgres 17 or more
+recent**. See :ref:`cdc_continuity` for the full picture; the short version:
+
+ - pg_auto_failover doesn't create or manage your logical replication
+   slots itself -- that stays your CDC tool's job, same as with any other
+   Postgres cluster.
+
+ - What pg_auto_failover does manage, automatically, on Postgres 17+: the
+   ``synchronized_standby_slots`` setting on the primary (kept in sync with
+   the current failover candidates), the ``dbname`` Postgres needs in
+   ``primary_conninfo`` to run its slot-sync worker, and ``sync_replication_slots``
+   / ``hot_standby_feedback`` on every standby. Together, these are what let
+   a logical slot created with ``failover => true`` keep working after a
+   promotion, instead of being silently left behind on the old primary.
+
+ - ``wal_level = logical`` is the default on every node pg_auto_failover
+   creates (see :ref:`wal_level_default`) -- you only need to set it
+   yourself when opting back in on a cluster created with
+   ``--wal-level replica``, or one created before this default existed
+   (see :ref:`wal_level_rollout`). You still need to create your slot(s)
+   with ``failover => true`` yourself: pg_auto_failover manages the
+   infrastructure that keeps them working across a promotion, not slot
+   creation itself.
+
+On Postgres versions before 17, Postgres itself has no mechanism to keep a
+logical slot's position correct on a standby without losing unconsumed
+changes, so there is nothing pg_auto_failover can safely do there: a
+logical slot on the primary does not survive a failover, the same as
+without pg_auto_failover. This was the situation discussed in `issue #400`__
+and `issue #1030`__ before Postgres 17 added slot synchronization.
+
+__ https://github.com/hapostgres/pg_auto_failover/issues/400
+__ https://github.com/hapostgres/pg_auto_failover/issues/1030

docs/fault-tolerance.rst

@@ -1,3 +1,5 @@
+.. _fault_tolerance:
+
 Failover and Fault Tolerance
 ============================
 

docs/index.rst

@@ -55,6 +55,7 @@ __ https://github.com/hapostgres/pg_auto_failover
    architecture-multi-standby
    failover-state-machine
    fault-tolerance
+   cdc
    security
 
 .. toctree::

docs/operations.rst

@@ -1,3 +1,5 @@
+.. _operations:
+
 Operating pg_auto_failover
 ==========================
 
@@ -194,6 +196,8 @@ monitor.
 And when the upgrade is done we can use ``pg_autoctl show state`` on the
 monitor to see that eveything is as expected.
 
+.. _upgrading_from_previous_versions:
+
 Upgrading from previous pg_auto_failover versions
 -------------------------------------------------
 

docs/ref/configuration.rst

@@ -229,6 +229,13 @@ Name of the PostgreSQL replication slot used in the streaming replication
 setup automatically deployed by pg_auto_failover. Replication slots can't be renamed
 in PostgreSQL.
 
+On Postgres 17 and more recent, pg_auto_failover also maintains the
+``synchronized_standby_slots`` Postgres setting on the primary, computed
+from the formation's current failover candidates: this is not a
+pg_auto_failover configuration key of its own (there is nothing to set),
+but it is what lets a failover-enabled logical replication slot survive a
+promotion. See :ref:`cdc_continuity` for the full picture.
+
 **replication.maximum_backup_rate**
 
 When pg_auto_failover (re-)builds a standby node using the ``pg_basebackup``

docs/ref/pg_autoctl.rst

@@ -52,6 +52,7 @@ pg_autoctl provides the following commands::
     state          Prints monitor's state of nodes in a given formation and group
     settings       Print replication settings for a formation from the monitor
     standby-names  Prints synchronous_standby_names for a given group
+    synchronized-standby-slots  Prints synchronized_standby_slots for a given group (Postgres 17+ CDC continuity)
     timeline       Show the timeline history known to the monitor for a group, and each node's position against it
     file           List pg_autoctl internal files (config, state, pid)
     systemd        Print systemd service file for this node

docs/ref/pg_autoctl_show.rst

@@ -13,6 +13,7 @@ pg_autoctl show - Show pg_auto_failover information
    pg_autoctl_show_state
    pg_autoctl_show_settings
    pg_autoctl_show_standby_names
+   pg_autoctl_show_synchronized_standby_slots
    pg_autoctl_show_timeline
    pg_autoctl_show_file
    pg_autoctl_show_systemd

docs/ref/pg_autoctl_show_synchronized_standby_slots.rst

@@ -0,0 +1,100 @@
+.. _pg_autoctl_show_synchronized_standby_slots:
+
+pg_autoctl show synchronized-standby-slots
+===========================================
+
+pg_autoctl show synchronized-standby-slots - Prints synchronized_standby_slots for a given group (Postgres 17+ CDC continuity)
+
+Synopsis
+--------
+
+This command prints the current value for the Postgres 17+
+``synchronized_standby_slots`` setting for the primary Postgres server of
+the target group (default ``0``) in the target formation (default
+``default``), as computed by the monitor: the comma-separated physical
+replication slot names of the group's current failover candidates. See
+:ref:`cdc_continuity` for the full picture of what this setting is for::
+
+  usage: pg_autoctl show synchronized-standby-slots  [ --pgdata ] --formation --group
+
+    --pgdata      path to data directory
+    --monitor     pg_auto_failover Monitor Postgres URL
+    --formation   formation to query, defaults to 'default'
+    --group       group to query formation, defaults to all
+    --json        output data in the JSON format
+
+Options
+-------
+
+--pgdata
+
+  Location of the Postgres node being managed locally. Defaults to the
+  environment variable ``PGDATA``. Use ``--monitor`` to connect to a monitor
+  from anywhere, rather than the monitor URI used by a local Postgres node
+  managed with ``pg_autoctl``.
+
+--monitor
+
+  Postgres URI used to connect to the monitor. Must use the ``autoctl_node``
+  username and target the ``pg_auto_failover`` database name. It is possible
+  to show the Postgres URI from the monitor node using the command
+  :ref:`pg_autoctl_show_uri`.
+
+  Defaults to the value of the environment variable ``PG_AUTOCTL_MONITOR``.
+
+--formation
+
+  Show the current ``synchronized_standby_slots`` value for the given
+  formation. Defaults to the ``default`` formation.
+
+--group
+
+  Show the current ``synchronized_standby_slots`` value for the given group
+  in the given formation. Defaults to group ``0``.
+
+--json
+
+  Output a JSON formatted data instead of a table formatted list.
+
+Environment
+-----------
+
+PGDATA
+
+  Postgres directory location. Can be used instead of the ``--pgdata``
+  option.
+
+PG_AUTOCTL_MONITOR
+
+  Postgres URI to connect to the monitor node, can be used instead of the
+  ``--monitor`` option.
+
+XDG_CONFIG_HOME
+
+  The pg_autoctl command stores its configuration files in the standard
+  place XDG_CONFIG_HOME. See the `XDG Base Directory Specification`__.
+
+  __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
+
+XDG_DATA_HOME
+
+  The pg_autoctl command stores its internal states files in the standard
+  place XDG_DATA_HOME, which defaults to ``~/.local/share``. See the `XDG
+  Base Directory Specification`__.
+
+  __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
+
+Examples
+--------
+
+::
+
+   $ pg_autoctl show synchronized-standby-slots
+   'pgautofailover_standby_2,pgautofailover_standby_3'
+
+   $ pg_autoctl show synchronized-standby-slots --json
+   {
+       "formation": "default",
+       "group": 0,
+       "synchronized_standby_slots": "pgautofailover_standby_2,pgautofailover_standby_3"
+   }

docs/ref/pg_autoctl_watch.rst

@@ -84,6 +84,16 @@ currently selected formation, the formation replication setting
 :ref:`number_sync_standbys`, and then in the right most position the current
 time.
 
+Whenever the group has at least one failover candidate, the header also
+shows a "CDC Slots" count: how many physical slot names the
+monitor-computed ``synchronized_standby_slots`` setting currently lists for
+that group (see :ref:`cdc_continuity`). This is the same list regardless of
+whether you actually have a failover-enabled logical replication slot in
+use -- it reflects the topology, not live slot activity. Watching this
+number drop during a failover is a quick way to notice a candidate dropping
+out of the list; see :ref:`pg_autoctl_show_synchronized_standby_slots` for
+the full slot name list.
+
 The second section displays one line per node, and each line contains a list
 of columns that describe the current state for the node. This list can
 includes the following columns, and which columns are part of the output

src/bin/common/pgsetup.c

@@ -1996,6 +1996,31 @@ pg_setup_standby_slot_supported(PostgresSetup *pgSetup, int logLevel)
 }
 
 
+/*
+ * pg_setup_synchronized_standby_slots_supported returns true when the
+ * target Postgres instance represented in pgSetup is Postgres 17 or more
+ * recent, the first version that implements the "synchronized_standby_slots"
+ * GUC and the pg_sync_replication_slots() slot-sync worker that keeps
+ * failover-enabled logical replication slots usable across a physical
+ * standby promotion. Before 17, Postgres has no mechanism to keep a logical
+ * slot's position correct on a standby without losing unconsumed changes,
+ * so pg_auto_failover's CDC-continuity support only applies to 17+.
+ */
+bool
+pg_setup_synchronized_standby_slots_supported(PostgresSetup *pgSetup)
+{
+	int pg_version = 0;
+
+	if (!parse_pg_version_string(pgSetup->pg_version, &pg_version))
+	{
+		/* errors have already been logged */
+		return false;
+	}
+
+	return pg_version >= 1700;
+}
+
+
 /*
  * pgsetup_parse_hba_level parses a string that represents an HBAEditLevel
  * value.

src/bin/common/pgsetup.h

@@ -233,6 +233,7 @@ typedef struct pg_setup
 	NodeReplicationSettings settings;       /* node replication settings */
 	SSLOptions ssl;                         /* ssl options */
 	char citusClusterName[NAMEDATALEN];     /* citus.cluster_name */
+	char walLevel[NAMEDATALEN];             /* wal_level, defaults to "logical" */
 } PostgresSetup;
 
 #define IS_EMPTY_STRING_BUFFER(strbuf) (strbuf[0] == '\0')
@@ -277,6 +278,7 @@ SSLMode pgsetup_parse_sslmode(const char *sslMode);
 char * pgsetup_sslmode_to_string(SSLMode sslMode);
 
 bool pg_setup_standby_slot_supported(PostgresSetup *pgSetup, int logLevel);
+bool pg_setup_synchronized_standby_slots_supported(PostgresSetup *pgSetup);
 
 HBAEditLevel pgsetup_parse_hba_level(const char *level);
 char * pgsetup_hba_level_to_string(HBAEditLevel hbaLevel);

src/bin/common/pgsql.c

@@ -1489,6 +1489,37 @@ pgsql_set_synchronous_standby_names(PGSQL *pgsql,
 }
 
 
+/*
+ * pgsql_set_synchronized_standby_slots sets the Postgres 17+
+ * synchronized_standby_slots GUC on the local Postgres to the value
+ * computed on the pg_auto_failover monitor: the comma-separated list of
+ * physical replication slot names of the current failover candidates. This
+ * is what makes a failover-enabled logical replication slot wait for every
+ * promotable standby before its confirmed position advances, so the slot
+ * stays usable by a CDC consumer across a promotion.
+ */
+bool
+pgsql_set_synchronized_standby_slots(PGSQL *pgsql,
+									 char *synchronized_standby_slots)
+{
+	char quoted[BUFSIZE] = { 0 };
+	GUC setting = { "synchronized_standby_slots", quoted };
+
+	if (sformat(quoted, BUFSIZE, "'%s'", synchronized_standby_slots) >= BUFSIZE)
+	{
+		log_error("Failed to apply the synchronized_standby_slots value \"%s\": "
+				  "pg_autoctl supports values up to %d bytes and this one "
+				  "requires %lu bytes",
+				  synchronized_standby_slots,
+				  BUFSIZE,
+				  (unsigned long) strlen(synchronized_standby_slots));
+		return false;
+	}
+
+	return pgsql_alter_system_set(pgsql, setting);
+}
+
+
 /*
  * pgsql_replication_slot_maintain advances the current confirmed position of
  * the given replication slot up to the given LSN position, create the
@@ -1590,6 +1621,37 @@ pgsql_drop_replication_slot(PGSQL *pgsql, const char *slotName)
 }
 
 
+/*
+ * pgsql_drop_stale_failover_slots drops every logical replication slot that
+ * still has failover enabled but isn't a synced copy (failover AND NOT
+ * synced): a slot in exactly that state on a node that is becoming a
+ * standby again is necessarily a stale leftover from when this node was
+ * primary, not something the slot-sync worker created. Left in place, it
+ * blocks that worker from starting at all ("same name slot ... already
+ * exists on the standby"), so the standby never gets a working synced copy
+ * (see issue #1188: this happens whenever demotion is interrupted, e.g. by
+ * demote_timeout, so the normal demotion cleanup below doesn't get a chance
+ * to run first).
+ *
+ * The failover/synced columns only exist from Postgres 17 onward (the
+ * version that introduces failover-enabled logical slots and the slot-sync
+ * worker in the first place), so this must only ever be called when
+ * pg_setup_synchronized_standby_slots_supported() is true.
+ */
+bool
+pgsql_drop_stale_failover_slots(PGSQL *pgsql)
+{
+	char *sql =
+		"SELECT pg_drop_replication_slot(slot_name) "
+		"  FROM pg_replication_slots "
+		" WHERE failover AND NOT synced AND NOT active";
+
+	log_trace("pgsql_drop_stale_failover_slots");
+
+	return pgsql_execute_with_params(pgsql, sql, 0, NULL, NULL, NULL, NULL);
+}
+
+
 /*
  * BuildNodesArrayValues build the SQL expression to use in a FROM clause to
  * represent the list of other standby nodes from the given nodeArray.

src/bin/common/pgsql.h

@@ -375,9 +375,12 @@ bool pgsql_replication_slot_exists(PGSQL *pgsql, const char *slotName,
 								   bool *slotExists);
 bool pgsql_create_replication_slot(PGSQL *pgsql, const char *slotName);
 bool pgsql_drop_replication_slot(PGSQL *pgsql, const char *slotName);
+bool pgsql_drop_stale_failover_slots(PGSQL *pgsql);
 bool postgres_sprintf_replicationSlotName(int64_t nodeId, char *slotName, int size);
 bool pgsql_set_synchronous_standby_names(PGSQL *pgsql,
 										 char *synchronous_standby_names);
+bool pgsql_set_synchronized_standby_slots(PGSQL *pgsql,
+										  char *synchronized_standby_slots);
 bool pgsql_replication_slot_create_and_drop(PGSQL *pgsql,
 											NodeAddressArray *nodeArray);
 bool pgsql_replication_slot_maintain(PGSQL *pgsql, NodeAddressArray *nodeArray);

src/bin/pg_autoctl/cli_common.c

@@ -241,6 +241,22 @@ cli_common_keeper_getopts(int argc, char **argv,
 				break;
 			}
 
+			case 'K':
+			{
+				/* { "wal-level", required_argument, NULL, 'K' }, */
+				if (strcmp(optarg, "logical") != 0 && strcmp(optarg, "replica") != 0)
+				{
+					errors++;
+					log_error("Invalid value for --wal-level: \"%s\", "
+							  "expected \"logical\" (the default) or \"replica\"",
+							  optarg);
+				}
+
+				strlcpy(LocalOptionConfig.pgSetup.walLevel, optarg, NAMEDATALEN);
+				log_trace("--wal-level %s", LocalOptionConfig.pgSetup.walLevel);
+				break;
+			}
+
 			case 'S':
 			{
 				/* { "skip-pg-hba", required_argument, NULL, 'S' }, */

src/bin/pg_autoctl/cli_common.h

@@ -141,6 +141,7 @@ extern CommandLine show_state_command;
 extern CommandLine show_settings_command;
 extern CommandLine show_file_command;
 extern CommandLine show_standby_names_command;
+extern CommandLine show_synchronized_standby_slots_command;
 extern CommandLine show_timeline_command;
 
 /* cli_watch.c */