Skip to content

Commit 4c8affa

Browse files
committed
docs(cluster_setup): cover synchronous replication and switchover
The cluster_setup page only described a single asynchronous standby: a replication user, wal_level=replica, pg_basebackup -R, and a raw pg_stat_replication dump. Nothing covered synchronous replication, how to promote a standby, what has to be cleaned up afterwards, or how the Oracle-compatible listener behaves on a standby. Add four sections to the CN and EN pages: - synchronous replication: both synchronous_standby_names notations, what they turn sync_state into (sync vs quorum), and the lag columns; - promotion drill: pg_ctl promote / pg_promote(), the read-only check, the split-brain check, and pg_rewind to re-attach the old primary; - per-node ivorysql.port consistency: the same instance answers with compatible_mode=pg or oracle depending on which listener the client reaches, so a standby that only sets port either fails to bind the Oracle socket or serves the wrong semantics; - replication monitoring: sync_priority, sync_state, and the lag columns. The promotion section documents the trap that costs the most time: a standby built with pg_basebackup -R inherits synchronous_standby_names, so right after promotion every commit waits in wait_event_type=IPC, wait_event=SyncRep and statement_timeout does not interrupt it. Clearing the parameter needs two separate statements because ALTER SYSTEM cannot run inside the same transaction as the reload.
1 parent 7d4ad9d commit 4c8affa

2 files changed

Lines changed: 383 additions & 1 deletion

File tree

‎CN/modules/ROOT/pages/master/cluster_setup.adoc‎

Lines changed: 191 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,3 +174,194 @@ ivorysql=# \l
174174
test | ivorysql | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | |
175175
(4 rows)
176176
```
177+
178+
== 同步复制
179+
180+
前文搭建的是异步流复制:主库提交后不必等待备库确认。对可靠性要求更高的场景可以打开同步复制,代价是**每次提交都要等备库把 WAL 刷到磁盘**,主库的事务延迟会叠加备库的写盘延迟与网络往返。在主节点 `postgresql.conf` 中追加(端口沿用本文示例的主库 `port = 5532`、`ivorysql.port = 1621`):
181+
182+
[source,text]
183+
----
184+
port = 5532
185+
ivorysql.port = 1621
186+
synchronous_standby_names = 'FIRST 1 (walreceiver)'
187+
synchronous_commit = on
188+
----
189+
190+
其中 `synchronous_commit = on` 是默认值,真正决定同步与否的是 `synchronous_standby_names`:为空即异步,非空则按名单等待备库确认。该参数有两种用法,**备库在 `pg_stat_replication.sync_state` 里的取值随写法不同**:
191+
192+
. 命名制:直接写备库的 `application_name`(`'walreceiver'`),或写成带优先级的 `FIRST 1 (walreceiver)`。被选中的备库 `sync_state` 为 `sync`。
193+
194+
+
195+
[source,text]
196+
----
197+
application_name | state | sync_state | sync_priority | sent_lsn | replay_lsn | write_lag | flush_lag | replay_lag
198+
------------------+-----------+------------+---------------+-----------+------------+-----------------+-----------------+-----------------
199+
walreceiver | streaming | sync | 1 | 0/3000000 | 0/3000000 | 00:00:00.000093 | 00:00:00.000093 | 00:00:00.000093
200+
(1 row)
201+
----
202+
203+
. 法定人数制:写 `'ANY 1 ({asterisk})'`,不点名,任意 1 个备库确认即可。此时备库 `sync_state` 为 `quorum`。
204+
205+
+
206+
[source,text]
207+
----
208+
application_name | state | sync_state | sync_priority | sent_lsn | replay_lsn | write_lag | flush_lag | replay_lag
209+
------------------+-----------+------------+---------------+-----------+------------+-----------------+-----------------+-----------------
210+
walreceiver | streaming | quorum | 1 | 0/3000000 | 0/3000000 | 00:00:00.000133 | 00:00:00.000133 | 00:00:00.000133
211+
(1 row)
212+
----
213+
214+
验收方式是在主库上看 `sync_state`:出现 `sync` 或 `quorum` 说明同步复制已经生效;仍是 `async`,通常是参数没被加载(漏了 reload 或重启),或备库的 `application_name` 与名单不匹配。名单里的备库全部断开时,主库的写事务会一直停在同步复制的等待上——维护与切换之前必须先处理这个参数,见下一节。
215+
216+
== 主备切换演练
217+
218+
切换(promote)是把备库提升为可写的主库。两种方式等价:
219+
220+
. 命令行:
221+
222+
+
223+
[source,bash]
224+
----
225+
$ pg_ctl -D /home/ivy/synctest/standby -w -t 60 promote
226+
waiting for server to promote.... done
227+
server promoted
228+
----
229+
230+
. SQL(在备库上以超级用户连接后执行):
231+
232+
+
233+
[source,sql]
234+
----
235+
SELECT pg_is_in_recovery();
236+
SELECT pg_promote(wait => true, wait_seconds => 60);
237+
SELECT pg_is_in_recovery();
238+
----
239+
240+
+
241+
[source,text]
242+
----
243+
before=true
244+
t
245+
after=false
246+
----
247+
248+
`pg_promote` 返回 `t` 表示提升请求已发出并等待完成;`pg_is_in_recovery()` 由 `true` 变 `false` 即提升成功。提升之前备库是只读的,写操作会被拒绝:
249+
250+
[source,text]
251+
----
252+
ERROR: cannot execute CREATE TABLE in a read-only transaction
253+
----
254+
255+
[CAUTION]
256+
用 `pg_basebackup -R` 搭出来的备库**继承**了主库的 `synchronous_standby_names` 与 `synchronous_commit`。提升之后它成为主库,但通常没有任何备库连着,此时提交会停在同步复制等待上——实测把 `statement_timeout` 设为 `8s`,第 5 秒采样该事务仍在等待:
257+
258+
[source,text]
259+
----
260+
pid | state | wait_event_type | wait_event | backend_xmin | query
261+
------+--------+-----------------+------------+--------------+--------------------------------------------------------------
262+
1132 | active | IPC | SyncRep | 773 | set statement_timeout='8s'; insert into sync_probe(id,note)
263+
----
264+
265+
所以提升后的第一件事,是在新主库上清空(或改写成本节点真实的备库名单):
266+
267+
[source,sql]
268+
----
269+
ALTER SYSTEM SET synchronous_standby_names = '';
270+
SELECT pg_reload_conf();
271+
----
272+
273+
两条语句要分开执行;写进同一个 `-c` 用分号连写会报 `ERROR: ALTER SYSTEM cannot run inside a transaction block`。清空后 `sync_names=[]`,同一条 `INSERT` 立即返回 `INSERT 0 1`。
274+
275+
旧主库并不知道自己已经不是主库,实测它仍在以主库身份运行,且没有任何备库连它:
276+
277+
[source,text]
278+
----
279+
old_primary_in_recovery=false replicas=0 slot_active=false
280+
----
281+
282+
两台节点都能写就是双主,必须先停掉旧主库,再用 `pg_rewind` 把它的数据目录拉回新主的时间线:
283+
284+
[source,bash]
285+
----
286+
$ pg_rewind -D <old_primary_data_dir> \
287+
--source-server="host=<new_primary_ip> port=<new_primary_port> user=ivorysql dbname=postgres"
288+
----
289+
290+
[CAUTION]
291+
执行前旧主必须已经停库——它此刻仍是可写主库,不停就重演会得到两个各写各的主库。`pg_rewind` 只负责把数据页差异搬回去,回退完成后仍要按前文重新核对本节点的 `port`、`ivorysql.port` 与 `primary_conninfo`。
292+
293+
== Oracle 兼容端口在主备上的一致性
294+
295+
IvorySQL 的一个实例同时监听两个 TCP 端口:PostgreSQL 端口 `port` 与 Oracle 兼容端口 `ivorysql.port`。启动日志里能看到两组监听地址与 socket:
296+
297+
[source,text]
298+
----
299+
LOG: listening on IPv4 address "127.0.0.1", port 5532
300+
LOG: listening on IPv4 address "127.0.0.1", port 1621
301+
LOG: listening on Unix socket "/tmp/.s.PGSQL.5532"
302+
LOG: listening on Unix socket "/tmp/.s.PGSQL.1621"
303+
----
304+
305+
**连哪个监听,决定这个会话看到的兼容模式。** 在单个实例上直接查:
306+
307+
[source,sql]
308+
----
309+
SHOW port;
310+
SHOW ivorysql.port;
311+
SHOW ivorysql.compatible_mode;
312+
----
313+
314+
实测值(同一个实例、同一份数据,只有连接端口不同):
315+
316+
[cols="1,1,1,1,1"]
317+
|===
318+
| 节点 | port | ivorysql.port | 连接端口 | compatible_mode
319+
| 主库 | 5532 | 1621 | 5532 | pg
320+
| 备库 | 5533 | 1622 | 5533 | pg
321+
| 备库 | 5533 | 1622 | 1622 | oracle
322+
|===
323+
324+
也就是说 `ivorysql.port` 不是 `port` 的别名,而是一个独立的监听:连 5533 得到 `pg`,连 1622 得到 `oracle`。因此**主备必须逐节点配置 `ivorysql.port`**:备库照前文改 `postgresql.conf` 时,`port` 与 `ivorysql.port` 要一起改。只改 `port`、把 `ivorysql.port` 留在 1521,应用走 1521 连备库时要么落到占用该端口的另一台实例上,要么在本机直接起不来:
325+
326+
[source,text]
327+
----
328+
LOG: listening on IPv4 address "127.0.0.1", port 5534
329+
LOG: could not bind IPv4 address "127.0.0.1": Address already in use
330+
HINT: Is another postmaster already running on port 1621? If not, wait a few seconds and retry.
331+
WARNING: could not create listen socket for "localhost"
332+
FATAL: could not create any Oracle TCP/IP sockets
333+
LOG: database system is shut down
334+
----
335+
336+
(`pg_ctl` 侧同时给出 `pg_ctl: could not start server`。)同一台机器上即可复现:主库仍占用 1621 时,用另一份数据目录以 `-p 5534 -c ivorysql.port=1621` 启动第二个实例。应用侧同理:应用要的是 Oracle 兼容语义,就必须连 `ivorysql.port`;而连到的那个端口值取自**所连节点自己**的配置,所以备库上的 `ivorysql.port` 必须是该备库真实监听的端口。
337+
338+
== 复制状态监控
339+
340+
主库的 `pg_stat_replication` 每个备库一行。日常监控只需要下列几列:
341+
342+
[source,sql]
343+
----
344+
SELECT application_name, state, sync_state, sync_priority,
345+
sent_lsn, write_lsn, flush_lsn, replay_lsn,
346+
write_lag, flush_lag, replay_lag, reply_time
347+
FROM pg_stat_replication;
348+
----
349+
350+
[cols="1,3"]
351+
|===
352+
| 列 | 含义
353+
| `application_name` | 备库自报的名字,默认 `walreceiver`;命名制同步复制按它挑选同步备库
354+
| `state` | `streaming` 表示 WAL 正在实时流式传输
355+
| `sync_state` | `async`(异步)、`sync`(命名制下的同步备库)、`quorum`(法定人数制下的同步备库)
356+
| `sync_priority` | 命名制名单里的优先级,数字越小越优先
357+
| `sent_lsn` | 主库已发送到该备库的 WAL 位置
358+
| `write_lsn` | 备库已写入磁盘的 WAL 位置
359+
| `flush_lsn` | 备库已刷盘的 WAL 位置,同步复制等的就是它
360+
| `replay_lsn` | 备库已回放的位置,备库上的读能看到这里
361+
| `write_lag` / `flush_lag` / `replay_lag` | 上面三个位置落后于 `sent_lsn` 的时间间隔
362+
| `reply_time` | 该备库最近一次状态回报的时间
363+
|===
364+
365+
判读:`replay_lsn` 落后 `sent_lsn` 说明还有 WAL 没回放完,备库上的读可能是旧数据;`write_lag`、`flush_lag`、`replay_lag` 为空说明最近一次状态回报周期内没有新的 WAL 需要处理;`sync_state` 只有 `async` 则仍是异步复制,主库提交不等备库。
366+
367+
备库只读由 `transaction_read_only = on` 保证,写操作一律被拒绝(报错原文见上一节的只读示例)。需要写入时按「主备切换演练」提升,不要尝试在备库上关掉只读。

‎EN/modules/ROOT/pages/master/cluster_setup.adoc‎

Lines changed: 192 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -175,4 +175,195 @@ ivorysql=# \l
175175
| | | | | | | | ivorysql=CTc/ivorysql
176176
test | ivorysql | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | |
177177
(4 rows)
178-
```
178+
```
179+
180+
== Synchronous replication
181+
182+
The cluster built above is asynchronous streaming replication: the primary does not wait for a standby to confirm a commit. If you need stronger durability, enable synchronous replication. The cost is that **every commit waits until a standby has flushed the WAL to disk**, so transaction latency on the primary includes the standby's disk write and the network round trip. Append the following to `postgresql.conf` on the primary node (the ports follow the examples in this chapter: `port = 5532` and `ivorysql.port = 1621` on the primary):
183+
184+
[source,text]
185+
----
186+
port = 5532
187+
ivorysql.port = 1621
188+
synchronous_standby_names = 'FIRST 1 (walreceiver)'
189+
synchronous_commit = on
190+
----
191+
192+
`synchronous_commit = on` is the default, so the parameter that actually decides whether replication is synchronous is `synchronous_standby_names`: leave it empty and the cluster stays asynchronous; set it and commits wait for the standbys in the list. There are two notations, and **the value a standby reports in `pg_stat_replication.sync_state` depends on which one you use**:
193+
194+
. Named notation: write the standby's `application_name` (`'walreceiver'`), or the form with a priority, `FIRST 1 (walreceiver)`. The selected standby reports `sync_state = sync`.
195+
196+
+
197+
[source,text]
198+
----
199+
application_name | state | sync_state | sync_priority | sent_lsn | replay_lsn | write_lag | flush_lag | replay_lag
200+
------------------+-----------+------------+---------------+-----------+------------+-----------------+-----------------+-----------------
201+
walreceiver | streaming | sync | 1 | 0/3000000 | 0/3000000 | 00:00:00.000093 | 00:00:00.000093 | 00:00:00.000093
202+
(1 row)
203+
----
204+
205+
. Quorum notation: write `'ANY 1 ({asterisk})'`, with no name list, so any one standby may confirm. The standby then reports `sync_state = quorum`.
206+
207+
+
208+
[source,text]
209+
----
210+
application_name | state | sync_state | sync_priority | sent_lsn | replay_lsn | write_lag | flush_lag | replay_lag
211+
------------------+-----------+------------+---------------+-----------+------------+-----------------+-----------------+-----------------
212+
walreceiver | streaming | quorum | 1 | 0/3000000 | 0/3000000 | 00:00:00.000133 | 00:00:00.000133 | 00:00:00.000133
213+
(1 row)
214+
----
215+
216+
To verify it, check `sync_state` on the primary: `sync` or `quorum` means synchronous replication is in effect; if it is still `async`, the setting was most likely never loaded (a missing reload or restart), or the standby's `application_name` does not match the name list. When every standby in the list is disconnected, write transactions on the primary stay parked in the synchronous replication wait — deal with this parameter before any maintenance or switchover, see the next section.
217+
218+
== Failover drill
219+
220+
Promoting a standby turns it into a writable primary. Both ways below are equivalent:
221+
222+
. From the command line:
223+
224+
+
225+
[source,bash]
226+
----
227+
$ pg_ctl -D /home/ivy/synctest/standby -w -t 60 promote
228+
waiting for server to promote.... done
229+
server promoted
230+
----
231+
232+
. From SQL (run it on the standby as a superuser):
233+
234+
+
235+
[source,sql]
236+
----
237+
SELECT pg_is_in_recovery();
238+
SELECT pg_promote(wait => true, wait_seconds => 60);
239+
SELECT pg_is_in_recovery();
240+
----
241+
242+
+
243+
[source,text]
244+
----
245+
before=true
246+
t
247+
after=false
248+
----
249+
250+
`pg_promote` returns `t` once the promotion has been requested and waited for, and `pg_is_in_recovery()` going from `true` to `false` means the promotion succeeded. Before promotion the standby is read-only and writes are rejected:
251+
252+
[source,text]
253+
----
254+
ERROR: cannot execute CREATE TABLE in a read-only transaction
255+
----
256+
257+
[CAUTION]
258+
A standby built with `pg_basebackup -R` **inherits** the primary's `synchronous_standby_names` and `synchronous_commit`. After promotion it is the primary, but it usually has no standby attached, so commits park in the synchronous replication wait. Measured with `statement_timeout` set to `8s`, the transaction was still waiting when sampled at the 5 second mark:
259+
260+
[source,text]
261+
----
262+
pid | state | wait_event_type | wait_event | backend_xmin | query
263+
------+--------+-----------------+------------+--------------+--------------------------------------------------------------
264+
1132 | active | IPC | SyncRep | 773 | set statement_timeout='8s'; insert into sync_probe(id,note)
265+
----
266+
267+
So the first thing to do after promotion is to clear the setting (or rewrite it with this node's real standby list) on the new primary:
268+
269+
[source,sql]
270+
----
271+
ALTER SYSTEM SET synchronous_standby_names = '';
272+
SELECT pg_reload_conf();
273+
----
274+
275+
Run the two statements separately; combining them in one `-c` with a semicolon fails with `ERROR: ALTER SYSTEM cannot run inside a transaction block`. After clearing it `sync_names=[]`, and the same `INSERT` returned `INSERT 0 1` immediately.
276+
277+
The old primary has no idea it is no longer the primary. Measured after the promotion it was still running as a primary, with no standby attached to it:
278+
279+
[source,text]
280+
----
281+
old_primary_in_recovery=false replicas=0 slot_active=false
282+
----
283+
284+
Two writable nodes is split brain. Stop the old primary first, then pull its data directory back onto the new primary's timeline with `pg_rewind`:
285+
286+
[source,bash]
287+
----
288+
$ pg_rewind -D <old_primary_data_dir> \
289+
--source-server="host=<new_primary_ip> port=<new_primary_port> user=ivorysql dbname=postgres"
290+
----
291+
292+
[CAUTION]
293+
The old primary must already be shut down — it is still a writable primary right now, and rewinding while it runs leaves you with two primaries accepting writes. `pg_rewind` only moves the data page differences back, so after the rewind still re-check this node's `port`, `ivorysql.port` and `primary_conninfo` as described earlier in this chapter.
294+
295+
== Oracle-compatible port consistency across primary and standby
296+
297+
A single IvorySQL instance listens on two TCP ports at once: the PostgreSQL port `port` and the Oracle-compatible port `ivorysql.port`. The startup log shows both listener pairs:
298+
299+
[source,text]
300+
----
301+
LOG: listening on IPv4 address "127.0.0.1", port 5532
302+
LOG: listening on IPv4 address "127.0.0.1", port 1621
303+
LOG: listening on Unix socket "/tmp/.s.PGSQL.5532"
304+
LOG: listening on Unix socket "/tmp/.s.PGSQL.1621"
305+
----
306+
307+
**The listener a session connects to decides the compatibility mode it sees.** Query it directly on one instance:
308+
309+
[source,sql]
310+
----
311+
SHOW port;
312+
SHOW ivorysql.port;
313+
SHOW ivorysql.compatible_mode;
314+
----
315+
316+
Measured values (same instance, same data, only the connection port differs):
317+
318+
[cols="1,1,1,1,1"]
319+
|===
320+
| Node | port | ivorysql.port | Connected to | compatible_mode
321+
| Primary | 5532 | 1621 | 5532 | pg
322+
| Standby | 5533 | 1622 | 5533 | pg
323+
| Standby | 5533 | 1622 | 1622 | oracle
324+
|===
325+
326+
In other words `ivorysql.port` is not an alias of `port` but a separate listener: connecting to 5533 gives `pg`, connecting to 1622 gives `oracle`. That is why **`ivorysql.port` must be configured on every node**: when you edit `postgresql.conf` on the standby as described earlier, change `port` and `ivorysql.port` together. Change only `port` and leave `ivorysql.port` at 1521, and an application that connects to the standby on 1521 either lands on whichever other instance already owns that port or the node simply refuses to start:
327+
328+
[source,text]
329+
----
330+
LOG: listening on IPv4 address "127.0.0.1", port 5534
331+
LOG: could not bind IPv4 address "127.0.0.1": Address already in use
332+
HINT: Is another postmaster already running on port 1621? If not, wait a few seconds and retry.
333+
WARNING: could not create listen socket for "localhost"
334+
FATAL: could not create any Oracle TCP/IP sockets
335+
LOG: database system is shut down
336+
----
337+
338+
(`pg_ctl` also reports `pg_ctl: could not start server`.) You can reproduce this on a single machine: while the primary still owns 1621, start a second instance from another data directory with `-p 5534 -c ivorysql.port=1621`. The same applies on the application side: an application that needs Oracle-compatible semantics has to connect to `ivorysql.port`, and the value of that port comes from the configuration of **the node it actually connects to**, so the `ivorysql.port` on a standby must be the port that standby really listens on.
339+
340+
== Replication monitoring
341+
342+
`pg_stat_replication` on the primary has one row per standby. Day-to-day monitoring only needs these columns:
343+
344+
[source,sql]
345+
----
346+
SELECT application_name, state, sync_state, sync_priority,
347+
sent_lsn, write_lsn, flush_lsn, replay_lsn,
348+
write_lag, flush_lag, replay_lag, reply_time
349+
FROM pg_stat_replication;
350+
----
351+
352+
[cols="1,3"]
353+
|===
354+
| Column | Meaning
355+
| `application_name` | The name the standby reports, `walreceiver` by default; named synchronous replication picks the synchronous standby by it
356+
| `state` | `streaming` means WAL is being streamed in real time
357+
| `sync_state` | `async` (asynchronous), `sync` (synchronous standby under named notation), `quorum` (synchronous standby under quorum notation)
358+
| `sync_priority` | Priority within the name list, lower numbers first
359+
| `sent_lsn` | WAL position the primary has sent to this standby
360+
| `write_lsn` | WAL position this standby has written to disk
361+
| `flush_lsn` | WAL position this standby has flushed, which is what synchronous replication waits for
362+
| `replay_lsn` | WAL position this standby has replayed; reads on the standby see data up to here
363+
| `write_lag` / `flush_lag` / `replay_lag` | How far behind `sent_lsn` those three positions are, as time intervals
364+
| `reply_time` | When this standby last reported its status
365+
|===
366+
367+
How to read it: `replay_lsn` behind `sent_lsn` means WAL is still waiting to be replayed, so reads on the standby may return old data; `write_lag`, `flush_lag` or `replay_lag` empty means no new WAL needed processing during the last status interval; `sync_state` only `async` means replication is still asynchronous and commits on the primary do not wait for a standby.
368+
369+
A standby is read-only because `transaction_read_only = on`, and every write is rejected (see the read-only example in the previous section for the exact error). When you need to write, promote it as described in "Failover drill" instead of trying to turn read-only off on the standby.

0 commit comments

Comments
 (0)