Add PostgreSQL node failover leases - #1000
Conversation
|
👋 Thanks for assigning @tnull as a reviewer! |
|
I’m primarily looking for a concept ACK on the overall direction at this stage. The code has already gone through several iterations, and it will probably need another pass once we agree that the lease, fencing, and process-exit approach is the right path.
|
Make PostgresStore acquire a table-scoped lease during construction and fence every mutation with it. Treat runtime lease loss as process-fatal so servers can exit without final persistence and restart from durable state.
629386d to
056be7a
Compare
|
Would be nice if this is abstracted out to also work with VSS. That way multi-device with a VSS backend can work |
|
The safety-critical part seems mostly backend-specific to me, since lease validation and the KV mutation must be atomic within the backend. What layer of this do you envision abstracting? If generalizing this later would not require much rewriting, I would prefer to leave any necessary code movement to a future VSS lease PR. |
|
Friendly ping @tnull. I would like to agree on the overall approach before doing further implementation cleanup. If you do not currently have the bandwidth to take a look, please let me know too. |
I'll try to get back to a first look some time this week, but review on PRs for v0.8 has priority right now. I think to begin with my first question is whether we don't need a more general version of this anyways, i.e., something a |
For an initial version, I think rebuilding can remain the responsibility of the infrastructure, for example Kubernetes. ldk-node reports the lease lost, ldk-server exits, and the infrastructure restarts the process. The wrapper you propose seems quite a bit broader, and I do not think it is needed to meet the initial objective of failover? As mentioned in #1000 (comment), making lease+mutation atomic is backend-specific, so a general wrapper would not replace that work. |
4 participants
Addresses #932 by adding minimal active/passive failover support for nodes using the built-in PostgreSQL store.
PostgresStoreacquires an exclusive lease for its KV table, renews it in the background, and fences every mutation by validating and renewing the lease in the same transaction. Lease loss is terminal for the node and is exposed to the application (ldk-server) so the process can exit and restart from persisted state.This initial implementation is intentionally limited to PostgreSQL. Generic stores and VSS are out of scope. The API, timing configuration, and backend support can be extended later if operational experience requires it.