common: guard ecdh() against HSM fd being non-blocking - #9402
Draft
daywalker90 wants to merge 6 commits into
Draft
common: guard ecdh() against HSM fd being non-blocking#9402daywalker90 wants to merge 6 commits into
daywalker90 wants to merge 6 commits into
Conversation
daywalker90
force-pushed
the
fix-websocket-macos
branch
from
August 11, 2026 13:38
119dd88 to
2b01cff
Compare
On macOS under load, subdaemons intermittently die: connectd: **BROKEN** STATUS_FAIL_HSM_IO: No hsmd ECDH response channeld: exits 0 after WIRE_HSMD_GET_PER_COMMITMENT_POINT Both are a synchronous wire_sync_read() returning NULL on a fresh connection. The HSM fd in a subdaemon is one end of a socketpair created by hsmd (whose io loop sets O_NONBLOCK on the other end) and passed hsmd -> lightningd -> subdaemon via SCM_RIGHTS. On macOS the O_NONBLOCK flag follows the shared open file description across that chain, so the subdaemon's fd can be non-blocking, and read()/write() return EAGAIN before hsmd's (fast) reply has landed. Linux keeps the descriptions independent, which is why this never reproduces there. The previous attempts to force the fd blocking from the subdaemon (io_fd_block in ecdh_hsmd_setup) cannot win, because hsmd owns the other end of the same open file description and keeps it non-blocking. Fix at the shared choke point instead: make wire_sync_read() and wire_sync_write() tolerant of O_NONBLOCK by polling on EAGAIN and resuming, preserving any partial read. "Sync" then really means "read or write a complete message", regardless of the fd's blocking state. This covers connectd's ecdh(), channeld's hsm_req(), and every other subdaemon that does synchronous HSM I/O (openingd, closingd, onchaind). Also drop the now-unneeded io_fd_block() toggling from ecdh_hsmd_setup(), and include errno in the HSM I/O failure messages so a recurrence is diagnosable from the daemon log. Changelog-Fixed: connectd: fix intermittent "No hsmd ECDH response" crash on macOS under load (issue ElementsProject#9060).
daywalker90
force-pushed
the
fix-websocket-macos
branch
from
August 12, 2026 10:44
2b01cff to
35f36e9
Compare
The "tolerate a non-blocking fd" change to wire_sync_read() did not fix the
macOS flake; it only changed where it fails. Instead of failing cleanly on
EAGAIN ("No hsmd ECDH response") it now read a corrupted/desynced message
and aborted (SIGABRT in fromwire_hsmd_ecdh_resp, tal "Not a valid header").
Revert wire_sync.c to upstream.
We still don't know whether the HSM fd is O_NONBLOCK (EAGAIN), sees EOF (hsmd
closed the socketpair), or delivers garbage, so make the failure diagnosable:
- report errno in the "Write ECDH to hsmd failed" and "No hsmd ECDH response"
messages, and
- if hsmd's reply won't parse, hex-dump the exact bytes we read before
failing.
These are temporary diagnostics to pin down the macOS-only flake and should
be removed once root-caused.
channeld and openingd intermittently die on the macOS CI flake (test_websocket), e.g. "channeld died (0)" from hsm_req()'s wire_sync_read() returning NULL, and "openingd died" from its sync HSM I/O. connectd's ECDH is immune because ecdh_hsmd_setup() makes the HSM fd blocking. On macOS the O_NONBLOCK flag of the HSM socketpair is shared across the SCM_RIGHTS chain with hsmd's io loop, so the subdaemon's end can be O_NONBLOCK and a synchronous read() spuriously EAGAINs. Explicitly assert the fd is blocking at the top of each sync-HSM subdaemon's main(), before any wire_sync HSM I/O. On Linux the fd is already blocking, so this is a no-op (it mirrors subd.c:read_fds() and ecdh_hsmd_setup()). Changelog-Fixed: closingd: fix intermittent death from non-blocking HSM fd on macOS
daywalker90
force-pushed
the
fix-websocket-macos
branch
from
August 12, 2026 13:40
b17c3ac to
516a033
Compare
This reverts commit 516a033.
The macOS CI flake manifests as subdaemons dying on synchronous HSM I/O: connectd "No hsmd ECDH response: Undefined error: 0" (errno 0 = EOF), and channeld/openingd/simpleclosed dying in hsm_req(). Making the HSM fd blocking (io_fd_block) was proven ineffective, so this adds diagnostics to pin down where the EOF comes from: - ecdh_hsmd: report the fd number alongside errno on sync failure, and drop the io_fd_block() call (it contends over the shared open file description and is not the fix). - hsmd: log "Destroying client N" in destroy_client(), so we can tell whether hsmd is dropping a subdaemon's HSM client (subdaemon then reads EOF) versus the subdaemon's fd being aliased. - channeld/simpleclosed hsm_req: report errno and fd on failure. All temporary diagnostics to be removed once root-caused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #9323