Skip to content

Make the pool's invariant the default, and stop the library killing its host - #17

Merged
Sunrisepeak merged 1 commit into
masterfrom
fix/pool-invariants-and-sigpipe
Sep 5, 2026
Merged

Make the pool's invariant the default, and stop the library killing its host#17
Sunrisepeak merged 1 commit into
masterfrom
fix/pool-invariants-and-sigpipe

Conversation

@Sunrisepeak

@Sunrisepeak Sunrisepeak commented Sep 5, 2026

Copy link
Copy Markdown
Member

Closes #15, closes #16. 0.3.0.

#15 named two of the eleven paths that could leak a connection. All eleven are fixed here; four of the other nine turned up while verifying the report, and two were regressions 0.2.10 had introduced.

What was wrong

#16 — a write to a socket whose peer has gone away raises SIGPIPE, and a program that has not disarmed it (the default) is killed with exit status 141. The fd is one this library created, and the write is most often the close_notify its own pool clean-up sends. mbedtls guards against this in net_prepare with a process-wide signal(SIGPIPE, SIG_IGN); replacing mbedtls's network layer with a custom BIO dropped that guard and put nothing in its place.

#15 — a socket whose body was not read to the end went back into the pool, and the next request read the leftovers as its status line. Ten paths could do it, and every one has the same shape: an early return that did nothing, where doing nothing left the dirty socket pooled.

Three of them are ordinary paths with no timeout and no truncation involved:

a redirect whose body is never read the comment there promised a drain that was never written; the recursive call picks the socket straight back up
a non-2xx whose error body is never read
a download whose output file cannot be opened

And one needs no timing at all: send() held Content-Length in an int, so 4294967296 became 0, the body was taken for empty, and the socket went back with four gigabytes owed on it. That is the only pool defect a server can trigger with a single header.

Why there were eight and not one

send, send_stream and download_to_file each carried a near-copy of the status-line parse, the header loop and the body loop, and every past hardening landed on one or two of the three:

hardening send send_stream download_to_file
strict Content-Length (#14)
int64_t length
strict chunk size (#9)
CRLF after chunk data
pool clean-up on early exit

#14 is the most recent instance: it added a Content-Length branch to one copy and introduced two regressions doing it.

What this does

Also fixed along the way: a status line is now required to be one (BBBB 999 XHTTP/1.1 200 OK used to be reported as an ordinary 999); a header block cut short by a timeout is an error rather than the end of the headers; the CRLF after chunk data is verified everywhere; 1xx/204/304 are no longer read for a body they do not have; TLS back-pressure is waited on rather than retried once and abandoned.

New API — all additive, nothing changes shape

  • HttpResponse::bodyComplete / bodyError — a truncated 200 and a complete 200 were indistinguishable. ok() deliberately does not consult them, so existing if (res.ok()) means exactly what it did.
  • HttpClientConfig::maxResponseBodyBytes (64 MiB) — the size send() was about to allocate came from a header.
  • HttpClientConfig::retryOnStaleConnection (true) — a server closing an idle keep-alive connection is routine, and the client cannot see it until it writes. That reached callers as statusCode = 0, "No response" for a request the server never saw. One attempt, on a pooled connection, before a single response byte has arrived.
  • parse_status_line, exported and testable without a server.
  • TlsSocket::read_someData/WouldBlock/Eof/Error. read() collapsed an end of stream and a not-yet-ready transport into the same 0, which is why callers could not tell a finished body from a stalled one. read() is unchanged.

Tests

tests/ had no keep-alive coverage at all, which is the direct reason #14's regression merged green. It now scripts an in-process TLS server — the library speaks only HTTPS, so a plain listener cannot reach the code under test — and every pool test asserts how many TCP connections the server saw as well as what came back. #15's own report contains a case whose output is byte-for-byte correct and whose only symptom is the connection count.

47 tests, and each fix was checked by mutation:

mutation tests that fail
remove MSG_NOSIGNAL SigPipe.WritingToADepartedPeer…
empty the guard's destructor 6 pool tests
drain_body returns true without reading 2
parse_status_line mines digits again 3

Two tests did not survive that and were rewritten. The SIGPIPE child stopped at its first failed write — which returns ECONNRESET without a signal — so it passed either way. And the pool servers stalled rather than sending their remaining bytes, so the connection was silent rather than dirty and the stale-connection retry rescued the second request whether or not the guard worked.

Two things about where this runs

examples/openkal builds these sources above openkal, the portable kernel ABI, and makes a real HTTPS request through kal_net_connect — one static binary with 1363 mbedtls symbols and 107 kal_* symbols. It is a separate CI job because whether MSG_NOSIGNAL or SO_NOSIGPIPE exists is decided by the C library rather than the operating system (openkal-musl defines the first and not the second), and a #ifdef that is wrong about that compiles cleanly on the gcc job and fails there — which is how 5e7d66f reached master.

templates/ ships three starting points for mcpp new --template tinyhttps, one per entry point: fetch (default), download, stream. tools/template_smoke.sh renders and builds them against the working tree, because mcpp new can only reach a template that is already published — checking them after the release makes the first user the one who finds out. It has already caught one: import std carries no stdout macro, so a progress bar flushed through it compiled here and not in a generated project.

Two more the self-review found, one of them mine

  • A 1xx interim response was returned as the answer (RFC 9112 §2.1 requires reading past them). 103 Early Hints — sent proactively by several CDNs — and 100 Continue — sent for any request carrying Expect: 100-continue — reached the caller as the result. Worse, once a 1xx counted as "a response with no body", the connection was marked clean with the real response still on it, for the next request to read as its own: A socket whose body was not read to the end goes back into the pool, and the next request reads the remainder as its status line #15 arriving by another door, introduced by this PR's own no-body rule.
  • A peer's FIN was reported as a transport error. The BIO answered recv() == 0 with MBEDTLS_ERR_NET_CONN_RESET; mbedtls's own BIO passes the zero through, and ssl_fetch_input tests for exactly that zero to produce SSL_CONN_EOF (ssl_msg.c:2251). Since most servers close without a TLS close_notify, a body whose framing is the close (legal, RFC 9112 §6.3) read as truncated — and download_to_file returned ok() == false for a file that had arrived complete and correct.

Both are mutation-verified: reverting either makes two specific tests fail.

Two more unbounded loops, found while writing the reader

Neither was reported and both are memory- or time-exhaustion vectors a server controls:

  • a header block accumulates into a std::map, so an endless supply of short, well-formed header lines grows the client's memory without limit;
  • a trailer section is discarded rather than kept, but holds the call open just as long — every line resets the read timeout, so nothing else stops it.

Both are bounded now (200 lines), as each line's length already was (8 KiB).

Housekeeping

mcpp.lock is removed and ignored. Its own header says it does not pin anything — "index dependencies are re-resolved from their constraints each time" — so for a library, whose consumers resolve from its constraints, it recorded one machine's resolution and changed nothing about anyone else's. Eight of the ten mcpplibs packages already ignore it, the scaffold template among them.

Upgrading

Nothing was removed or changed shape; existing code compiles unchanged.

Two things for the release note:

  1. If you are on 0.2.10 and use send_stream, upgrade. A1/A2 are regressions Keep the error body of a failed streaming request, and fix the three framing defects the review found #14 introduced.
  2. Malformed responses that used to be accepted in silence now report an error. They are not new failures — they are failures that were previously invisible. bodyError says which.

@Sunrisepeak
Sunrisepeak force-pushed the fix/pool-invariants-and-sigpipe branch from 94bd45d to bdc7017 Compare September 5, 2026 19:26
…ts host

Closes #15 and #16. Version 0.3.0.

#16 first, because it has to be. A write to a socket whose peer has gone away
raises SIGPIPE, and a program that has not disarmed it — the default — is
killed rather than told. The fd is one this library created and the write is
usually the close_notify its own pool clean-up sends, so this is the library's
defect. mbedtls guards against it in net_prepare with a process-wide
signal(SIGPIPE, SIG_IGN); replacing mbedtls's network layer with a custom BIO
dropped that guard and put nothing in its place. MSG_NOSIGNAL, and SO_NOSIGPIPE
where that does not exist, is the better replacement anyway: a library has no
business changing its host's signal disposition, and a program that wants
SIGPIPE on its own stdout still gets it. Which of the two applies is decided by
the target's own <sys/socket.h> and by nothing else — measured: glibc and musl
carry MSG_NOSIGNAL and not SO_NOSIGPIPE, Darwin the reverse, Windows neither
and no signal to raise. P0 had to land before everything else here, because
everything else makes the drop path more common.

Then #15. It named two of the eleven paths that could leak a connection; all
eleven are fixed. Four of the other nine turned up while verifying the report —
a redirect, a non-2xx and a failed file open, none of which involve a timeout or
a truncation at all, plus a Content-Length past 32 bits — and two more were
regressions 0.2.10 had introduced into the streaming reader.

Every one of the eleven has the same shape: an early-return path that did
nothing, where doing nothing left a socket with unread bytes in the pool for the
next request to pick up. So dropping is now what doing nothing means — a
PooledConnection guard whose destructor drops, and one call to keep() on the
single path where the body was read to the end its framing declared.

The reason there were eleven rather than one is that send, send_stream and
download_to_file each carried a near-copy of the status-line parse, the header
loop and the body loop, and every past hardening had landed on one or two of the
three. #14 is the most recent example: it added a Content-Length branch to one
copy and introduced two regressions doing it. There is now one status-line
parser, one header reader and one body reader. read_body returns where the body
ended, and that is the same question as whether the connection can be reused.

Two further unbounded loops found while writing that reader, neither reported:
a header block accumulates into a map, so an endless supply of short, well
formed header lines grows the client's memory without limit; a trailer section
is discarded but holds the call open just as long, because every line resets the
read timeout. Both are bounded now, as each line's length already was.

Added, all of it additive: HttpResponse::bodyComplete and bodyError, because a
truncated 200 and a complete 200 were indistinguishable to the caller; ok() does
not consult them, so existing code means what it did. maxResponseBodyBytes,
because the size send() was about to allocate came from a header.
retryOnStaleConnection, because a server closing an idle keep-alive connection
is routine and the client cannot see it until it writes — that failure reached
callers as "No response" for a request the server never saw. The window is one
attempt, on a pooled connection, before a single response byte has arrived.

tests/ had no keep-alive coverage at all, which is why #14's regression merged
green. It now scripts an in-process TLS server — the library speaks only HTTPS,
so a plain listener cannot reach the code under test — and every pool test
asserts how many TCP connections the server saw as well as what came back. The
report for #15 contains a case whose output is byte-for-byte correct and whose
only symptom is the connection count.

Each fix was checked by mutation. Three tests did not survive that and were
rewritten: the SIGPIPE child stopped at its first failed write, which returns
ECONNRESET without a signal; and the pool servers stalled rather than sending
their remaining bytes, so the connection was silent rather than dirty and the
stale-connection retry rescued the second request whether or not the guard
worked.

Two more found by review after the above was written, one of them introduced by
it. Reading past a 1xx is new: a 103 Early Hints or a 100 Continue was returned
to the caller as the answer, and — once a 1xx counted as a response with no body
— the connection was marked clean with the real response still sitting on it,
which is #15 arriving by another door. And the BIO answered a peer's FIN with
MBEDTLS_ERR_NET_CONN_RESET where mbedtls's own passes the zero through, so
`ssl_fetch_input`'s test for exactly that zero (ssl_msg.c:2251) never fired; a
body whose framing IS the close then read as truncated, and download_to_file
reported ok() == false for a file that had arrived complete.

Also here, because all three are about the same claim — that this library works
where it says it does:

examples/openkal builds these sources above openkal, the portable kernel ABI,
and makes a real HTTPS request through kal_net_connect. It is a separate CI job
because whether MSG_NOSIGNAL or SO_NOSIGPIPE exists is decided by the C library
rather than the operating system, and a #ifdef that is wrong about that compiles
cleanly on the gcc job and fails there — which is how 5e7d66f reached master.

templates/ ships three starting points for `mcpp new --template tinyhttps`, one
per entry point. tools/template_smoke.sh renders and builds them against the
working tree, because `mcpp new` can only reach a template that is already
published, and checking them after the release makes the first user the one who
finds out. It has already caught one: import std carries no stdout macro, so a
progress bar flushed through it compiled here and not in a generated project.

mcpp.lock is removed and ignored. The file's own header says it does not pin
anything — "index dependencies are re-resolved from their constraints each
time" — so for a library, whose consumers resolve from its constraints, it
recorded one machine's resolution and changed nothing about anyone else's.
Eight of the ten mcpplibs packages already ignore it, the scaffold template
among them.
@Sunrisepeak
Sunrisepeak force-pushed the fix/pool-invariants-and-sigpipe branch from bdc7017 to 485806a Compare September 5, 2026 19:40
@Sunrisepeak
Sunrisepeak merged commit 40c1a4f into master Sep 5, 2026
2 checks passed
@Sunrisepeak
Sunrisepeak deleted the fix/pool-invariants-and-sigpipe branch September 5, 2026 19:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant