Skip to content

Keep-alive connections are closed without Connection: close, breaking socket reuse (code-server extension install fails; npm ~90x slower) #211

Description

@nakamura3104

Summary

Under boundary, HTTP responses are returned without the upstream Connection: keep-alive header, but the proxy closes the connection after each response. Clients that pool and reuse sockets (Node.js https.Agent({keepAlive:true})) therefore reuse a socket the proxy has already closed, and every reused request fails with socket hang up (ECONNRESET).

The failure is deterministic and alternates: request 1 succeeds (new socket), request 2 fails (reused socket), request 3 succeeds (new socket after the failure), and so on.

This is not allowlist-, DNS-, or TLS-related. The requests are ALLOWed in the audit log; there are zero DENY entries.

Environment

  • boundary v0.10.0, jail type nsjail (default)
  • Container image codercom/enterprise-base:ubuntu (--cap-add=NET_ADMIN --cap-add=SYS_ADMIN)
  • Ubuntu 24.04.3 LTS, kernel 6.8.0, arm64
  • Node v18.19.1 / npm 9.2.0

Reproduction

// repro.js — 5 sequential GETs over one pooled keep-alive socket
const https = require("https");
const agent = new https.Agent({ keepAlive: true, maxSockets: 1 });
const url = process.argv[2] || "https://registry.npmjs.org/express";
(async () => {
  for (let i = 1; i <= 5; i++) {
    await new Promise((resolve) => {
      https.get(url, { agent }, (res) => {
        res.resume();
        res.on("end", () => { console.log(`req${i} OK status=${res.statusCode}`); resolve(); });
      }).on("error", (e) => { console.log(`req${i} ERR ${e.message}`); resolve(); });
    });
  }
  process.exit(0);
})();
$ boundary --allow "domain=registry.npmjs.org" -- node repro.js
req1 OK status=200
req2 ERR socket hang up     <-- reused socket
req3 OK status=200
req4 ERR socket hang up
req5 OK status=200

$ node repro.js             # same command without boundary
req1 OK status=200
req2 OK status=200
req3 OK status=200
req4 OK status=200
req5 OK status=200

Reproduces 100% of the time. The HTTP status code is irrelevant (observed with both 200 and 302 responses).

NODE_DEBUG=http shows the socket being poisoned before the failure:

HTTP: READ on FREE socket - destroying poisoned socket
HTTP: CLIENT socket onClose

Root cause (as far as we could determine)

Comparing response headers for the same URL, with and without boundary:

header direct through boundary
connection keep-alive (absent)
content-length 0 0

The upstream advertises keep-alive, boundary does not pass it through, and the connection is closed after the response — but no Connection: close is sent to the client either. RFC 9112 §9.6 requires a sender that intends to close the connection to signal it. Without that signal the client keeps the socket in its pool and the next write fails.

Impact

Measured in the same jail with the same allowlist:

client result
curl unaffected (new connection per invocation)
Python requests / urllib3 unaffected, even with max_retries=0
Node.js https.Agent({keepAlive:true}) fails on every reused socket
code-server --install-extension <id> always fails with socket hang up (no retry)
npm install express (67 packages, cold cache) succeeds but takes 91s vs ~1s without boundary (~90x slower)

code-server extension installation is unusable, and npm install — which appears as a usage example in boundary --help — is roughly 90x slower because every reused socket has to be retried.

We could not find a flag to change connection handling (checked all options in --help), so this cannot be worked around through configuration.

Expected behaviour

Either of the following would fix it:

  1. Send Connection: close on responses where the proxy closes the connection, or
  2. Pass the upstream Connection: keep-alive through and actually keep the connection open for reuse.

Option 2 is preferable for performance, but option 1 alone makes every affected client correct again.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions