fix(providers): draw local sandbox host ports below the ephemeral range, in turn

[earakely-scale]

Summary

A local sandbox fixed each host port it publishes by binding port 0, closing the socket and keeping the number. Docker bound the port later: after the image pull for an agent, or after every image load for a gateway, so seconds to minutes later. In between, two things could take it:

• An outgoing connection. Port 0 draws from the same ephemeral range the OS gives outgoing connections their local ports from. On Docker Desktop and Rancher Desktop it's worse: the probe runs on the host, but Docker binds inside the VM, whose sockets the probe can't see. On a Mac running Rancher Desktop under a load of concurrent deploys, the VM held 1,081 loopback sockets in its ephemeral range (32768–60999), and none in 20000–29999.
• Another deploy that drew the same number, in this process or in another one, since nothing recorded the port until Docker bound it.

The deploy then failed before anything ran:

driver failed programming external connectivity on endpoint agent-local-84d852bf (…): failed to bind host port 127.0.0.1:58876/tcp: address already in use

In the PortSim v5 sweeps (12 models, on that Mac), 66 attempts failed this way, and 7 weeks lost all three of their attempts to it. Every failed port lay in 49532–60994, where the Mac's ephemeral range overlaps the VM's.

The change

_free_host_port() now does two things:

• Draws from 20000–29999. That range is below every default ephemeral range: 32768 up on Linux, including the Docker Desktop and Rancher Desktop VMs, and 49152 up on macOS and Windows. It's also below Kubernetes' node ports, 30000–32767. So by default no outgoing connection holds one, on the host or in a VM.
• Hands the ports out in turn. A cursor file in the per-user state root, locks/host-ports beside agent-env's other lock files, is held under flock while each port is drawn. Every agent-env process that shares the state root (the same XDG_STATE_HOME, or home directory) shares the cursor. It isn't kept in the sandbox root, which a run leaves empty. The bind probe still skips a port in use (EADDRINUSE), and any other bind error raises. No port is handed out again until the whole range has gone round, 10,000 draws later.

create_vm makes its draws in a thread, since a draw waits for any other process's draw to finish. Nothing else changes: create_vm still fixes every port up front, so tunnel_urls, .port-map.json and every publish site are as before.

Why not let Docker pick the port (-p 127.0.0.1::8000, then docker port)

• create_vm has to return tunnel_urls. deploy_sandbox records them in the task context before any container publishes, for run_docker_container and install_agent to use later.
• A host-mode install_agent binds the port itself, with no Docker involved.
• On Linux the same port is published on loopback and on the bridge gateway, so other containers can reach host.docker.internal:<port>. Two ephemeral bindings needn't get the same number.
• It isn't atomic on Docker Desktop or Rancher Desktop. Docker picks inside the VM's ephemeral range, and the host then has to bind that same number, which overlaps the host's own ephemeral range.

No retry on a failed bind

Both causes are gone, so a retry would only be defensive code for a case nothing produces now. The chaos tests below check that nothing is left for it to catch.

Limits

• Processes with different state roots each keep their own cursor. Examples: several agent-env containers driving one Docker daemon, or a test run (which gets its own state root) next to a real run on the same machine. Each starts at a random port, so they rarely meet, but they can. In a container the probe can't see the host's ports either. Processes that share XDG_STATE_HOME share the cursor.
• A Linux host whose net.ipv4.ip_local_port_range reaches below 30000 lets outgoing connections take these ports again. Concurrent deploys still never draw the same one.
• The range holds some well-known service ports, such as 27017 (MongoDB) and 22000 (Syncthing). The probe skips a port a running service holds. A service started while a sandbox holds its port can't bind it until that sandbox ends.

Testing

• Unit tests for the allocator, in local_sandbox_host_ports_test.py:
  • ports come in turn and skip those in use;
  • the cursor wraps round the range;
  • a missing or unreadable cursor starts at a random port;
  • a full range is an error;
  • 8 threads × 50 draws never share a port. This test fails 5 runs out of 5 with the flock removed.
  • the bind probe skips a port in use (EADDRINUSE), and raises on any other bind error;
  • on Linux, with real sockets: a port held on another address (127.0.0.2) is skipped. With the probe narrowed to 127.0.0.1, this test fails.
• On an Ubuntu 24.04 box, the file's 19 tests pass. CI's unit job passes.

Chaos test 1: deploys back to back

It ran on c6i.2xlarge EC2 instances (Ubuntu 24.04, Docker 29.9.0, ephemeral range 32768–60999), 4 minutes a phase. In each phase:

• Deploys: 3 processes × 6 workers share one sandbox root and one state root. Each worker runs LocalSandboxProvider.create_container back to back for a small HTTP image, waits until it serves, holds it 0–3 s, then terminates it.
• Kills: every 40–70 s, one process is killed with SIGKILL mid-deploy and restarted.
• Connection churn: a generator holds 16,000 loopback connections and opens 800 new ones a second, closing the oldest, so the ephemeral range stays busy, as on a loaded machine.
• Squatters: on the native box, listeners hold 300 random ports in 20000–29999, so the allocator has to skip them.

On the split-namespace box, the deploying processes run in containers that drive the host's Docker through its socket. Their probe therefore can't see the host's sockets, as on Docker Desktop and Rancher Desktop.

Box	Build	Deploys	Failed to bind a host port	Ports held by two live deploys at once	Host ports used
native	0.9.1309	770	44 (5.7%)	9	32809–60921
native	aae072e	706	0	0	21784–22515
split namespace	0.9.1309	963	738 (77%)	0	32797–60997
split namespace	aae072e	697	0	0	20000–29999 (the cursor wrapped)
split namespace	beefd98, state root shared	724	0	0	24932–25664

At aae072e the cursor lived in the sandbox root, which every process shared. On the native box every process also shares one home, so the cursor's later move to the state root doesn't change what those rows test. The split-namespace containers each have their own home, so the beefd98 run mounts one state root into all of them.

There were no other failures in any phase: every deploy that bound its port served. Median deploy time was unchanged: 2.42 s → 2.56 s native, 2.80 s → 2.37 s split (2.33 s at beefd98).

Chaos test 2: PortSim v5 weeks

The workload that hit the race: the portsim-lines weeks of agentenv-portsim-plugin v0.5.1. Each episode deploys a gateway env (gateway, pgweb and db-mcp published), a planner agent and 7–9 line agents: 13–15 containers and 11–13 host ports. A stand-in model answers each planner turn with the week's advance, so every week plays to its end at no cost.

It ran on two c6i.8xlarge instances (32 vCPU, 64 GB, Ubuntu 24.04, Docker 29.9.0), one per build, at the same time, for 20 minutes each:

• agent-env portsim sweep run --lines --tasks all --k 60 --parallel 24: 24 attempts at a time, each its own agent-env run process. The sweep retries a failed attempt up to twice.
• Every 40–70 s one running attempt is killed with SIGKILL. The sweep retries it, and its containers stay behind until a janitor removes them after 10 minutes, so the allocator must skip their ports.
• The same connection churn and 300 squatted ports as above.

Build	Attempts	Weeks scored	Failed to bind a host port	Other failures	Killed by the test, or stopped at the end	Most containers at once	Host ports published
0.9.1309	488	7	442 (91%): pgweb 198, db-mcp 181, gateway 45, agents 18	0	17, 22	79	350, none in 20000–29999
beefd98	204	163	0	0	18, 23	301	1,983, all in 20000–29999

• 0.9.1309 fails far more here than on the Mac (about 1 attempt in 10 there). The churn takes most of a port's window, since an env's ports are bound 10–15 s after they're drawn. Its attempts mostly failed within seconds, so that box ran at about a quarter of the other's containers.
• The branch ran 24 attempts and up to 301 containers at once, and none failed to bind. That includes the containers 18 killed attempts left behind (the janitor removed 47 of them), whose ports the allocator had to skip.
• Every scored week ended done. The branch's median week took 128 s under that load; alone, a week takes 48 s.

Follow-up

After this ships, bump the PortSim plugin's framework pin and run its remaining weeks at full parallelism.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new blocking issue remains.

Summary

Local sandboxes now draw host ports from 20000–29999 using a shared, locked cursor. create_vm runs those draws off the event loop.

• Local sandboxes take host ports in turn from a lower range.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[create_vm] --> B[Draw ports in a worker thread]
  B --> C[Lock the shared cursor]
  C --> D[Probe the next port]
  D -->|Busy| E[Try the next port]
  E --> D
  D -->|Free| F[Save cursor and close lock]
  F --> G[Create sandbox and save port map]

Loading

_{Reviews (4) · Last reviewed commit: "fix(providers): draw local sandbox host ..." · Reviewed by Greptile}