Summary
The sandbox guest rootfs ships nameserver 1.1.1.1 in /etc/resolv.conf, but the guest is told to upload generated files to a Docker Compose service name (EGRESS_GATEWAY_URL=http://egress_gateway:3190). A public resolver can never resolve a Compose service name, so whenever an actual DNS lookup happens the upload is never attempted and every generated file is silently discarded.
The failure is invisible to the caller: code executes fine, the API returns 200, and files is simply [].
{"session_id":"1XmnEa4VgxkSwILdFNCl0","files":[],"stdout":"","stderr":"","code":0,"signal":null,"message":null,"status":null,"wall_time":97}Because the response looks valid, the calling model typically invents an explanation for the missing file — in our case "/mnt/data has insufficient disk space", with no ENOSPC anywhere and ~100 GB free.
Why it usually seems to work
launcher/entrypoint.sh resolves the service names to IPs once at container start:
resolve_url EGRESS_GATEWAY_URL
resolve_url FILE_SERVER_URL
resolve_host_port SANDBOX_FORWARD_TARGET
[entrypoint] EGRESS_GATEWAY_URL: egress_gateway -> 172.18.0.7
[entrypoint] SANDBOX_FORWARD_TARGET: egress_gateway -> 172.18.0.7
So in the normal case the guest is handed a literal IP and never performs a lookup — the broken resolver is latent. This works right up until the resolved address is wrong or a lookup does occur.
Two ways it breaks
1. Stale pinned address after a restart. resolve_url runs once per container lifetime. Compose assigns 172.18.0.x in start order, which varies between runs, so restarting the stack can move egress_gateway while sandbox-runner keeps the address it captured at its start. The runner then uploads to whatever container now holds the old address. Observed on our deployment:
[entrypoint] EGRESS_GATEWAY_URL: egress_gateway -> 172.18.0.5 # runner pinned .5
[entrypoint] EGRESS_GATEWAY_URL: egress_gateway -> 172.18.0.7 # egress_gateway actually moved to .7
At that moment api held 172.18.0.5, so uploads hit api:3190, where nothing listens:
{"level":50,"job":"sbx_FPxRIAHONdoHa_1aKhvG9Q","file":"ci-healthcheck-roundtrip.txt",
"err":{"type":"Error","message":"Unable to connect. Is the computer able to access the url?",
"code":"ConnectionRefused","path":"http://egress_gateway:3190/sessions/<grant>/objects/<id>"},
"msg":"Error uploading file"}
{"level":40,"dropped":1,"kept":0,"msg":"Pruned files from response because upload did not reach file_server"}Note the error is ConnectionRefused, not a DNS error — it connected to a real but wrong container.
2. Actual lookup falls through to 1.1.1.1. When anything does force a resolution of the name, the query goes to Cloudflare and returns NXDOMAIN, and no TCP connection is attempted at all. Captured with tcpdump on the bridge, same host, two runs:
broken: A?/AAAA? egress_gateway. -> 1.1.1.1 -> NXDomain x2, ZERO SYNs to 3190
working: zero DNS queries, SYN -> SYN-ACK straight to egress_gateway:3190
Both modes share one root cause: the guest cannot resolve the name it is given, so correctness depends entirely on the entrypoint's start-time snapshot being and remaining correct.
Impact
Every generated artifact — charts, .xlsx, .pptx — is dropped. docker restart sandbox-runner appears to fix it (it re-runs resolve_url and re-pins a correct address), which makes this look transient and reappear later. For us it recurred on every stack restart, since our nightly maintenance restarts the stack.
Reproduction
- Bring up the compose stack with KVM/microVM sandboxing.
- Execute code that writes a file to
/mnt/data.
- Restart the stack so
egress_gateway is assigned a different 172.18.0.x than sandbox-runner captured at start (docker compose restart, or restart containers in a different order).
- Execute again. The response is
200 with files: []; docker logs sandbox-runner | grep -E 'Uploaded file|Pruned files' shows "dropped":N,"kept":0.
Environment
code-interpreter at 4b72e9d01654959b5cadd58f430242a5e4c57517; also verified against current main (297fead)- Baked/KVM path (
sandbox-runner-baked, LAUNCHER_ROOT_DISK_READ_ONLY=true)
- CentOS Stream 9 host, Docker CE, self-hosted single-node compose deployment
Confirmed directly in the built rootfs image:
$ docker exec sandbox-runner grep -aoE 'nameserver [0-9.]+' /sandbox-rootfs.img | sort | uniq -c
1 nameserver 1.0.0.1
1 nameserver 1.1.1.1
1 nameserver 10.0.0.1
There is no resolv.conf or nameserver handling anywhere in the repo, so the guest inherits this from its base image and nothing overrides it.
Suggested fix
Point the guest at Docker's embedded DNS (127.0.0.11) instead of a public resolver. It is reachable from the guest because libkrun's TSI creates the guest's sockets inside the sandbox-runner container's netns — which is exactly why the 1.1.1.1 queries above appear with the container's source address. It resolves Compose service names and forwards everything else to the host's resolvers, so the sandbox keeps working external DNS.
This has to be written at image build time: the guest root disk is mounted read-only (LAUNCHER_ROOT_DISK_READ_ONLY=true, LAUNCHER_ROOT_OPTIONS=ro), so a boot-time entrypoint cannot rewrite it. We tried that first and the guest logged WARNING: could not write /etc/resolv.conf and kept 1.1.1.1.
Applied to both stages that assemble a rootfs, so the baked and non-baked paths cannot drift:
-RUN sed -i '/^cgroup_mem_swap_max/d' /sandbox-rootfs/sandbox_api/config/sandbox.cfg
+RUN sed -i '/^cgroup_mem_swap_max/d' /sandbox-rootfs/sandbox_api/config/sandbox.cfg \
+ && printf '%s\n' 'nameserver 127.0.0.11' > /sandbox-rootfs/etc/resolv.conf
We have been running this in staging since 2026-08-11 and in production since 2026-08-26; file round-trips pass consistently across restarts and reboots, where previously they broke on most stack restarts.
Two alternatives, both of which we considered worse: pinning a static IP for egress_gateway in compose only papers over mode 1 and requires converting the implicit network to explicit IPAM; re-resolving the name per upload instead of once at start would fix staleness but still needs a resolver that can answer.
Happy to open a PR if the approach looks right.
Summary
The sandbox guest rootfs ships
nameserver 1.1.1.1in/etc/resolv.conf, but the guest is told to upload generated files to a Docker Compose service name (EGRESS_GATEWAY_URL=http://egress_gateway:3190). A public resolver can never resolve a Compose service name, so whenever an actual DNS lookup happens the upload is never attempted and every generated file is silently discarded.The failure is invisible to the caller: code executes fine, the API returns
200, andfilesis simply[].{"session_id":"1XmnEa4VgxkSwILdFNCl0","files":[],"stdout":"","stderr":"","code":0,"signal":null,"message":null,"status":null,"wall_time":97}Because the response looks valid, the calling model typically invents an explanation for the missing file — in our case "/mnt/data has insufficient disk space", with no ENOSPC anywhere and ~100 GB free.
Why it usually seems to work
launcher/entrypoint.shresolves the service names to IPs once at container start:So in the normal case the guest is handed a literal IP and never performs a lookup — the broken resolver is latent. This works right up until the resolved address is wrong or a lookup does occur.
Two ways it breaks
1. Stale pinned address after a restart.
resolve_urlruns once per container lifetime. Compose assigns172.18.0.xin start order, which varies between runs, so restarting the stack can moveegress_gatewaywhilesandbox-runnerkeeps the address it captured at its start. The runner then uploads to whatever container now holds the old address. Observed on our deployment:At that moment
apiheld172.18.0.5, so uploads hitapi:3190, where nothing listens:{"level":50,"job":"sbx_FPxRIAHONdoHa_1aKhvG9Q","file":"ci-healthcheck-roundtrip.txt", "err":{"type":"Error","message":"Unable to connect. Is the computer able to access the url?", "code":"ConnectionRefused","path":"http://egress_gateway:3190/sessions/<grant>/objects/<id>"}, "msg":"Error uploading file"} {"level":40,"dropped":1,"kept":0,"msg":"Pruned files from response because upload did not reach file_server"}Note the error is
ConnectionRefused, not a DNS error — it connected to a real but wrong container.2. Actual lookup falls through to 1.1.1.1. When anything does force a resolution of the name, the query goes to Cloudflare and returns NXDOMAIN, and no TCP connection is attempted at all. Captured with tcpdump on the bridge, same host, two runs:
Both modes share one root cause: the guest cannot resolve the name it is given, so correctness depends entirely on the entrypoint's start-time snapshot being and remaining correct.
Impact
Every generated artifact — charts,
.xlsx,.pptx— is dropped.docker restart sandbox-runnerappears to fix it (it re-runsresolve_urland re-pins a correct address), which makes this look transient and reappear later. For us it recurred on every stack restart, since our nightly maintenance restarts the stack.Reproduction
/mnt/data.egress_gatewayis assigned a different172.18.0.xthansandbox-runnercaptured at start (docker compose restart, or restart containers in a different order).200withfiles: [];docker logs sandbox-runner | grep -E 'Uploaded file|Pruned files'shows"dropped":N,"kept":0.Environment
code-interpreterat4b72e9d01654959b5cadd58f430242a5e4c57517; also verified against currentmain(297fead)sandbox-runner-baked,LAUNCHER_ROOT_DISK_READ_ONLY=true)Confirmed directly in the built rootfs image:
There is no
resolv.confornameserverhandling anywhere in the repo, so the guest inherits this from its base image and nothing overrides it.Suggested fix
Point the guest at Docker's embedded DNS (
127.0.0.11) instead of a public resolver. It is reachable from the guest because libkrun's TSI creates the guest's sockets inside the sandbox-runner container's netns — which is exactly why the1.1.1.1queries above appear with the container's source address. It resolves Compose service names and forwards everything else to the host's resolvers, so the sandbox keeps working external DNS.This has to be written at image build time: the guest root disk is mounted read-only (
LAUNCHER_ROOT_DISK_READ_ONLY=true,LAUNCHER_ROOT_OPTIONS=ro), so a boot-time entrypoint cannot rewrite it. We tried that first and the guest loggedWARNING: could not write /etc/resolv.confand kept1.1.1.1.Applied to both stages that assemble a rootfs, so the baked and non-baked paths cannot drift:
We have been running this in staging since 2026-08-11 and in production since 2026-08-26; file round-trips pass consistently across restarts and reboots, where previously they broke on most stack restarts.
Two alternatives, both of which we considered worse: pinning a static IP for
egress_gatewayin compose only papers over mode 1 and requires converting the implicit network to explicit IPAM; re-resolving the name per upload instead of once at start would fix staleness but still needs a resolver that can answer.Happy to open a PR if the approach looks right.