fix(sandbox): a VM build or pull that fails, other than in a RUN command, runs once more

[earakely-scale]

Summary

A VM builds an image from its build context with one docker build and pulls one by name with one docker pull. One wrong answer from a registry failed the whole deploy, as in this build on modal_vm:

Step 1/10 : FROM mirror.gcr.io/library/python:3.12-slim@sha256:423ed6ab…
failed to copy: httpReadSeeker: failed open: could not fetch content descriptor sha256:9cbfea84… (application/vnd.oci.image.config.v1+json) from remote: not found

More than 50 other builds of the same image in that run succeeded, so the answer was transient. It's a 404, which dockerd's own download retries don't cover. Since infra images are rebuilt on a VM of another platform, a run from an arm64 Mac makes about five builds per modal_vm deploy, each pulling its base images.

The change

• Build: VmSandbox._build_image runs the docker build once more, 5 s later, unless a RUN command failed. Both builders say so in fixed words: the legacy builder (a VM with no buildx, such as modal_vm's) says "returned a non-zero code", and BuildKit says "did not complete successfully". So a failed fetch of a base image, a COPY --from image or a # syntax= frontend is retried on either builder, and a failing RUN fails once, with its output. Only the build reruns: the context is extracted in its own step first.
• Pull: _pull runs docker pull once more, 5 s later, on any failure. The linux/amd64 fallback for "no matching manifest" is unchanged, and a fallback pull is retried the same way.
• A retry logs one warning naming the image, the sandbox and the last line of output that says something, whether Docker wrote it to stdout or stderr.

Why match RUN failures rather than transient errors: the failure above is a "not found", and a list of transient-looking errors would have treated it as permanent. A broken Dockerfile or a COPY of a missing file wastes one retry, a few seconds.

Testing

• Unit tests: a build whose base image's blob 404s runs once more, on each builder's output, and only the build reruns. A RUN failure on either builder fails once with its message. A pull that fails once succeeds, and one that fails twice raises the second error. The context build's script list and the failed-build and cancelled-pull tests are updated. The retry tests fail on main, checked from a scratch worktree of main.
• The unit tier passes: 7,653 passed, 16 skipped.

Live check

Run on Docker 29.5.3 (macOS arm64, Rancher Desktop), using the real LocalSandboxProvider, load_docker_images and pull_images. A registry:2 sits behind a proxy that answers 404 to the first blob fetch of each image, as the mirror did. Each image is unique and pushed only to the registry, so Docker has to fetch it.

Scenario	main	this branch
Legacy builder (DOCKER_BUILDKIT=0), base image's first blob fetch 404s	3/3 fail with the error above	3/3 build after one retry
docker pull by name, first blob fetch 404s	3/3 fail	3/3 pull after one retry
BuildKit, same 404	builds: BuildKit absorbs it	builds, no retry needed
RUN … && false, on each builder	fails once	fails once, no retry, Docker's message kept

With a 404 on every blob's first fetch, two transient failures in a row, the legacy build fails after its one retry, as intended.

Chaos

15 minutes on each of three randomly picked EC2 VMs: Ubuntu 22.04 arm64 (Docker 29.1.3, no buildx, so the legacy builder) and Ubuntu 24.04 arm64 and amd64 (buildx 0.30.1).

• Each trial was a real LocalSandbox build or pull, run 2–3 at a time.
• The registry proxy served 0–2 planned faults per image: 404, 500, 503, 429, a reset connection or a truncated body, sometimes after a slow response.
• Every 45–100 s, Docker was restarted or a docker build / docker pull was killed.
• A fifth of the trials ran main.

Trials with no disruption	this branch	main
no fault	216/216, none retried	48/48
one fault	393/393	72/86
two faults	128/143	15/30
a RUN that fails	118, each failed once with Docker's message, none retried

• Docker itself absorbs 429s, 500s, 503s and truncated bodies. The retry is what recovers 404s and reset connections: a legacy-builder 404 succeeded 46/46 here and 1/8 on main.
• 35 disruptions (14 Docker restarts, 13 killed builds, 8 killed pulls): 159 of the 165 trials they hit succeeded. Each of the 6 failures had a fault as well as the disruption, two failures in a row.
• Across 1,271 trials, none hung (the longest took 11.7 s), and no build folder, sandbox folder or labelled container was left.
• On a real modal_vm VM (Docker 29.1.3, legacy builder), each of these behaved as expected:
  • a clean build;
  • a RUN failure, which said returned a non-zero code and wasn't retried;
  • a build killed mid-RUN, which recovered with one retry;
  • a pull.

Limits

• One retry: a registry that stays wrong for more than about 5 s still fails the deploy, about 5 s later than before.
• A RUN command that fails on a network blip (a pip install, say) isn't retried.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; the previous warning issue is fixed.

Summary

VM image builds and pulls retry once after five seconds. Builds that fail in a RUN command are not retried.

• VM image builds get another try when Docker fails outside a RUN command.
• VM image pulls get another try when Docker reports an error.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Build or pull image] --> B{Succeeded?}
  B -->|Yes| C[Continue]
  B -->|No| D{Build RUN failed?}
  D -->|Yes| E[Raise error]
  D -->|No| F[Log last nonempty output line]
  F --> G[Wait five seconds]
  G --> H[Retry once]
  H --> I{Succeeded?}
  I -->|Yes| C
  I -->|No| E

Loading

_{Reviews (2) · Last reviewed commit: "fix(sandbox): a retried build's warning ..." · Reviewed by Greptile}