[APP-5196] Measure macOS test compile cache behavior - #14797
Closed
kevinyang372 wants to merge 20 commits into
Closed
[APP-5196] Measure macOS test compile cache behavior#14797kevinyang372 wants to merge 20 commits into
kevinyang372 wants to merge 20 commits into
Conversation
`Run MacOS tests` shared a 25m job timeout with Linux and Windows. macOS is compile-bound and had no headroom under that cap (p95 25.1m), so 8.5% of macOS executions were cancelled with "exceeded the maximum execution time of 25m0s" over 2026-07-28 → 2026-08-05 (18/211). Linux (p90 13.8m) and Windows (p90 17.6m) hit 0% under the same cap. Move `timeout-minutes` onto the matrix so macOS gets 40m while Linux and Windows keep 25m, so a genuine hang there is still caught. Co-Authored-By: Warp <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
warp-agent-staging
Bot
added
the
warpy-factory
Phase 0 measured only one valid warm-vs-scratch pair, and target reuse in it was weak (warm target 33.85->62.30 GiB vs scratch 0->28.78 GiB), so most of the 67.13s benefit came from dependency downloads rather than reused build output. Testing that against a dedicated cache volume needs the experiment to be able to select one. Add a `cache_scope` dispatch input that switches the warm arm between the profile's shared cache volume and a dedicated tag via Namespace's documented profile override, plus a `warmup` input that marks a run as an uncounted seeding pass. Both are recorded in metrics.env, the step summary, and the artifact name so a warm-up can be excluded from statistics after the fact. Cache volume *size* is deliberately not set here. `nscloud-cache-size-*` is a runner-label-only companion and Namespace refuses to schedule a job whose `runs-on` carries more than one `nscloud` label, so pairing it with a profile label would stop the job from ever being acquired. Sizing the tagged volume to the 84 GB target is a Namespace dashboard change. Co-Authored-By: Warp <agent@warp.dev>
A dedicated Namespace profile now exists with the same machine spec as mac-ci but an 84 GB cache volume, so the isolated arm selects that profile directly instead of forcing a cache tag onto the production profile. Selecting a whole profile already isolates the cache volume, which makes the `overrides.cache-tag` override and its env redundant, and drops the `cache_tag` diagnostic that only ever described that override — the recorded runner profile now identifies the volume. The env comment claiming volume size is unreachable from the workflow described the old design and is replaced with the current one, keeping the warning that an `nscloud-cache-size-*` label must never sit beside a profile label since Namespace then refuses to schedule the job at all. Co-Authored-By: Warp <agent@warp.dev>
Empty commit to fire a pull_request synchronize event and confirm workflow run creation has recovered. Co-Authored-By: Warp <agent@warp.dev>
`inputs` is null on a pull_request event, so every PR-triggered run resolved to the shared arm and counted itself. The isolated arm and the uncounted warm-up were therefore unreachable, and workflow_dispatch is unavailable while this workflow lives only on a branch. Resolve the arm in a small params job and consume it through `needs`, which `runs-on` can read even though it cannot read `env`. A dispatch still wins when present; otherwise two committed defaults select the arm, so switching arms is a one-line change. The job validates both values and fails loudly on an unrecognized one rather than silently falling back to a measurement that would be attributed to the wrong volume. Routing the profile through one output also removes the duplicated profile literal that previously had to be kept in sync between `runs-on` and the diagnostics input. Defaults are set to the isolated arm as an uncounted warm-up, which is the next step the campaign needs. Co-Authored-By: Warp <agent@warp.dev>
The isolated warm-up seeded the 84 GB volume, so subsequent isolated runs have real cached state to reuse and their numbers are meaningful. Co-Authored-By: Warp <agent@warp.dev>
Pairs the isolated measurement with the production 50 GB volume on an adjacent commit whose Rust source is byte-identical, so the two arms differ only by cache volume. Co-Authored-By: Warp <agent@warp.dev>
Co-Authored-By: Warp <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
Co-Authored-By: Warp Agent <agent@warp.dev>
![warp-agent-staging[bot]](https://avatars.githubusercontent.com/in/2196476?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Overview
This draft adds macOS compile-cache instrumentation and an isolated immutable-seed experiment. The experiment controls are sound, but the PR needs human decisions on its landing scope and the certainty of its reported conclusion.
Concerns
❓ [QUESTION] The four valid pairs support no evidence of a material 84 GB advantage, not a general proof that 84 GB cannot improve performance. Confirm the Conclusion is intentionally limited to the observed campaign or revise it to preserve the Limits section’s stated uncertainty.
Verdict
Checks: CI ✅ (required checks green; experiment treatment failures are intentional/invalid-data signals) · focused Python tests ✅ · format/Clippy ✅ · workspace test compilation n/a (sandbox SIGKILL)
Found: 0 critical, 0 important, 0 suggestions, 2 human decisions
Request changes
Review run
https://oz.staging.warp.dev/runs/019fd487-e1bc-74c4-88b1-07da63204d54
| # p95 was 25.1m, so ~8.5% of macOS runs were cancelled with | ||
| # "exceeded the maximum execution time of 25m0s". 40m puts p95 at | ||
| # ~63% of the cap while still catching a genuine hang. | ||
| timeout_minutes: 40 |
Contributor
There was a problem hiding this comment.
❓ [QUESTION] This branch includes the 40-minute production timeout change from #14753, although the draft says timeout policy is unchanged and calls #14753 separate. Before this draft can be promoted or merged, which landing plan applies: land #14753 then rebase this branch without its stacked commit, merge this PR and close #14753, or extract the selected permanent instrumentation into a clean follow-up?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This draft adds the measurement layer needed to explain and reduce macOS Rust test-compilation variance:
Compile tests;The normal test graph, production runner profile/cache tags, platform coverage, and timeout policy are unchanged. The treatment profiles and cache tags are experiment-only.
Plan: https://staging.warp.dev/drive/notebook/XRPm3vHYEAnqBinHTSejjl
Investigation conversation: https://staging.warp.dev/conversation/265bb837-85df-4179-b93f-c36b035e3a73
Why the original experiment was discarded
The earlier concurrent warm/scratch matrix was invalid for cache-retention attribution: scratch still attached the same Namespace profile volume for non-Rust cache paths and could supersede the warm fork under Namespace's successful-job last-writer-wins semantics. Its fingerprint parser also counted missing fingerprints while labeling them dirty and missed Cargo's actual dirty/stale records.
Those results are not used below.
Refined experiment
Run 31204853640, commit
8fb39ff6bcf62fd28d87157038be8e310ab18c51:warp-macos-tests-final-50gb-v1warp-macos-tests-final-84gb-v1Validate blockjob accepted a block only when both treatments and scratch were comparable. A red treatment job was expected; the validator was the validity signal.All valid arms used tree
ad1eeaba8b654600efb613cfa13d4ed2a4b50d3c, environment8bbc2dd4689b35b89c1f75f8f1f609d67f8ea43187e995926a9753d66252aa0f, Apple M4 Pro / 6 CPUs / 14 GB, macOS 26.3.1 build 25D2128, Xcode 26.3 build 17C529, and Rust/Cargo 1.92.0.Results
Warm-hit performance
Four attempts restored both treatment seeds and passed every validity gate:
CPU medians were 391.97s for 50 GB, 390.93s for 84 GB, and 1738.98s for scratch. Peak RSS medians were 3.84 GiB, 3.83 GiB, and 6.55 GiB respectively.
There is no evidence that 84 GB is faster than 50 GB when the immutable seed restores. The paired elapsed differences range from −23.20s to +13.34s and CPU differences are similarly noisy around zero.
Both warm profiles are substantially faster than scratch when they restore: median Cargo elapsed is approximately 80% lower and CPU time approximately 77.5% lower. Each treatment compiled 78 units per valid block with 241 dirty and 47 stale fingerprint records and no downloads; scratch compiled 1,100 units, had 1,578 missing fingerprints, and downloaded 1,184 dependencies.
Retention reliability
The ten measurement attempts restored:
both, both, 50-only, both, neither, both, 84-only, neither, 84-only, 50-onlyThe 84 GB profile did not improve seed retention. Misses were validated as absent/stale treatment state while the no-volume scratch control remained isolated and all job inputs matched.
Conclusion
Increasing the Namespace cache from 50 GB to 84 GB does not improve warm-cache compile performance or retention in this experiment. Volume capacity is not the primary explanation for the observed run-to-run variance.
The valuable behavior is binary:
Because either profile restored only 60% of the time, switching production CI to 84 GB would add storage without solving the reliability problem. Keep the macOS 40-minute timeout headroom while investigating Namespace generation selection/retention or piloting a deterministic content-addressed compiler cache such as
sccache. The validated low-risk TUI target cleanup remains independent follow-up work.Limits
Linked issue
https://linear.app/warpdotdev/issue/APP-5196/macos-ci-tests-hit-25m-job-timeout-85percent-of-run-macos-tests-jobs
Related timeout-headroom PR: CI: give the macOS tests job timeout headroom (25m → 40m, macOS only) #14753
Screenshots or video are not applicable because this changes CI instrumentation only.
Testing
cargo fmt --all -- --checkPresubmit Clippy commands for the workspace,
warp, andwarp_completer, all with-D warningsactionlint v1.7.7YAML parsing for changed workflows/actions
Shellcheck for all changed inline Bash
7 Python tests covering fingerprint classification, live-log filtering, target inventory, symlinks, and seed-marker exclusion
Three adversarial read-only workflow reviews before campaign launch
Live immutable-seed campaign: one seed attempt plus ten measurement attempts; all artifacts retained under run 31204853640
I have manually tested my changes locally with
./script/run(not applicable to CI-only changes)Agent Mode
CHANGELOG-NONE
Co-Authored-By: Warp Agent agent@warp.dev