Skip to content

CI: give the macOS tests job timeout headroom (25m → 40m, macOS only) - #14753

Closed
warp-agent-staging[bot] wants to merge 1 commit into
masterfrom
factory/macos-ci-timeout-headroom
Closed

CI: give the macOS tests job timeout headroom (25m → 40m, macOS only)#14753
warp-agent-staging[bot] wants to merge 1 commit into
masterfrom
factory/macos-ci-timeout-headroom

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Description

Run MacOS tests in .github/workflows/ci.yml hit the job-level timeout-minutes: 25 ceiling on 18/211 = 8.5% of executions over 2026-07-28 → 2026-08-05, while Run Linux tests and Run Windows tests hit it 0/211 = 0% under the same shared cap.

GitHub does not surface these as conclusion timed_out — they land as cancelled with the check-run annotation "The job has exceeded the maximum execution time of 25m0s", which is why they read as generic CI flake.

Root cause: macOS simply has no headroom under a 25m cap.

OS p50 p90 p95 job-timeout rate
macOS 19.6m 24.5m 25.1m 8.5%
Linux 13.2m 13.8m 0%
Windows 16.1m 17.6m 0%

The long pole is the Compile tests step (cargo nextest --no-run): p50 8.7m / p90 12.4m / max 15.5m. It is not one hung test, and it is not the nextest 60s per-test slow-timeout tracked separately in CODE-1748.

Change: move timeout-minutes from the tests job onto the matrix entries so it can be scoped per OS — macOS gets 40m, Linux and Windows keep 25m. Raising only macOS means a genuine hang on Linux/Windows is still caught by the tighter cap.

At 40m the measured macOS p95 (25.1m) sits at ~63% of the cap, satisfying the ticket's "p95 ≤ ~80% of the cap" criterion.

This PR also drops a dangling comment at the end of the tests job that described the old single job timeout; the rationale now lives inline on each matrix entry.

Before / after expectation (re-measurable): Run MacOS tests job-timeout rate 8.5% → ~0%, with Linux/Windows staying at 0%. Re-measure by sampling recent Warp CI runs and counting Run MacOS tests jobs whose conclusion is cancelled at the configured cap.

Out of scope (follow-up)

  • Cutting macOS Compile tests wall time (ticket Solution item 2). Deliberately not attempted here. On namespace-profile-mac-ci the workflow already skips Swatinem/rust-cache and uses namespacelabs/nscloud-cache-action with the persistent rust + brew caches, and Populate Build Cache re-runs this same workflow on master. Without per-run cache hit-rate data there is no concrete defect to fix, so changing that path would be speculative CI restructuring rather than a contained improvement.
  • Splitting the macOS suite into separate compile / test-shard jobs (ticket Solution item 3). Not done — it is a structural change to the job graph and required checks, and it is not needed to close the timeout rate. Worth a follow-up ticket if macOS compile time keeps growing, since the 40m cap buys headroom rather than making the job faster.

Linked Issue

APP-5196

Testing

Testing-exempt category: config-only. This changes only CI workflow configuration. A regression test could assert nothing beyond the literal YAML value it would be reading back, so it would detect no logic defect and would break on any future retune. Per the ticket's own testing note, validation here is a config check plus a green macOS CI run, not an in-repo unit test.

What was validated instead:

  • Workflow lints clean. actionlint v1.7.7 on .github/workflows/ci.yml reports the same 8 findings before and after this change — all pre-existing unknown runner label warnings for the repo's custom runners (ubuntu-latest-large, windows-latest-large), none introduced here. In particular timeout-minutes: ${{ matrix.timeout_minutes }} validates: matrix is an allowed context for job-level timeout-minutes.
  • The matrix expression resolves on a real run. Warp CI run 31057557339 on this PR is fully green, including all three matrix entries: Run MacOS tests 17m46s, Run Linux tests 15m25s, Run Windows tests 15m17s. A timeout-minutes expression that failed to resolve to a valid integer is a workflow validation error that would have prevented the tests job from starting at all, so all three entries running proves the per-OS values took effect (40 / 25 / 25).
  • No timeout annotation. Checked the annotation stream on all three test check-runs for this PR's head SHA: none contains "exceeded the maximum execution time". The only annotations present are pre-existing notices unrelated to this change (Node 20 deprecation, a tags input warning from trunk-io/analytics-uploader, Homebrew tap-trust notices, shell version notices).
  • Caveat, stated plainly: this run's macOS job finished at 17m46s, comfortably under the old 25m cap, so it does not by itself exercise the new 40m ceiling. The ceiling only matters on the slow tail, which is why the durable check is the re-measured rate below rather than any single run.
  • .config/nextest.toml is untouched — the 60s per-test slow-timeout is unchanged, as intended.

This is a headless CI-configuration change with no user-visible surface, so no UI verification applies.

  • Manual testing not applicable — the change is exercised by this PR's own CI run.

Agent Mode

  • Warp Agent Mode - This PR was created via Warp's AI Agent Mode

Originating thread: https://warpdev.slack.com/archives/C0BDQDW8V5E/p1785969478020779

`Run MacOS tests` shared a 25m job timeout with Linux and Windows. macOS
is compile-bound and had no headroom under that cap (p95 25.1m), so 8.5%
of macOS executions were cancelled with "exceeded the maximum execution
time of 25m0s" over 2026-07-28 → 2026-08-05 (18/211). Linux (p90 13.8m)
and Windows (p90 17.6m) hit 0% under the same cap.

Move `timeout-minutes` onto the matrix so macOS gets 40m while Linux and
Windows keep 25m, so a genuine hang there is still caught.

Co-Authored-By: Warp <agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Aug 5, 2026
@warp-agent-staging warp-agent-staging Bot added the warpy-factory Label associated to the "Wob the Wuilder" factory on staging, also known as Warpy Factory. label Aug 5, 2026
@warp-agent-staging
warp-agent-staging Bot marked this pull request as ready for review August 5, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed warpy-factory Label associated to the "Wob the Wuilder" factory on staging, also known as Warpy Factory.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants