Make the CI unit test timeout fire before the job timeout does - #72259
Merged
potiuk merged 2 commits intoAug 30, 2026
Conversation
boring-cyborg
Bot
added
area:dev-tools
backport-to-v3-3-test
rjgoyln
changed the title
rjgoyln
force-pushed
the
fix/ci-total-test-timeout-job-budget
branch
from
August 29, 2026 17:29
c1a7c44 to
5242a0a
Compare
Member
|
Nice idea |
The test timeout exists so that a hanging test group is stopped while the job still has time to dump the container logs and upload them as artifacts. It never got that chance. The alarm is armed in the last step of the job, but the job budget starts at the first one, and checking out the repo, pulling the CI image and running the migration tests routinely take twenty minutes or more before any test runs. A fixed sixty minute test timeout could therefore only fire in a job GitHub had already cancelled, so every hang surfaced as a bare job timeout with no logs to explain it.
rjgoyln
force-pushed
the
fix/ci-total-test-timeout-job-budget
branch
from
August 30, 2026 05:48
5242a0a to
4a83dd9
Compare
rjgoyln
marked this pull request as ready for review
August 30, 2026 06:39
rjgoyln
requested review from
amoghrajesh,
ashb,
bugraoz93,
gopidesupavan,
jason810496,
jscheffl and
potiuk
as code owners
August 30, 2026 06:39
potiuk
reviewed
Aug 30, 2026
The line reported the budget left in seconds next to a job timeout in minutes, so anyone reading it had to convert one to compare them.
Contributor
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 14397bc v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
1 task
potiuk
added a commit
that referenced
this pull request
Aug 30, 2026
… does (#72259) (#72299) * Make the CI unit test timeout fire before the job timeout does The test timeout exists so that a hanging test group is stopped while the job still has time to dump the container logs and upload them as artifacts. It never got that chance. The alarm is armed in the last step of the job, but the job budget starts at the first one, and checking out the repo, pulling the CI image and running the migration tests routinely take twenty minutes or more before any test runs. A fixed sixty minute test timeout could therefore only fire in a job GitHub had already cancelled, so every hang surfaced as a bare job timeout with no logs to explain it. * Print the CI test budget in minutes and seconds The line reported the budget left in seconds next to a job timeout in minutes, so anyone reading it had to convert one to compare them. (cherry picked from commit 14397bc) Co-authored-by: rjgoyln <151457491+rjgoyln@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The parallel test timeout exists so that a hanging test group is stopped while the job can still dump the container logs and upload them as artifacts. It has never managed that: the alarm is armed in the job's last step, while the 65 minute budget starts at the first. Restoring the CI image takes six to eight minutes and the migration tests another four to nineteen, so a fixed 60 minute test timeout could only fire in a job GitHub had already cancelled — every hang surfaced as a bare job timeout, with no logs and no sign of which test group was stuck.
Deriving the timeout from what is left of the budget, rather than raising the budget, restores the guarantee
additional-ci-image-checks.ymlgets frombreeze shell --max-time; that one holds only because nothing of variable length runs before it.timeout-minutesbecomes aworkflow_callinput that no caller has to pass: job-leveltimeout-minutescannot readenv, so an input is the only way for it and the tests to share one number instead of two that can drift.Tests
On this PR's own run the unit test jobs were given between 2707s and 3271s, depending on how much the image restore and the migration tests had already spent — every one of them under the 3600s the timeout was previously pinned at.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines