fix(runner): persist failure checks when server scenarios throw - #538
Open
mohammedmessaoudene-cmd wants to merge 1 commit into
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Author
|
The CI workflow for this PR is awaiting approval ( @rinaldofesta, the focused |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #451 — this addresses only the missing
checks.jsonafter a server scenario exception.Problem
When
scenario.runthrows synchronously or rejects, the server runner exits before writingchecks.json. In suite mode the CLI reports a synthetic failure in memory, but the result directory has no report.Change
FAILUREcheck used by the existing suite fallback, preserve the console diagnostic, then use the existing wire-check and report-writing path.finallywhen the scenario promise rejects.checks.json.Behavior note
Single-scenario exceptions now participate in the existing expected-failures policy, as suite exceptions already do. Without a baseline they still exit with code 1. If explicitly covered by
--expected-failures, they may exit with code 0, while the report retainsFAILURE. Uncovered failures still exit with code 1. This is an intentional normalization, not a claim that all exit behavior is unchanged.Validation
Validated on base
7169291ec0b68eb370fddcd9947313ab0d5e4156, Windows x64 / Node 22.16.0:git diff --checkpassed.tools/listexchange, timeout, skips, suite continuation and baseline handling. The compiled stock CLI also passesserver-initializeagainst both SDKs; a controlled HTTP failure stays red and a genuinely inapplicable revision stays skipped.Scope
These are minimal real SDK servers and controlled runner tests, not full SDK conformance certification. No Go test was run. No new normative scenario, check identifier, traceability manifest or baseline file is changed.
This does not address all of #451, the scenario corrections in #537, or the setup-failure classification in #327. It does not add atomic/crash-safe writing, cancellation, or recovery of checks never returned by a throwing scenario.