ci: upload playwright artifacts for flaky retries#18345
ci: upload playwright artifacts for flaky retries#18345Hona wants to merge 1 commit intoanomalyco:devfrom
Conversation
Keep retry traces and reports available when Playwright marks a test flaky so the first failed attempt can still be downloaded from CI.
There was a problem hiding this comment.
Pull request overview
Adds Playwright JSON reporting and CI logic to detect flaky retries and upload e2e artifacts not only on failures, but also when retries turn failures into flaky passes—making flakes visible/downloadable from the same workflow run artifacts.
Changes:
- Add Playwright JSON reporter output (
results.json) alongside the existing HTML + line reporters. - Add a CI step to parse the JSON report and detect flaky tests.
- Upload Playwright artifacts when the job fails or when flaky retries are detected.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
packages/app/playwright.config.ts |
Adds JSON reporter output to enable CI-side flaky detection. |
.github/workflows/test.yml |
Detects flakes from the JSON report and uploads Playwright artifacts on failure or flake. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const report = JSON.parse(fs.readFileSync(file, "utf8")) | ||
| const count = Number(report?.stats?.flaky ?? report?.suitesSummary?.flaky ?? 0) | ||
| fs.appendFileSync(process.env.GITHUB_OUTPUT, `flaky=${count > 0}\nflaky_count=${count}\n`) |
There was a problem hiding this comment.
The flake-detection script will throw and fail the workflow if results.json exists but is malformed/partial (e.g., Playwright process interrupted, file truncated). Since this step runs if: always(), a parse error could incorrectly turn an otherwise-successful run into a failed job. Consider wrapping the JSON read/parse in try/catch and defaulting outputs to flaky=false (or emitting a separate flaky_parse_error output) instead of letting the step fail.
| const report = JSON.parse(fs.readFileSync(file, "utf8")) | |
| const count = Number(report?.stats?.flaky ?? report?.suitesSummary?.flaky ?? 0) | |
| fs.appendFileSync(process.env.GITHUB_OUTPUT, `flaky=${count > 0}\nflaky_count=${count}\n`) | |
| try { | |
| const raw = fs.readFileSync(file, "utf8") | |
| const report = JSON.parse(raw) | |
| const count = Number(report?.stats?.flaky ?? report?.suitesSummary?.flaky ?? 0) | |
| fs.appendFileSync( | |
| process.env.GITHUB_OUTPUT, | |
| `flaky=${count > 0}\nflaky_count=${count}\n` | |
| ) | |
| } catch (error) { | |
| console.error("Failed to read or parse Playwright results.json for flake detection:", error) | |
| fs.appendFileSync(process.env.GITHUB_OUTPUT, "flaky=false\nflaky_count=0\n") | |
| process.exit(0) | |
| } |
Summary
Testing