Skip to content

ci: upload playwright artifacts for flaky retries#18345

Open
Hona wants to merge 1 commit intoanomalyco:devfrom
Hona:chore/upload-flakey-tests
Open

ci: upload playwright artifacts for flaky retries#18345
Hona wants to merge 1 commit intoanomalyco:devfrom
Hona:chore/upload-flakey-tests

Conversation

@Hona
Copy link
Member

@Hona Hona commented Mar 20, 2026

Summary

  • add a Playwright JSON report so CI can detect when retries converted a failure into a flaky pass
  • upload Playwright traces, test results, and the HTML report when the e2e job fails or when flaky retries are detected
  • keep artifact retention and paths unchanged so failed retry attempts stay downloadable from the same workflow run

Testing

  • bun typecheck

Keep retry traces and reports available when Playwright marks a test flaky so the first failed attempt can still be downloaded from CI.
@Hona Hona requested a review from adamdotdevin as a code owner March 20, 2026 05:28
Copilot AI review requested due to automatic review settings March 20, 2026 05:28
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Playwright JSON reporting and CI logic to detect flaky retries and upload e2e artifacts not only on failures, but also when retries turn failures into flaky passes—making flakes visible/downloadable from the same workflow run artifacts.

Changes:

  • Add Playwright JSON reporter output (results.json) alongside the existing HTML + line reporters.
  • Add a CI step to parse the JSON report and detect flaky tests.
  • Upload Playwright artifacts when the job fails or when flaky retries are detected.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
packages/app/playwright.config.ts Adds JSON reporter output to enable CI-side flaky detection.
.github/workflows/test.yml Detects flakes from the JSON report and uploads Playwright artifacts on failure or flake.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +118 to +120
const report = JSON.parse(fs.readFileSync(file, "utf8"))
const count = Number(report?.stats?.flaky ?? report?.suitesSummary?.flaky ?? 0)
fs.appendFileSync(process.env.GITHUB_OUTPUT, `flaky=${count > 0}\nflaky_count=${count}\n`)
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flake-detection script will throw and fail the workflow if results.json exists but is malformed/partial (e.g., Playwright process interrupted, file truncated). Since this step runs if: always(), a parse error could incorrectly turn an otherwise-successful run into a failed job. Consider wrapping the JSON read/parse in try/catch and defaulting outputs to flaky=false (or emitting a separate flaky_parse_error output) instead of letting the step fail.

Suggested change
const report = JSON.parse(fs.readFileSync(file, "utf8"))
const count = Number(report?.stats?.flaky ?? report?.suitesSummary?.flaky ?? 0)
fs.appendFileSync(process.env.GITHUB_OUTPUT, `flaky=${count > 0}\nflaky_count=${count}\n`)
try {
const raw = fs.readFileSync(file, "utf8")
const report = JSON.parse(raw)
const count = Number(report?.stats?.flaky ?? report?.suitesSummary?.flaky ?? 0)
fs.appendFileSync(
process.env.GITHUB_OUTPUT,
`flaky=${count > 0}\nflaky_count=${count}\n`
)
} catch (error) {
console.error("Failed to read or parse Playwright results.json for flake detection:", error)
fs.appendFileSync(process.env.GITHUB_OUTPUT, "flaky=false\nflaky_count=0\n")
process.exit(0)
}

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants