Skip to content

feat: aggregate expect.soft() failures per test#312

Open
Skn0tt wants to merge 3 commits into
microsoft:mainfrom
Skn0tt:feat/soft-assertions
Open

feat: aggregate expect.soft() failures per test#312
Skn0tt wants to merge 3 commits into
microsoft:mainfrom
Skn0tt:feat/soft-assertions

Conversation

@Skn0tt
Copy link
Copy Markdown
Member

@Skn0tt Skn0tt commented May 4, 2026

Summary

  • Moves soft assertion collection from fixture teardown to test call phase using pytest_runtest_call hook.
  • At end of test: 0 failures → noop, 1 failure → re-raise as FAILED, ≥2 failures → raise ExceptionGroup as FAILED.
  • Eliminates spurious "passed" entries when soft assertions fail—the test is correctly reported as FAILED, not passed+failed.
  • Requires playwright-python with the soft-assertions hook (feat(assertions): add expect.soft() for collecting multiple failures playwright-python#3065).
  • Backward-compatible with older Playwright versions that lack soft-assertions support.

Companion PR (core hook): microsoft/playwright-python#3065

Example output

When running tests with one single soft failure, multiple soft failures, one passing case, and one soft+body failure case:

  • Single soft failure is reported as FAILED with the assertion details (e.g. expected 'goodbye', actual 'hello').
  • Multiple soft failures are reported as FAILED with an ExceptionGroup containing each soft failure.
  • Matching soft assertions pass cleanly.
  • A body exception plus a soft assertion failure reports both in a grouped ExceptionGroup as FAILED.
FAILED tests/test_soft.py::test_soft_plus_body_failure - ExceptionGroup: Test and soft assertion failures (2 sub-exceptions)
FAILED tests/test_soft.py::test_single_soft_failure - AssertionError: Locator expected to have text 'goodbye'
FAILED tests/test_soft.py::test_multiple_soft_failures - ExceptionGroup: Soft assertion failures (2 sub-exceptions)
PASSED tests/test_soft.py::test_soft_passes

Adds an autouse function-scoped fixture in both pytest-playwright and
pytest-playwright-asyncio that wraps each test in a soft-assertion
collection scope. At end of test, collected failures are re-raised:
zero → noop, one → re-raise, multiple → BaseExceptionGroup.

Requires playwright-python with the soft-assertions hook
(microsoft/playwright-python#1272).
return
if len(errors) == 1:
raise errors[0]
raise _BaseExceptionGroup("Soft assertion failures", errors)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder what happens if the test has a soft failure followed by the hard failure? Are we going to see both errors?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, see test_soft_assertion_does_not_shadow_body_failure

Copy link
Copy Markdown

@dgozman dgozman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good. It would be nice to include pytest output in the description, to see how it looks like for the user.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants