Skip to content

TEST: Separate score cancellation watchdog from attack startup - #3139

Merged
Richard Lundeen (richlundeen) merged 4 commits into
microsoft:mainfrom
richlundeen:richlundeen-score-cancellation-timeout
Oct 12, 2026
Merged

Richard Lundeen (richlundeen) merged 4 commits into
microsoft:mainfrom
richlundeen:richlundeen-score-cancellation-timeout

Conversation

@richlundeen

Copy link
Copy Markdown
Contributor

Description

The merge queue for #3121 failed in an unrelated score cancellation test on macOS / Python 3.13. Failed job log.

The five-second watchdog covered the whole attack, including target setup and conversation persistence. The failed stack was still in PromptNormalizer.add_conversation_to_memory_async, reading ConversationEntry through the aiosqlite worker queue. It had not reached score validation. The log does not show a score rollback or error-result write failure.

Wait up to 30 seconds for the existing validation-ready event, then keep the five-second cleanup deadline. Use the existing non-cancelling completion helper so the watchdog does not add another cancellation to the operation under test. Assert that the held validation read received cancellation and that error propagation did not require releasing the read. Query scores with an explicit type filter; an unfiltered query returns no rows by design and cannot check rollback.

This is a test-only change. The auxiliary-scorer failure, transaction-start event, blocked validation read, exception cause, and persisted attack-error assertions remain in place. No production code or changes from #3121 are included.

Tests and Documentation

Validation on Windows / uv-managed Python 3.12.12:

  • uv run --no-sync pytest tests\unit\executor\attack\test_scoring_expectation_transport.py -q --durations=5 — passed, 41 tests.
  • uv run --no-sync pytest tests\unit\executor\attack\test_scoring_expectation_transport.py tests\unit\memory\test_sqlite_cancellation.py -n 4 --dist=loadfile -q — passed, 78 tests on the final diff.
  • $env:UV_NO_SYNC='1'; uv run --no-sync pre-commit run --all-files — all hooks passed on the final diff.

A temporary pytest plugin outside the repository added a 5.1-second async delay before target sending. The original test failed with TimeoutError before scoring; the revised test passed with the same delay. The revised test does not add this delay to normal CI runs. The original macOS / Python 3.13 environment was not available locally; CI must confirm that platform.

No product documentation changes apply to this test-only fix. JupyText was not run.

Wait for validation readiness before timing cleanup. Assert the validation read was cancelled and query stored scores with an explicit filter.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 226dfd07-b984-4cc9-a0f3-502d3c02d318
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6d66ad46-6cb7-40a5-9290-e6d67dd48ab9
Use a finite shared watchdog for analytics correctness queries while preserving dedicated deadline and cancellation coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6d66ad46-6cb7-40a5-9290-e6d67dd48ab9
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 12, 2026
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 12, 2026
Merged via the queue into microsoft:main with commit ab81c63 Oct 12, 2026
55 checks passed
@richlundeen
Richard Lundeen (richlundeen) deleted the richlundeen-score-cancellation-timeout branch October 12, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants