Repository navigation
TEST: Assert capability probe outcomes - #3279
Merged
Roman Lutz (romanlutz) merged 2 commits intoOct 18, 2026
Merged
Roman Lutz (romanlutz) merged 2 commits into
Roman Lutz (romanlutz) merged 2 commits into
Conversation
Wait for the mocked request before expiring the timeout case, keep a finite correctness watchdog for other outcomes, and verify timeout cancellation and capability results. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f6823ad6-881a-4bbe-bf55-7b4ccc04a13a
Preserve the watchdog, request-entry synchronization and task cleanup merged in microsoft#3268. Keep only the returned JSON capability assertions as the remaining delta. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e52c81e0-1386-4099-9cfc-b52c7632320f
Roman Lutz (romanlutz)
changed the title
Contributor
|
Asserting the returned TargetCapabilities instance directly closes an important verification gap in these discovery tests. Previously the test verified that the underlying client call was attempted and cancelled on timeout, but never asserted that the returned capability flags actually reflected the failure or success state. Returning the capabilities object from probe_async and asserting supports_json_output ensures the probe outcome contract is explicitly validated. |
Roman Lutz (romanlutz)
approved these changes
Oct 18, 2026
Contributor
|
I merged in the latest changes in main which reduced this changeset considerably. Updated title and description accordingly |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Merged current
main(345df8fd0233467273d86fd89dbecea8ff02b274) and kept only the additional capability-result assertions now that #3268 has merged.#3268 already fixed the original capability-probe test timing issue. This PR preserves its shared
TEST_WATCHDOG_SECONDS, request-entry synchronization, real timeout cancellation, and owned-task cleanup unchanged.The remaining one-file change makes
test_responses_probes_suppress_provider_io_and_preserve_statecheck the returnedTargetCapabilities, rather than discarding it:supports_json_output=True.supports_json_output=False.These assertions run across all four combinations of initialized tool state and cached target identity. Existing cancellation propagation, provider-I/O suppression, tool state, configuration, and identity assertions remain intact.
This is additional result coverage, not another fix for the timing issue already addressed by #3268. No production behavior, deadlines, dependencies, or public APIs change.
Tests and Documentation
Validation of the merged, narrowed change on Windows with Python 3.14.4, using this worktree's uv environment with the
litellmextra:uv run --no-sync python -m pytest -q -p no:cacheprovider --tb=short tests\unit\prompt_target\test_discover_target_capabilities.py: 92 passed. Includes all 16 provider-state variants and the real short-deadline retry test.uv run --no-sync ruff check --no-cache tests\unit\prompt_target\test_discover_target_capabilities.py: passed.uv run --no-sync ruff format --check --no-cache tests\unit\prompt_target\test_discover_target_capabilities.py: passed, file already formatted.uv run --no-sync ty check --output-format concise tests\unit\prompt_target\test_discover_target_capabilities.py: passed.git diff --cached --check: passed before committing.LITELLM_LOCAL_MODEL_COST_MAP=Truewas set for the local test process. Tests use isolated SQLite and mocked provider boundaries, with no live providers or credentials.No documentation or notebooks changed, so JupyText is not applicable. The full unit suite, all-extras environment, and macOS execution were not run for this update. Earlier CI results do not validate this new merge.