feat: Add role-aware Scenario target-attempt accounting (#3043)

[Nimit Jain (Nimit3418)]

Description

Implements the role-aware Scenario producer accounting requested in #3043. Latest main is merged, including the shared analytics SDK/calculators from #3059; this PR adds the role projection rather than a second outcome calculator.

• Add typed ScenarioProducerCounts for target-facing, orchestration, and unknown-role attempts, errors, and persisted retries. A target-facing producer result is not proof of a physical target request: preparation failures remain target-facing attempts.
• Keep planned logical progress, latest-attempt outcomes, saved-plan denominators, cached results, and raw history unchanged. Overall producer totals include all saved attempts, including unmatched configurations. Atomic-name producer totals cover every configuration recorded with that name. Configuration-specific progress remains conservative when attribution cannot resolve a planned group.
• Aggregate producer categories in SQL, with exact SQL Server case/length matching and explicit unknown-role handling. Retain all producer counters through the identifier-only legacy SDK fallback.
• Carry shared typed counts through SDK, run detail/history/progress, JSON, and HTML. Renderers do not infer roles or recalculate accounting.
• Cover same-name unplanned configurations, absent hashes with ambiguous names, unmatched seeds, producer errors/retries, clamping, report totals, and unchanged logical progress with real persisted SQLite regressions.

Closes #3043.

No intentional breaking change to existing outcome denominators or planned completion. No hierarchy endpoints, new request tracing, schema migration, or attack/scorer behavior change.

Tests and Documentation

Validated on Windows / Python 3.14.4 after integrating main. The new regressions reproduced the defect before the fix (3 failed, 1 passed), then all four cases passed. Final-head targeted validation:

uv run --frozen --no-sync pytest -q -n 4 --dist=loadfile tests\unit\analytics tests\unit\backend\test_scenario_progress_read_model.py tests\unit\backend\test_scenario_run_service.py tests\unit\memory\memory_interface\test_interface_scenario_history.py tests\unit\memory\memory_interface\test_interface_seed_dataset_summary_query.py tests\unit\memory\test_azure_sql_memory.py tests\unit\models\test_analytics.py tests\unit\models\test_scenario_progress.py tests\unit\models\test_import_boundary.py tests\unit\common\test_lazy_package_imports.py tests\unit\output\scenario_result tests\unit\output\test_derivation.py --timeout=90

Result: 1,113 passed, 1 skipped (existing Azure-only skip).

• uv run --frozen --no-sync pre-commit run --all-files: passed, including formatting, lint, documentation, import/naming checks, migrations, and the configured full production type check. Normal commit hooks also passed without skips.
• uv run --frozen --no-sync ty check pyrit\analytics\scenario_statistics.py pyrit\models\scenario_progress.py pyrit\backend\services\scenario_progress_read_model.py: passed.
• git diff --check: passed.

Updated analytics documentation and model/progress descriptions to distinguish producer populations from logical-unit populations. SQLite tests use isolated databases; Azure SQL coverage is compilation/offline only. No live targets, dataset downloads, notebook execution, or local full-repository test run was performed. Final-head hosted CI is enabled and pending; the targeted results above are not a claim that every hosted check has passed.

JupyText was not run because the effective changeset updates Markdown documentation, not paired notebook/Python examples.

[Nimit Jain (Nimit3418)]

@microsoft-github-policy-service agree

[Nimit Jain (Nimit3418)]

Hi Roman Lutz (@romanlutz), thanks for the guidance and the heads-up on the overlapping changes!

I've updated the PR to coordinate with the shared OutcomeStatistics calculation from #3059 and fully addressed the role-aware accounting constraints you outlined.

Here’s a breakdown of how it was implemented in this update:

1. Integrated with FEAT: Attack analytics SDK #3059's OutcomeStatistics:
   Reused compute_outcome_statistics directly rather than building a custom success-rate calculation.
   We cleanly append producer_counts to ScenarioProgressCounts while leaving the success_rate_decided and success_rate_all logic (and the base denominators) completely untouched.

2. Analytics / SQL Counting Parity:
   Analytics owns the policy: scenario_statistics.py now explicitly loops over attempts and buckets them into target_facing, orchestration, and unknown (tracking attempts, errors, and retries for each). We rely on AttackResultMetadata as the role contract and explicitly preserve UNKNOWN roles rather than discarding them.
   SQL Aggregate Match: I updated _build_scenario_history_aggregate_statement to mirror the SDK's selection logic exactly. The SQL case() statements explicitly filter on is_planned == 1 when tallying producer counts, ensuring we correctly ignore unplanned/legacy attempts just like the SDK does. This guarantees 100% mathematical parity between the database rollups and the in-memory calculations.

3. Preserved Existing Base Metrics:
   As requested, I added separate role-aware rollups without redefining or mutating any existing counts. unit_count, completed_units, successful_units, and the latest-attempt selection logic are all preserved.
   The edge cases work as intended: for example, 2 target-facing child results + 1 Sequential envelope will successfully show as 2 target-facing attempts and 1 orchestration attempt inside producer_counts, while still correctly rolling up as exactly 1 outer planned unit in the core metrics. Preparation failures also cleanly map to their respective producer categories.

4. Testing & Validation:
   Expanded the SQLite parity tests (test_scenario_statistics_parity.py) with role_aware_accounting scenarios to cover mixed-role histories and validate the fallback/unplanned grouping behavior.
   Added Azure SQL query-compilation checks (test_azure_sql_memory.py) to ensure the JsonScalar role-extraction compiles and executes safely.
   Ran the full suite locally—all 24,000+ unit tests are passing cleanly.

Let me know if there's anything else you'd like adjusted. Otherwise, it should be good for another look whenever you have time!

[Nimit Jain (Nimit3418)]

Hi Roman Lutz (@romanlutz),

Thanks for the detailed review! I've addressed all 4 review comments, rebased onto the latest main, and added dedicated regression tests for the reproduction cases.

Here is a summary of the updates:

1. Unmatched attempts in producer accounting (#discussion_r4240306102):

   • Planned-unit filtering remains intact for progress metrics (planned, completed, succeeded), but producer accounting now calculates role totals across all attempts independently (producer_attempts=attempts).
   • Removed the is_planned == 1 predicate from the target_facing_*, orchestration_*, and unknown_role_* SQL rollups in memory_interface.py.
   • Added test_unmatched_attempts_visible_in_producer_accounting reproducing the 4-row scenario (1 planned success, 1 unmatched error, 1 unmatched orchestration, 1 malformed role) and asserting parity across SDK, SQL history aggregate, and service read models.

2. SQL Server string comparisons (#discussion_r4240306104):

   • Added _get_scenario_role_match_condition hook. In AzureSQLMemory, this enforces binary collation (COLLATE Latin1_General_100_BIN2) for case sensitivity and a DATALENGTH guard (datalength(role_expr) == datalength(N'role_val')) to prevent SQL Server's ANSI space-padding behavior from matching strings like "target_facing " or "TARGET_FACING".
   • Added test_scenario_role_match_condition_uses_binary_collation_and_datalength_guard for MSSQL dialect compilation and test_malformed_roles_fall_back_to_unknown_in_producer_accounting verifying that trailing-space and uppercase strings fall back to UNKNOWN.

3. Legacy SDK fallback recounting (#discussion_r4240306106):

   • In ScenarioRunService._recount_with_sdk_statistics_async, copied all 9 producer metrics (target_facing_*, orchestration_*, unknown_role_*) from overall.producer_counts into the recounted aggregate.
   • Added test_ambiguous_objective_with_identifier_recount_updates_producer_fields simulating needs_sdk_statistics=True to confirm producer metrics are preserved upon recount.

4. Scenario reports export (#discussion_r4240306108):

   • Extended GroupStatistics and ScenarioOverview in _derivation.py with typed producer_counts.
   • Serialized producer_counts in JSON reports under both stats and groups.
   • Updated the HTML template with role summary cards (target-facing, orchestration, unknown) and per-group breakdown columns.
   • Verified via test_sdk_api_and_reports_report_identical_statistics and test_html_report_renders_role_aware_accounting.

All linters (ruff check & ruff format) are passing, and the full test suite passes cleanly with 0 failures (25,525 passed). Ready for your re-review whenever you have time!