FEAT: Chain-of-Thought Rendering for Attack Results and Scenario Results

[ValbuenaVC]

Description

Adds opt-in rendering of OpenAI Responses API reasoning summaries across PyRIT’s output layer.

This PR:

• Adds include_reasoning_trace=False to conversation, attack-result, and scenario-result output APIs.
• Renders provider-generated reasoning summaries while keeping them hidden by default.
• Validates persisted reasoning pieces against the OpenAI Responses reasoning-item shape.
• Prevents reasoning content from leaking after conversion to another data type.
• Renders literal <reasoning-summary> tags in subdued gray for Pretty output.
• Renders escaped \<reasoning-summary\> tags in Markdown.
• Supports objective, pruned, and adversarial attack conversations.
• Allows scenario output to expand contained attack results and show their reasoning summaries.
• Clearly labels summaries as provider-generated summaries, not raw hidden chain-of-thought.

No breaking behavior is introduced because reasoning remains opt-in.

Background and Scope

The original story also requested investigation into JSON Schema adoption for scorers and attacks.

That infrastructure and adoption already landed through:

• [FEAT] Support JSON Schema in Responses #1177 — OpenAI Chat/Responses JSON Schema support
• FEAT: Adding Json Schema Pipeline to SeedPrompts #1432 — SeedPrompt schema pipeline, persistence, capability adaptation, and refusal-scorer adoption
• FEAT: Adding json schema to all scorer datasets #2017 — schema adoption across additional scorer datasets
• MAINT: Unify adversarial-chat JSON schema across Crescendo, TAP, and PAIR #2039 and MAINT: AdversarialConversationManager - unify adversarial-chat context, schema, and multimodal feedback #2053 — shared adversarial-chat schema across Crescendo, TAP/PAIR, Red Teaming, and related attacks

Schemas remain domain-specific; this PR does not introduce a universal response schema or change target/scorer/attack semantics.

CoPyRIT already maps persisted reasoning pieces into its existing Reasoning panel and has mapper/component test coverage. This PR adds the missing PyRIT output parity. CoPyRIT currently has no scenario-results UI, so scenario frontend rendering is outside this PR.

Tests and Documentation

Added or expanded unit coverage for:

• Pretty and Markdown conversation rendering
• Attack objective, pruned, and adversarial conversations
• Scenario-to-attack rendering composition
• Empty and malformed reasoning summaries
• Exact OpenAI Responses payload validation
• Original/converted data-type leakage prevention
• Exact tag literals, spacing, and Pretty colors
• Public helper argument forwarding

Local results:

• 171 passed
• pyrit.output: 98% statement coverage
• Ruff and formatting checks pass
• All required PR checks currently pass, including frontend unit, E2E, lint, and type-check workflows

Updated the paired output documentation with conversation, attack, and scenario examples and clarification that OpenAI exposes summaries rather than raw chain-of-thought.

[Copilot]

Pull request overview

Adds opt-in provider reasoning-summary rendering across conversation, attack-result, and scenario-result output.

Changes:

• Parses and validates OpenAI reasoning-summary payloads.
• Adds pretty and Markdown rendering with reasoning hidden by default.
• Propagates reasoning options through helpers and adds unit coverage.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pyrit/output/conversation/base.py	Adds reasoning filtering and parsing.
pyrit/output/conversation/pretty.py	Renders pretty reasoning blocks.
pyrit/output/conversation/markdown.py	Renders Markdown reasoning blocks.
pyrit/output/attack_result/base.py	Extends the rendering contract.
pyrit/output/attack_result/pretty.py	Propagates reasoning through pretty output.
pyrit/output/attack_result/markdown.py	Propagates reasoning through Markdown output.
pyrit/output/scenario_result/base.py	Extends the scenario rendering contract.
pyrit/output/scenario_result/pretty.py	Adds scenario-level reasoning output.
pyrit/output/helpers.py	Exposes reasoning options in helpers.
tests/unit/output/conftest.py	Adds shared reasoning fixtures.
tests/unit/output/conversation/test_reasoning.py	Tests parsing, visibility, and formatting.
tests/unit/output/attack_result/test_pretty.py	Tests pretty attack reasoning.
tests/unit/output/attack_result/test_markdown.py	Tests Markdown attack reasoning.
tests/unit/output/scenario_result/test_pretty.py	Tests scenario reasoning output.
tests/unit/output/test_helpers.py	Tests helper argument forwarding.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

pyrit/output/scenario_result/pretty.py:341

• When scenario reasoning is enabled, this nested attack printer renders each full objective conversation, and PrettyConversationMemoryPrinter displays every image_path piece in notebooks. Because this printer is always created with its default blur_images=False and output_scenario_async exposes no blur option, requesting reasoning summaries can unexpectedly display unblurred attack images with no way for callers to opt into the safety control available on output_attack_async. Please either render only the reasoning blocks or plumb the image-blur settings through the scenario API and nested printer.

        attack_result_printer = PrettyAttackResultMemoryPrinter(
            sink=sink,
            width=width,
            indent_size=indent_size,
            enable_colors=enable_colors,
        )