Repository navigation
fix(handlers): SAO-17638 LLM span double-encoding and workflow input - #259
Conversation
opikalova
left a comment
There was a problem hiding this comment.
🤖 This review was generated by the Astra agent (claude-opus-5). It may contain mistakes.
Verdict: request_changes — The I/O fix skips the default Responses API path, and the new user-input helper corrupts structured message content.
General Comments
-
🟠 major (testing): The new tests only drive
generation_span. The OpenAI Agents SDK selectsOpenAIResponsesModelby default, so real traces carryResponseSpanData. No test drives aresponse_spanwith alist[dict]input through the processor and asserts the exportedgen_ai.input.messages. Please add that case. Without it, the regression this change targets stays untested on the path most users hit. -
🟡 minor (other):
ruff format --checkfails on all 4 changed files. The repository runsruff-formatas a pre-commit hook, so please runpoetry run pre-commit run --files <changed-files>. The reported items are: -
src/splunk_ao/handlers/openai_agents/handler.pyline 36: a stray third blank line after_logger. -
src/splunk_ao/handlers/openai_agents/handler.pyline 359: line over 120 characters. -
src/splunk_ao/utils/openai_agents.pyline 291: line over 120 characters. -
tests/test_openai_agents.pylines 138, 170, 193, 205. -
🟡 minor (documentation):
CHANGELOG.mdkeeps an empty[Unreleased]section. This change alters the exported content of OpenAI Agents LLM and workflow spans, which users observe directly. Please add aFixedentry under[Unreleased].
2d26692 to
367269c
Compare
…plunkAOTracingProcessor (SAO-17638) Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…SAO-17638) Use the same user-message fallback in _update_owned_root, drop drive-by whitespace and comment edits, apply ruff format, and add a CHANGELOG entry. Co-Authored-By: Claude Code <noreply@anthropic.com>
367269c to
5c0dcf1
Compare
|
Follow-up: the valid review points outside this PR's scope are tracked in SAO-18193:
|
opikalova
left a comment
There was a problem hiding this comment.
🤖 This review was generated by the Astra agent (claude-opus-5-5). It may contain mistakes.
Verdict: approve — The scoped Chat Completions fix is correct and the earlier blocking points are fixed or tracked in SAO-18193. Two minor gaps remain.
|
Changes look good, but there is one regression introduced. See below for details - Normalize all supported output sequences - This affects custom models using tuple output; OpenAI's built-in Chat Completions model supplies lists. Since this is a regression and affects |
…7638) GenerationSpanData.output is typed Sequence[Mapping], so a custom model may pass a tuple. Only lists were unwrapped, so a tuple reached LoggedLlmSpan, failed validation, and the LLM span was dropped. Unwrap any sequence and keep the first choice. Add the reviewer's tests for tuple output and for input assigned after the span starts, as OpenAIChatCompletionsModel does. Co-Authored-By: Claude Code <noreply@anthropic.com>
…OG (SAO-17638) OpenAIChatCompletionsModel assigns the generation input after the span starts, so the workflow input fallback is only set at span end. Add a test that mirrors that and checks the workflow span shows the user message; it fails if the span-end fallback is removed. Scope the CHANGELOG entry to Chat Completions models, since the Responses API path is unchanged here.
|
Fixed: |
Summary
Fixes two issues in
SplunkAOTracingProcessor(OpenAI Agents SDK) from SAO-17638:_extract_llm_dataserializedGenerationSpanDatainput and output to a JSON string, butLoggedLlmSpanalready converts message lists, so the whole conversation collapsed into one
message. The lists now pass through; for output, the first choice is kept
because
LoggedLlmSpanaccepts a single output message."<Type> Step"as input. Agent, turn andworkflow spans have no input of their own. They now use the latest user message
in the first LLM call's input, on both the incremental (default) and
ingestion-hook paths.
Scope
Chat Completions (
GenerationSpanData), which the reported setup uses. ResponsesAPI (
ResponseSpanData) handling is unchanged.Not changed (by design)
Test plan
poetry run pytest tests/test_openai_agents.py tests/test_openai_agents_utils.py -n 0(2 network-dependent VCR tests also fail locally on main)
ruff checkandruff format --checkon changed files,invoke type-checksystem/user/assistant/tool messages; all workflow spans show the user prompt
🤖 Generated with Claude Code