Skip to content

fix(ai-monitoring): Raise conversation title max_tokens to 192 - #122948

Draft
sehr-m wants to merge 1 commit into
masterfrom
sehrmoosabhoy/conversation-title-max-tokens
Draft

fix(ai-monitoring): Raise conversation title max_tokens to 192#122948
sehr-m wants to merge 1 commit into
masterfrom
sehrmoosabhoy/conversation-title-max-tokens

Conversation

@sehr-m

@sehr-m sehr-m commented Aug 27, 2026

Copy link
Copy Markdown
Member

_generate_title_with_legacy_seer asks Seer for structured output with max_tokens=64 and reasoning="off" (so thinking_budget=0 and no padding) — a hard 64-token ceiling on visible output. On noisy input the model over-generates, the JSON never closes, and the response is unparseable. All three retries re-send an identical request at temperature=0.2, so they truncate identically, and llm_total_models: 1 means there is no fallback model behind it.

seer.llm.structured_invalid over the last 24h, grouped by finish_reason:

finish_reason count
max_tokens 1,819
stop 7
unknown 1

Truncation is ~99.6% of direct structured failures. 192 leaves ~180 tokens for the title after the JSON envelope, against a happy path of ~25.

Notes:

  • Paired with getsentry/seer#7944, which fixes the identical 64 in seer's conversation_title one-shot handler. The ai-monitoring.conversation-title-generation.oneshot-rollout-rate option defaults to 0.0, so all real traffic is currently on this legacy path — but the one-shot path inherited the same bug and needs both.
  • User-visible impact of the failure is cosmetic: generate_conversation_title falls back to fallback_title_from_message, so a conversation shows a truncated first message instead of a real title. The cost is ~3,700 wasted flash-lite calls/day and ~29k Sentry error events (SEER-945), which is the largest issue in the seer project.
  • This does not close SEER-945 on its own. The raise in seer's llm_proxy.py:120 still turns any remaining parse failure into a 500, and other structured callers have their own budgets.

Title generation capped visible output at 64 tokens with reasoning off, so
any over-generation truncated the JSON mid-string and left the response
unparseable. All three retries re-send an identical request at temperature
0.2, so they truncate identically and the call fails with no fallback model
behind it.

Datadog `seer.llm.structured_invalid` over 24h: 1,819 failures tagged
`finishreason.max_tokens` against 7 tagged `stop`, so truncation is ~99.6%
of direct structured failures.

192 leaves ~180 tokens for the title after the JSON envelope, against a
happy path of ~25.
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Aug 27, 2026

@chromy chromy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! fyi @vgrozdanic

@vgrozdanic vgrozdanic left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for fixing this 🙏

We can merge this today, but on Monday, i am planning to clear all of these legacy seer API code since we migrated to oneshot Seer API.

I'll leave it up to you if you want to merge this today, or just close the PR and on Monday i clean up all of this "old" code, it's already not being called as we are on 100% rollout of the new path that is using oneshot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants