fix(ai-monitoring): Raise conversation title max_tokens to 192 - #122948
Draft
sehr-m wants to merge 1 commit into
Draft
fix(ai-monitoring): Raise conversation title max_tokens to 192#122948sehr-m wants to merge 1 commit into
sehr-m wants to merge 1 commit into
Conversation
Title generation capped visible output at 64 tokens with reasoning off, so any over-generation truncated the JSON mid-string and left the response unparseable. All three retries re-send an identical request at temperature 0.2, so they truncate identically and the call fails with no fallback model behind it. Datadog `seer.llm.structured_invalid` over 24h: 1,819 failures tagged `finishreason.max_tokens` against 7 tagged `stop`, so truncation is ~99.6% of direct structured failures. 192 leaves ~180 tokens for the title after the JSON envelope, against a happy path of ~25.
vgrozdanic
approved these changes
Aug 28, 2026
vgrozdanic
left a comment
Member
There was a problem hiding this comment.
thank you for fixing this 🙏
We can merge this today, but on Monday, i am planning to clear all of these legacy seer API code since we migrated to oneshot Seer API.
I'll leave it up to you if you want to merge this today, or just close the PR and on Monday i clean up all of this "old" code, it's already not being called as we are on 100% rollout of the new path that is using oneshot
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_generate_title_with_legacy_seerasks Seer for structured output withmax_tokens=64andreasoning="off"(sothinking_budget=0and no padding) — a hard 64-token ceiling on visible output. On noisy input the model over-generates, the JSON never closes, and the response is unparseable. All three retries re-send an identical request attemperature=0.2, so they truncate identically, andllm_total_models: 1means there is no fallback model behind it.seer.llm.structured_invalidover the last 24h, grouped byfinish_reason:max_tokensstopunknownTruncation is ~99.6% of direct structured failures. 192 leaves ~180 tokens for the title after the JSON envelope, against a happy path of ~25.
Notes:
conversation_titleone-shot handler. Theai-monitoring.conversation-title-generation.oneshot-rollout-rateoption defaults to 0.0, so all real traffic is currently on this legacy path — but the one-shot path inherited the same bug and needs both.generate_conversation_titlefalls back tofallback_title_from_message, so a conversation shows a truncated first message instead of a real title. The cost is ~3,700 wasted flash-lite calls/day and ~29k Sentry error events (SEER-945), which is the largest issue in the seer project.llm_proxy.py:120still turns any remaining parse failure into a 500, and other structured callers have their own budgets.