Preserve Gemini thought signatures through streamed tool calls - #206
Draft
fanny-riols wants to merge 1 commit into
Draft
Preserve Gemini thought signatures through streamed tool calls#206fanny-riols wants to merge 1 commit into
fanny-riols wants to merge 1 commit into
Conversation
ALMGeminiClient hits Gemini's raw OpenAI-compatible endpoint directly, where multi-turn thought signatures ride under extra_content.google on each tool_call. litellm.stream_chunk_builder's tool-call merge (used by _assemble_stream_chunks for the streaming path) only knows how to carry over id/type/function/provider_specific_fields, so it silently dropped extra_content when reassembling streamed chunks -- causing Gemini 3.x audio-native models to fail on the next turn with "Function call is missing a thought_signature". Add _merge_streamed_tool_call_extras to alm_base.py: rebuilds tool_calls from the raw stream chunks, preserving any key litellm's merge doesn't recognize, and is a no-op when no such extra keys are present (e.g. for vLLM-hosted models). Also update system.py's thought-signature log check to recognize extra_content alongside CASCADE's provider_specific_fields.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ALMGeminiClient hits Gemini's raw OpenAI-compatible endpoint directly, where multi-turn thought signatures ride under
extra_content.googleon each tool_call.litellm.stream_chunk_builder's tool-call merge (used by_assemble_stream_chunkswhen streaming the LLM response) only knows how to carry over id/type/function/provider_specific_fields, so it silently dropped extra_content when reassembling streamed chunks -- causing Gemini 3.x audio-native models to fail on the next turn with "Function call is missing a thought_signature".Add
_merge_streamed_tool_call_extrasto alm_base.py: rebuilds tool_calls from the raw stream chunks, preserving any key litellm's merge doesn't recognize, and is a no-op when no such extra keys are present (e.g. for vLLM-hosted models). Also update system.py's thought-signature log check to recognize extra_content alongside CASCADE's provider_specific_fields.