(stt): reset the retry budget on any event, not only on a final transcript

[GregHolmes]

Summary

RecognizeStream._main_task gives a stream max_retry attempts (3 by default). _num_retries only ever increments, and its single reset lives in _metrics_monitor_task, on FINAL_TRANSCRIPT:

https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/stt/stt.py#L541-L543

That makes the budget a stream-lifetime counter replenished by the caller speaking, rather than a consecutive-failure counter replenished by the connection recovering.

A stream that reconnects cleanly but sits through silence never earns its budget back. max_retry drops spread over one long call then kill it permanently, with recoverable=False and no further reconnect, even though every one of those reconnects succeeded. The drops need not be consecutive or close together: a blip at minute 1, 10 and 40, then a fourth, is enough.

Why silence is the ordinary case, not an edge case

The gaps where a voice agent has no final transcript are exactly the gaps where it is most exposed: the agent is talking, the caller is on hold, an IVR is playing, someone stepped away. That is also when an idle socket is most likely to be reaped by a NAT or a load balancer. The condition that spends the budget and the condition that blocks its reset are the same condition.

Reproduction

Three streams whose connections all succeed and then drop. Ten drops each. The only difference is what they deliver in between.

what the stream delivers between drops	connections made	outcome
nothing at all	4	dies: failed to recognize speech after 3 attempts
a FINAL_TRANSCRIPT each time	11	survives
RECOGNITION_USAGE only	4	dies

The third row is the ordinary case: the Deepgram plugin emits a usage event every 5s while audio flows, speech or no speech. Same failures, opposite outcome, decided purely by whether somebody was talking.

Instrumenting _num_retries shows the mechanism. With transcripts it sits at a steady 1 forever, reset by the event and incremented by the failure. Without them it climbs 0, 1, 2, 3 and the stream is gone.

A runnable script is in the PR below as two tests, but the shape is just:

class _FlappingStream(RecognizeStream):
    async def _run(self) -> None:
        self.runs += 1
        if self._emit is not None:
            self._event_ch.send_nowait(self._emit)   # usage event, no transcript
            await asyncio.sleep(0)
        if self.runs <= self._drops:
            raise APIConnectionError("socket dropped")
        await asyncio.sleep(3600)

Is this intended?

I want to flag that it might be. "A stream that has never been understood should not retry forever" is a defensible rule, and resetting on FINAL_TRANSCRIPT implements it. If that is the intent, then the bug is narrower: the budget should still reset once a reconnect succeeds, rather than only once recognition succeeds.

Suggested fix

Reset on any event from the provider rather than only on FINAL_TRANSCRIPT. Any event proves the connection came up and delivered. A connection that has never delivered anything at all still gives up as before, so this does not become an unbounded retry.

Reported as #7208.

Happy to take it a different way if you would rather the reset hung off a successful connection explicitly, though that needs a signal out of _run() that the base class does not have today.

🤖 Generated with Claude Code

[longcw]

thanks for the pr, I think the analysis is right but need some changes:

[longcw]

I'd like to put the reset back in _metrics_monitor_task and widen it from FINAL_TRANSCRIPT to other events that verify the connection is health , instead of overriding send_nowait in _HealthySignallingChan.

[longcw]

also need to add that for MultiSpeakerAdapter  

agents/livekit-agents/livekit/agents/stt/multi_speaker_adapter.py

Lines 143 to 145 in 4de6232

	async for ev in event_aiter:
	if ev.type == SpeechEventType.FINAL_TRANSCRIPT:
	self._num_retries = 0