fix: session migration triggers a full reconnect instead of a resume - #1197
Conversation
…ting
For a node migration the server sends `LeaveRequest{Action: RESUME,
Reason: MIGRATION}`, which asks the client to reconnect with `reconnect=1`
and keep its session. The engine's leave handler did exactly that, but
`attemptReconnect` then unconditionally escalated any `leaveReconnect`
into a full reconnect:
if (... || [ClientDisconnectReason.leaveReconnect, ...].contains(reason)) {
fullReconnectOnNext = true;
}
That list predates protocol v13 (#439), when a leave with `can_reconnect`
could only mean a full reconnect. The v13 RESUME branch ported in #574
never updated it, so the resume branch has been dead code since: every
RESUME leave ran `restartConnection()`, emitting `RoomReconnectingEvent`,
dropping every `RemoteParticipant` and re-joining.
Drop `leaveReconnect` from the escalation list — the callers that do need
a full reconnect (the RECONNECT leave branch, the connection check) set
`fullReconnectOnNext` themselves. Also stop forcing the flag to false in
the RESUME branch: client-sdk-js and rust-sdks both treat an escalation as
sticky, so a resume that already failed at the media level is not
downgraded back into a resume loop.
Adds `test/core/leave_action_test.dart` covering both leave actions, and
implements `setConfiguration` on the mock peer connection (the resume path
applies the `ReconnectResponse` ICE servers).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`reconnect=1` is the query parameter the server actually keys off to distinguish a resume from a re-join, and it was the only part of the resume contract the test wasn't checking. Also documents why the socket close that follows the Leave is not simulated: a bare socket drop reconnects with reason `signal`, which resumes on its own, so delivering the close before the leave-driven attempt runs makes the test pass even when the leave action is ignored. In production the close arrives a round-trip later and never wins that race, which is why the reported bug reproduced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
handleReconnect replaces the pending retry timer together with its reason,
and the reason based escalation ran only when that timer fired. A
Leave{RESUME} arriving right after a peer connection failure therefore
swapped the reason to leaveReconnect and the failed connection was resumed
instead of restarted. Now that leaveReconnect no longer escalates on its
own, decide the escalation in handleReconnect so a later request cannot
drop it. The server side resumeConnection switch stays in attemptReconnect
so the latest ClientConfiguration wins.
The RECONNECT test now answers the re-join and waits for
RoomReconnectedEvent instead of tearing down mid restart, and asserts the
signal URL carries no reconnect flag. Add cases for a stale
fullReconnectOnNext, for resumeConnection DISABLED from the server, and for
a Leave{RESUME} racing a pending peer failure retry. The RESUME test also
checks the ReconnectResponse configuration reached both transports.
E2EContainer gains answerJoin() and a clientConfiguration option so tests
can drive a full reconnect and shape the join response.
| if ([ | ||
| ClientDisconnectReason.negotiationFailed, | ||
| ClientDisconnectReason.peerConnectionFailed, | ||
| ].contains(reason)) { | ||
| fullReconnectOnNext = true; |
There was a problem hiding this comment.
🟡 Successful resume leaves stale escalation
During an active resume, a peer failure makes handleReconnect set fullReconnectOnNext for a retry that can be canceled. attemptReconnect clears the retry after success, but not the flag. A later disconnect is suppressed, or the next resume becomes a full reconnect.
Learn more
A peer connection can report failure while resumeConnection is still restoring ICE. This block records the required full reconnect immediately and schedules a retry. If the active resume then reaches connected state, attemptReconnect cancels that retry but leaves fullReconnectOnNext true. The room subsequently drops an EngineDisconnectedEvent because the disconnect handler treats the flag as an active restart.
Example: A signal reconnect starts, then the primary peer connection briefly reports failed before its ICE restart reaches connected. The resume succeeds and its queued full reconnect is canceled. The next ordinary signal loss emits no RoomDisconnectedEvent; alternatively, a later migration performs an unnecessary full reconnect.
Recommended fix: Track a full-reconnect request separately from the flag consumed by the active attempt. After an attempt succeeds, either dispatch any escalation recorded during that attempt or clear it explicitly; do not cancel its retry while retaining only fullReconnectOnNext. Add a regression test where peerConnectionFailed arrives after _attemptingReconnect becomes true and the active resume subsequently succeeds.
Was this helpful? React with 👍 or 👎 to provide feedback.
Fixes CLT-3322.
Problem
When the server initiates a node migration, the SDK performs a full reconnect instead of a resume:
RoomReconnectingEventis emitted, everyRemoteParticipantis torn down (oneParticipantDisconnectedEventeach), the client re-joins, andRoomConnectedEventfires again. It should emitRoomResumingEventand keep the session intact.Reported by a customer testing
Room.sendSimulateScenario(migration: true)on 2.12.0:Migrations are routine on Cloud, so every Flutter client sees its remote participants disappear and re-join, subscriptions rebuilt, and per-participant UI state lost.
Root cause
For a migration the server sends
LeaveRequest{Action: RESUME, Reason: MIGRATION}(livekit/pkg/rtc/participant.go,MaybeStartMigration) and then closes the signal socket.RESUMEmeans: reconnect withreconnect=1, keep the session.Engine's leave handler did the right thing — clearedfullReconnectOnNextand calledhandleReconnect(ClientDisconnectReason.leaveReconnect). ButattemptReconnectimmediately re-set the flag:That list predates protocol v13 (added in #439, when a leave with
can_reconnectcould only mean a full reconnect). The v13RESUMEbranch was ported from client-sdk-js in #574 but the escalation was never updated — so the resume branch has been dead code ever since and everyRESUMEleave ended up inrestartConnection().Neither reference SDK behaves this way:
RTCEngine.attemptReconnectescalates only forresumeConnection === DISABLEDor a never-connected PeerConnection.on_session_eventroutesAction::Resumestraight into a resume cycle.Changes
lib/src/core/engine.dart:leaveReconnectfrom the escalation list inattemptReconnect. The callers that genuinely need a full reconnect — theRECONNECTleave branch andconnection_check/checks/checker.dart— already setfullReconnectOnNext = truethemselves.fullReconnectOnNext = falsein theRESUMEbranch. JS and Rust both treat an escalation as sticky, so a resume that already failed at the media level isn't downgraded back into a resume loop.test/mock/peerconnection_mock.dart: implementsetConfiguration(it threwUnimplementedError; the resume path applies theReconnectResponseICE servers to both transports).Tests
New
test/core/leave_action_test.dart:RESUME(migration) →RoomResumingEvent, noRoomReconnectingEvent, noParticipantDisconnectedEvent, remote participants retained,fullReconnectOnNextback to false.RECONNECT→RoomReconnectingEventand participants dropped, as before.Confirmed the
RESUMEtest fails against the pre-fix code (times out waiting forRoomReconnectedEvent, because the engine re-joins instead of resuming). Full suite (409 tests),flutter analyze,dart formatandimport_sorterall clean.Note for app developers
RoomResumingEventis the Flutter analog of JS'sSignalReconnecting;RoomReconnectingEventmeans a full reconnect. Both paths end inRoomReconnectedEvent.Follow-ups (not in this PR)
attemptReconnectearly-returns on_attemptingReconnect, and the successful attempt's_clearPendingReconnect()cancels the queued retry, leavingfullReconnectOnNextstale-true (which also suppresses the next legitimateRoomDisconnectedEvent). JS consumes the flag at attempt start and re-dispatches infinally; that can't be copied verbatim here becauseRoomreadsengine.fullReconnectOnNextduring the restart's join to skip fast-connect republishing.RoomConnectedEventagain on a full reconnect (driven offEngineJoinResponseEvent); JS only emitsReconnected. Changing that is a public-behavior change.🤖 Generated with Claude Code