You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
rivetkit: a node that shuts down right after its actors sleep leaves many of them allocated to it #5928
A node that shuts down right after its actors sleep leaves many of them allocated to it
Versions: rivetkit 3.0.0-preview.2 (N-API runtime, Bun 1.4.1, macOS arm64) and engine
3.0.0-preview.2, with the default pegboard settings.
What happens. A node serves 250 awake actors. On SIGTERM it calls c.sleep() on each of them,
waits until every onSleep has returned (about 100 ms), and then calls registry.shutdown(),
which returns within 30 ms. A newer node of the same pool is serving by then.
Result. For 36 to 134 of the 250 actors, the engine never takes the stopped report. Its log
shows:
failed to send stop command; the stop timer resends it (no_responders), hundreds of times;
then envoy is gone past its lost threshold; ending the epoch without its stopped report, once
for each actor it lost;
and Envoy connection closed: WebSocket protocol error: Connection reset without closing handshake.
Requests to those actors wait about 45 s, and the first 32 of them fail at the gateway deadline.
Across three runs, reads of the 250 homes from the new node had a p99 of 45 s, with 32 failed.
Expected.registry.shutdown() does not close the envoy before the engine has taken the
stopped events the envoy already sent. Or a node has a way to wait for that.
Workaround. Keep the envoy connected for a while after the last onSleep returns, then call registry.shutdown(). We wait 250 ms plus 8 ms per actor; 0.5 s was the shortest that held for
251 actors. With that wait, three runs had zero lost actors, a read p99 of 257 to 311 ms, and no
failed reads. Waiting after registry.shutdown() instead does not help, because shutdown has
already reset the connection.
Notes.
registry.shutdown() without c.sleep() first waits until each idle actor sleeps by itself
(our sleepTimeout of 30 s) and then hands over cleanly.
The /actors listing reports sleep_ts, connectable_ts and start_ts as always null in
this version, so a node cannot see when the engine took an actor's stop.
The engine logs envoy connection queue is near its capacity during the burst of stops.
Activity
linear-code commentedon Oct 28, 2026
ACT-6525