Skip to content

rivetkit: a node that shuts down right after its actors sleep leaves many of them allocated to it #5928

Open

Description

@freyzzzz

A node that shuts down right after its actors sleep leaves many of them allocated to it

Versions: rivetkit 3.0.0-preview.2 (N-API runtime, Bun 1.4.1, macOS arm64) and engine
3.0.0-preview.2, with the default pegboard settings.

What happens. A node serves 250 awake actors. On SIGTERM it calls c.sleep() on each of them,
waits until every onSleep has returned (about 100 ms), and then calls registry.shutdown(),
which returns within 30 ms. A newer node of the same pool is serving by then.

Result. For 36 to 134 of the 250 actors, the engine never takes the stopped report. Its log
shows:

  • failed to send stop command; the stop timer resends it (no_responders), hundreds of times;
  • then envoy is gone past its lost threshold; ending the epoch without its stopped report, once
    for each actor it lost;
  • and Envoy connection closed: WebSocket protocol error: Connection reset without closing handshake.

Requests to those actors wait about 45 s, and the first 32 of them fail at the gateway deadline.
Across three runs, reads of the 250 homes from the new node had a p99 of 45 s, with 32 failed.

Expected. registry.shutdown() does not close the envoy before the engine has taken the
stopped events the envoy already sent. Or a node has a way to wait for that.

Workaround. Keep the envoy connected for a while after the last onSleep returns, then call
registry.shutdown(). We wait 250 ms plus 8 ms per actor; 0.5 s was the shortest that held for
251 actors. With that wait, three runs had zero lost actors, a read p99 of 257 to 311 ms, and no
failed reads. Waiting after registry.shutdown() instead does not help, because shutdown has
already reset the connection.

Notes.

  • registry.shutdown() without c.sleep() first waits until each idle actor sleeps by itself
    (our sleepTimeout of 30 s) and then hands over cleanly.
  • The /actors listing reports sleep_ts, connectable_ts and start_ts as always null in
    this version, so a node cannot see when the engine took an actor's stop.
  • The engine logs envoy connection queue is near its capacity during the burst of stops.

Activity

  1. linear-code commented on Oct 28, 2026

    @linear-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions