refactor: centralize persisted sandbox lifecycle traversal

[elezar]

Description

Evaluate a small refactor that centralizes the common traversal mechanics used by persisted sandbox lifecycle sweeps without hiding their distinct behavior.

start_persisted_sandboxes, stop_persisted_sandboxes_on_shutdown, and persisted lifecycle recovery repeat the same general pattern:

1. Capture persisted sandbox IDs.
2. Acquire the per-sandbox lifecycle gate.
3. Re-read the sandbox record while holding the gate.
4. Handle records that disappeared or could not be read.
5. Inspect the current phase.
6. Perform an asynchronous, phase-specific action while retaining the gate.

Consider extracting only these common mechanics into a small traversal or map helper. For example, the caller could select the concurrency policy and provide an asynchronous callback. Start-, stop-, and recovery-specific RPCs, outcome handling, state mutations, aggregation, and logging should remain explicit at their call sites.

The refactor must preserve:

• the shutdown stop sweep's bounded concurrency;
• startup's current ordering and error semantics;
• the lifecycle gate's lifetime across each operation; and
• the correctness property that the sandbox record is re-read only after acquiring its gate.

This is a readability and maintainability follow-up to #2743. It should not block #2743.

This refactor does not solve the race in which a new sandbox can be created after the shutdown sweep captures its inventory. It does not replace #2826, which tracks quiescing and draining in-flight lifecycle requests before shutdown cleanup.

Context

PR #2743 expanded persisted sandbox start and shutdown-stop reconciliation. Review of its lifecycle loops found that their shared concurrency and correctness mechanics are difficult to compare, increasing the risk that future changes cause the implementations to drift.

The abstraction should remain deliberately narrow. A single generic lifecycle sweep that obscures the meaningful differences between start, stop, and recovery would make the code harder to review.

Related:

• refactor(compute): unify gateway restart reconciliation #2743
• bug(gateway): drain lifecycle requests before shutdown cleanup #2826

Definition of Done

• Common traversal mechanics are centralized only if the resulting code is easier to read and review.
• Start-, stop-, and recovery-specific RPCs, outcome handling, state mutations, aggregation, and logging remain visible at their call sites.
• Stop preserves bounded concurrency, and start preserves its current ordering and error semantics.
• The lifecycle gate remains held for the full per-sandbox operation, with the persisted record re-read after gate acquisition.
• Tests cover gate/re-read behavior, disappeared-record skipping, read failures, and the caller-selected concurrency choices.
• Existing lifecycle behavior has no regression.
• The change does not claim to resolve or replace the shutdown-draining work tracked by bug(gateway): drain lifecycle requests before shutdown cleanup #2826.