bug(gateway): drain lifecycle requests before shutdown cleanup

[elezar]

User Story

As an OpenShell operator, I want graceful gateway shutdown to quiesce lifecycle mutations before stopping local sandbox compute, so that no sandbox created by an already accepted request is left running unexpectedly while the gateway is down.

Problem Statement

During graceful shutdown, the gateway stops its listener accept loops and then begins sandbox cleanup. Accepted connections, however, are served by detached, untracked tasks. Cleanup snapshots the persisted sandbox inventory and stops only the sandboxes present in that snapshot.

An in-flight sandbox creation can therefore commit after cleanup captures the inventory. The new sandbox is absent from the stop sweep and Docker or Podman compute can remain running while the gateway is down.

Startup reconciliation is less exposed because it runs before listener tasks begin accepting requests. The shutdown race predates PR #2743 for Docker: the previous label-based cleanup also listed containers and then stopped that fixed set. PR #2743 generalizes lifecycle cleanup through driver RPCs but does not close the request-draining gap.

Related work:

• refactor(compute): define driver shutdown and restart lifecycle contract #2417 defines the broader driver shutdown and restart lifecycle contract.
• refactor(compute): unify gateway restart reconciliation #2743 implements persisted sandbox stop/start reconciliation.

Impact / Why This Matters

Graceful shutdown can report cleanup completion even though compute created by an already accepted request remains running. This weakens the operational guarantee of graceful shutdown and can leave unmonitored local workloads consuming resources while the gateway is unavailable.

The current workaround is to externally prevent lifecycle requests before shutting down the gateway. That is insufficient because the gateway does not expose or enforce a drain boundary, and an operator cannot reliably know when all accepted lifecycle requests have completed.

Drew (@drew) should determine whether this correctness gap must block the OpenShell 0.1.0 release or can be resolved afterward. Record that decision in this issue.

Acceptance Criteria

• Record whether this issue blocks the OpenShell 0.1.0 release and the reasoning for that decision.
• Once graceful shutdown begins, the gateway quiesces or rejects new sandbox lifecycle mutations.
• The gateway drains accepted, in-flight lifecycle mutations before taking the final persisted-sandbox inventory and running the stop sweep.
• A deterministic test covers a sandbox creation racing with graceful shutdown and verifies that no eligible local compute is missed.
• Document whether graceful shutdown provides a strict guarantee that eligible local compute is stopped or only best-effort cleanup.
• The design does not rely only on repeated inventory sweeps, because another mutation can always commit after the final sweep without a quiesce-and-drain boundary.

Reproduction Steps

1. Arrange for a sandbox create request to be accepted but delayed before its persisted sandbox record becomes visible to shutdown cleanup.
2. Trigger graceful gateway shutdown.
3. Allow cleanup to capture or finish processing the persisted sandbox inventory.
4. Release the delayed create request so that it commits and starts its local compute.
5. Observe that the newly persisted sandbox was not included in the shutdown stop sweep and its Docker or Podman compute may remain running.

Environment

• OpenShell: current shutdown architecture, including the implementation in PR refactor(compute): unify gateway restart reconciliation #2743
• OS: any supported gateway host
• Runtime, deployment, or integration: local Docker or Podman compute driver during graceful gateway shutdown