bug(supervisor): accept distinct provider environment revisions in process sidecar

[cjagwani]

Scope clarification

The original filing incorrectly described the linked NemoClaw GitHub Actions job as using “Docker process/network sidecars.” NemoClaw uses OpenShell’s standard Docker driver with one openshell-sandbox process in the default combined network,process mode. The process-only sidecar watcher described below is not exercised by that topology, so the NemoClaw run is not reproduction evidence for this issue and is being investigated separately.

This issue is scoped to the independently reproduced Kubernetes split-sidecar defect documented in the k3d reproduction.

User Story

As a sandbox operator using the Kubernetes split-sidecar topology, I want a fresh process execution to receive the latest valid provider environment after policy or provider changes, so credential binding converges without recreating the sandbox.

Problem Statement

The process-side control watcher can ignore a distinct provider-environment snapshot. provider_env_revision is derived from a SHA-256 digest and represented as a u64, but the watcher treats the value as a monotonically increasing counter and skips any update whose numeric value is less than or equal to the installed value.

A valid new snapshot whose digest-derived value is numerically lower is therefore never installed for child processes. The network-side desired state advances while fresh executions on the process side retain stale environment state.

Impact / Why This Matters

In Kubernetes split-sidecar mode, attaching or removing an endpoint-bound static provider credential may not converge for fresh sandbox executions. Whether the stale state appears on binding or unbinding depends on the opaque fingerprints’ numeric ordering.

A sandbox restart or recreation can mask the defect, but that is not a safe convergence contract. Repeating no-op provider updates is not a deterministic workaround.

Acceptance Criteria

• The process-side control watcher installs every genuinely newer provider-environment snapshot even when its opaque fingerprint is numerically lower than the currently installed fingerprint.
• An exact duplicate snapshot remains a no-op.
• Ordering or replay safety is explicit: a delayed stale snapshot cannot replace newer desired state.
• A regression test starts with a numerically high current fingerprint, delivers a distinct valid snapshot with a lower fingerprint, and verifies that subsequent child execution receives the new environment.
• A replay test verifies that an older desired-state generation is not reinstalled after a newer one.
• The Kubernetes split-sidecar path is covered.
• An integration or E2E test covers an initially endpointless provider through unbound → bound → unbound transitions and verifies that fresh execs receive the expected environment state without sandbox recreation.

Validated Reproduction

John Myers reproduced the defect through the public CLI on Docker-backed k3d using the Kubernetes sidecar overlay:

• OpenShell checkout: b2ea81822
• Reported version: 0.0.111-dev.6+gb2ea81822
• Docker Desktop: 29.6.2, ARM64
• Cluster: local k3d
• Supervisor topology: Kubernetes sidecar via ci/values-sidecar.yaml
• Sidecar overlay: processBinaryAwareNetworkPolicy: false

Observed transition:

State	Network-side desired environment	Fresh process result
Initial unbound	0 entries	ABSENT
Bound	revision 17142220022187618806, 1 entry	PRESENT
Unbound again	revision 7844884057133558206, 0 entries	PRESENT — stale

The newer unbound fingerprint is numerically lower than the prior bound fingerprint. The network supervisor installed it, but the process-side state did not converge.

Full commands, redacted observations, and cleanup steps are in the reproduction comment.

Source Diagnosis

In v0.0.106, the server computes provider_env_revision from SHA-256 content (domain openshell-provider-env-revision-v3), so its numeric value has no ordering semantics:

https://github.com/NVIDIA/OpenShell/blob/c4b500a7de64d0b66e3ee8098f58d14299092162/crates/openshell-server/src/grpc/policy.rs

The process-only sidecar watcher nevertheless rejects revision <= provider_credentials.snapshot().revision:

https://github.com/NVIDIA/OpenShell/blob/c4b500a7de64d0b66e3ee8098f58d14299092162/crates/openshell-sandbox/src/lib.rs

The correct design keeps the revision as an opaque content fingerprint and uses a separate monotonic delivery generation, or an equivalently tested transport-order contract, for replay safety.

Related issues checked for duplication: #2777, #1731, and #2518. They cover credential refresh lifecycle, pushed desired state, and policy hashing, but not this digest-order suppression in the process sidecar.

[johntmyers]

Looking at this and trying a reproduction with k3d sidecar topology. @cjagwani - is NemoClaw modeling a split sidecar topology with Docker for a specific reason or use case?

[johntmyers]

📋 triage-agent

Local sidecar reproduction

I reproduced the reported revision-ordering defect locally through the public CLI and the Docker-backed k3d sidecar topology. The hash ordering in this run made the unbind transition descend rather than the bind transition, so the observed symptom was a stale placeholder after credential removal. This exercises the same revision <= current_revision suppression reported in the issue.

No outbound request was required, and the commands below check only whether the environment key exists; they never print the credential or placeholder value.

Environment

• OpenShell checkout: b2ea81822
• Reported version: 0.0.111-dev.6+gb2ea81822
• Docker Desktop: 29.6.2, ARM64
• Cluster: local k3d
• Supervisor topology: Kubernetes sidecar via ci/values-sidecar.yaml
• Sidecar overlay setting: processBinaryAwareNetworkPolicy: false

1. Start the Docker-backed sidecar deployment

export HELM_K3S_CLUSTER_NAME=openshell-repro-2847
export HELM_K3S_LB_HOST_PORT=18080

mise run helm:k3s:create
KUBECONFIG="$PWD/kubeconfig" mise run helm:skaffold:run:sidecar

In a second terminal:

KUBECONFIG="$PWD/kubeconfig" \
  kubectl port-forward -n openshell svc/openshell 18090:8080

For subsequent CLI commands:

export OPENSHELL_GATEWAY_ENDPOINT=http://127.0.0.1:18090
openshell status

2. Import an endpointless provider profile

repro-profile.yaml:

id: rev-repro
display_name: Provider revision reproduction
category: other
credentials:
  - name: token
    env_vars: [REV_REPRO_TOKEN]
    required: true
    auth_style: bearer
    header_name: authorization
binaries:
  - path: /bin/sh

export REV_REPRO_TOKEN='local-fixture-not-a-real-credential'

openshell provider profile lint --file repro-profile.yaml
openshell provider profile import --file repro-profile.yaml
openshell provider create \
  --name rev-repro \
  --type rev-repro \
  --credential REV_REPRO_TOKEN

The bare --credential REV_REPRO_TOKEN form reads the value from the environment rather than placing it in CLI arguments.

3. Create the initial unbound policy

unbound.yaml:

version: 1

filesystem_policy:
  include_workdir: true
  read_only:
    - /usr
    - /lib
    - /proc
    - /dev/urandom
    - /etc
  read_write:
    - /tmp
    - /dev/null

landlock:
  compatibility: best_effort

network_policies:
  repro_api:
    name: repro_api
    endpoints:
      - host: api.example.com
        port: 443
        protocol: rest
        enforcement: enforce
        access: read-only
    binaries:
      - path: /bin/sh

Create the sandbox with the endpointless provider attached but no credential binding:

openshell sandbox create \
  --name sidecar-repro \
  --provider rev-repro \
  --policy unbound.yaml \
  --no-tty \
  -- sh -c 'echo ready'

Check only for environment-key presence in a fresh exec:

openshell sandbox exec --name sidecar-repro -- \
  sh -lc 'if [ "${REV_REPRO_TOKEN+x}" = x ]; then echo PRESENT; else echo ABSENT; fi'

Observed:

ABSENT

4. Add the credential binding

bound.yaml is identical to unbound.yaml except for credential_binding:

version: 1

filesystem_policy:
  include_workdir: true
  read_only:
    - /usr
    - /lib
    - /proc
    - /dev/urandom
    - /etc
  read_write:
    - /tmp
    - /dev/null

landlock:
  compatibility: best_effort

network_policies:
  repro_api:
    name: repro_api
    endpoints:
      - host: api.example.com
        port: 443
        protocol: rest
        enforcement: enforce
        access: read-only
        credential_binding:
          provider: rev-repro
    binaries:
      - path: /bin/sh

openshell policy set sidecar-repro \
  --policy bound.yaml \
  --wait \
  --timeout 90

openshell sandbox exec --name sidecar-repro -- \
  sh -lc 'if [ "${REV_REPRO_TOKEN+x}" = x ]; then echo PRESENT; else echo ABSENT; fi'

Observed:

Policy version 3 loaded (active version: 3)
PRESENT

The sandbox log showed the network supervisor install the bound environment:

Provider environment refreshed [revision:17142220022187618806 env_count:1]

5. Remove the binding again

openshell policy set sidecar-repro \
  --policy unbound.yaml \
  --wait \
  --timeout 90

openshell sandbox exec --name sidecar-repro -- \
  sh -lc 'if [ "${REV_REPRO_TOKEN+x}" = x ]; then echo PRESENT; else echo ABSENT; fi'

Observed:

Policy version 4 loaded (active version: 4)
PRESENT

PRESENT is incorrect here: the desired endpointless environment is unbound again, so a fresh exec should report ABSENT.

The network supervisor did install the newer desired state with no provider environment entries:

Provider environment refreshed [revision:7844884057133558206 env_count:0]

The critical ordering is:

newer unbound revision:  7844884057133558206
prior bound revision:   17142220022187618806

The newer revision is numerically lower. The network supervisor installed it, but the process-side state did not converge, and no matching Sidecar provider environment refreshed event appeared for the descending transition.

Result

State	Network-side desired environment	Fresh process result
Initial unbound	0 entries	ABSENT
Bound	revision 17142220022187618806, 1 entry	PRESENT
Unbound again	revision 7844884057133558206, 0 entries	PRESENT — stale

Classification: validated bug. This is live evidence that the split sidecar path treats an opaque provider-environment fingerprint as an ordered generation. In this hash pair the failure appears on removal; with the opposite numeric ordering it appears on activation as described in the issue.

Cleanup

openshell sandbox delete sidecar-repro
openshell provider delete rev-repro
openshell provider profile delete rev-repro

KUBECONFIG="$PWD/kubeconfig" mise run helm:skaffold:delete:sidecar
HELM_K3S_CLUSTER_NAME=openshell-repro-2847 mise run helm:k3s:delete

[johntmyers]

🏗️ build-plan

Implementation Plan

Issue type: fix
Complexity: Medium
Confidence: High — clear path

Summary

Keep provider_env_revision as the gateway's opaque content fingerprint, and add a sidecar-local monotonic generation that explicitly orders provider-environment deliveries. Distinct fingerprints will always publish regardless of numeric value, while the generation lets the process supervisor reject duplicate or delayed stale messages safely.

Scope

• crates/openshell-sandbox/src/sidecar_control.rs: carry a provider-environment generation in bootstrap/update messages, assign it atomically for each distinct fingerprint, and serialize state update plus publication across cloned publishers.
• crates/openshell-sandbox/src/lib.rs: initialize the process watcher from the bootstrap generation and order updates by generation rather than fingerprint value.
• e2e/rust/tests/credential_gating.rs: exercise an endpointless provider through unbound → bound → unbound policy updates on one running sandbox and verify fresh exec environment-key presence after each transition.
• architecture/sandbox.md: document that provider revisions are fingerprints and sidecar delivery uses a separate monotonic generation.

Implementation Steps

1. Extend the internal sidecar bootstrap and update wire messages with provider_env_generation.
2. Change Publisher::publish_provider_env to suppress only an exact current fingerprint, increment the local generation for every distinct desired snapshot, and broadcast while holding the state lock so cloned publishers cannot reorder generations.
3. Change the process-side watcher to install updates with a newer generation even when their fingerprint is numerically lower, and to reject equal/older generation replays.
4. Add deterministic unit coverage for descending fingerprints, exact duplicates, A → B → A transitions, bootstrap duplicate handling, and delayed stale replay.
5. Add the live CLI E2E transition and update the architecture invariant.

Test Plan

• Unit tests: sidecar_control.rs verifies a lower distinct fingerprint is delivered with the next generation, duplicates remain no-ops, and repeated content after an intervening state is a new generation. lib.rs verifies a newer generation installs a lower fingerprint and older/equal generation replays cannot replace it.
• Integration tests: The sidecar socket round-trip tests cover bootstrap/update serialization with the generation field.
• E2E tests: Extend credential_gating to keep one sandbox alive while binding then unbinding an endpointless credential. Alternating two distinct fingerprints makes the old defect deterministic because one transition must be numerically descending. Run the focused Kubernetes sidecar lane in addition to local validation.

Risks & Open Questions

• Publication must remain inside the same lock that assigns the generation; otherwise cloned Publisher instances could send generation N+1 before N.
• Generation overflow will use an explicit checked invariant rather than silently wrap. It is unreachable in practice within one sidecar connection lifetime.
• The wire format is internal to the two supervisor containers in one pod, which run the same image/version; no public protocol migration is required.
• No stakeholder decision is required. A fingerprint-equality-only fix would not satisfy the issue's replay-safety acceptance criterion.

Documentation Impact

• Add the ordering invariant to architecture/sandbox.md.
• No published UX or gateway configuration documentation changes; this adds no config, API, or driver option.
• No SELinux/AppArmor/LSM compatibility impact because process identity, execution, and /proc access are unchanged.

---

Revision 1 — initial plan

[prekshivyas]

@johntmyers wondering if we have an ETA on this fix?

[johntmyers]

🏗️ build-from-issue-agent

Implementation Complete

PR: #2849

What was built

Sidecar provider-environment delivery now uses a connection-local monotonic generation instead of numerically ordering opaque content fingerprints. Distinct lower-valued revisions converge, while duplicate and delayed stale messages remain no-ops.

Tests

• Unit: descending fingerprint, duplicate, A → B → A, and replay-order coverage; full openshell-sandbox suite passed (122 tests)
• Integration: sidecar Unix-socket bootstrap/update round trips
• E2E: focused Kubernetes sidecar credential-gating test passed (1 test, 131.45s)

Docs updated

• architecture/sandbox.md: provider fingerprint and sidecar delivery-generation invariant

The issue will auto-close when the PR is merged.

[cjagwani]

No—NemoClaw is not intentionally modeling a split process/network sidecar topology with Docker. I traced the original statement to a filing-time inference, not runtime evidence.

The exact NemoClaw job uses OpenShell's Docker driver with one openshell-sandbox process in the default combined network,process mode. The process-only sidecar watcher and its revision <= current comparison are therefore not exercised by that run.

I corrected the issue body to remove the NemoClaw run as reproduction evidence and scope the issue to the independently validated Kubernetes split-sidecar defect. Your k3d reproduction remains the authoritative evidence for #2847 and #2849. The NemoClaw combined-Docker credential-sync failure is being tracked as a separate unresolved cause.