Skip to content

A provider credential refresh closes every open network connection of a sandbox #5796

Open

Description

@sdrahn

User Story

I use OpenShell to run host-administration agents in sandboxes on SLES 16. Each agent gets an OAuth provider (client credentials from Keycloak), which the gateway renews, and calls a local OpenAI-compatible model server through an uninspected network endpoint.

I directly encountered connections from the sandbox ending in the middle of a request every few minutes, exactly when the gateway renewed the provider's token.

This affects every request that is open while a renewal happens: model calls of 30 s or more, and streaming responses through openshell forward.

Problem Statement

When a provider's credential value changes, every open TCP connection of the sandbox is closed. This happens with a gateway-renewed OAuth token, and also with openshell provider update. It also hits connections that carry no credential at all, to endpoints without protocol (L4 passthrough). The sandbox's policy is unchanged; the supervisor reports policy_changed:false provider_env_changed:true.

Impact / Why This Matters

  • Requests fail. The workload sees EOF on any request that spans a refresh. An agent loses its model call or tool result in the middle of a task.

  • It happens often. With 5-minute tokens (Keycloak's default) and refresh_before_seconds: 120, this happens every 4 minutes. With the default refresh_before of 300 s, it happens on every refresh sweep, once a minute.

  • Clients misbehave. A frontend whose stream breaks off resends its request while the first one is still being processed. In my case this mixed up an agent's conversation history.

  • The only workaround is long-lived tokens, with refresh_before_seconds well below their lifetime. That gives up short token lifetimes, and each refresh still drops every connection.

Acceptance Criteria

  • A refresh that changes only a provider credential's value leaves the sandbox's open connections up, with the same policy, providers and bindings. This holds both for a gateway-driven OAuth renewal and for openshell provider update --credential.
  • Requests made after the refresh carry the new credential value, including requests on an L7-inspected connection that was opened before it.
  • Connections to endpoints without protocol are not closed by a credential refresh.
  • A real policy change still closes connections as today: a new policy, a binding added or removed, a provider attached or detached, or fail-closed.

Reproduction Steps

  1. On the gateway host, serve a large file:
    mkdir /tmp/srv && dd if=/dev/zero of=/tmp/srv/big bs=1M count=200
    cd /tmp/srv && python3 -m http.server 8000
  2. Create a provider with a static credential:
    openshell provider create --name refresh-test --type generic --credential TOKEN=one
  3. Create a sandbox with that provider and a policy that allows the host endpoint without protocol (policy.yaml):
    version: 1
    network_policies:
      host:
        name: host
        endpoints:
          - { host: host.openshell.internal, port: 8000 }
        binaries:
          - { path: /usr/bin/curl }
    openshell sandbox create --name refresh-test --policy policy.yaml --provider refresh-test
  4. In the sandbox, start a slow download (about 3 minutes):
    openshell sandbox exec -n refresh-test -- curl --limit-rate 1M -o /dev/null http://host.openshell.internal:8000/big
  5. While it runs, change the credential's value:
    openshell provider update refresh-test --credential TOKEN=two
  6. The download ends at once with a transfer error, not after 200 MB. The supervisor logs Settings poll: config change detected [... policy_changed:false provider_env_changed:true] at that moment.

Suggested UX (if applicable)

None. No configuration or CLI change: open connections should survive a credential refresh.

Environment

  • OpenShell: v0.1.8 (gateway, CLI and supervisor)
  • OS: SUSE Linux Enterprise Server 16.1 (immutable), x86_64
  • Deployment: single host, gateway as a systemd service, sandboxes via an external systemd compute driver. Providers: OAuth client credentials from Keycloak (5-minute tokens, refresh_before_seconds: 120). Endpoint affected: host.openshell.internal:12400 without protocol (an OpenAI-compatible model server on the host).

Logs

# supervisor, one line per renewal, 4 minutes apart
Nov 20 07:40:06 openshell-supervisor-fence[166803]: OCSF CONFIG:DETECTED [INFO] Settings poll: config change detected [old_revision:17194639135693112131 new_revision:17194639135693112131 policy_changed:false provider_env_changed:true]
Nov 20 07:44:06 openshell-supervisor-fence[166803]: OCSF CONFIG:DETECTED [INFO] Settings poll: config change detected [old_revision:17194639135693112131 new_revision:17194639135693112131 policy_changed:false provider_env_changed:true]
Nov 20 07:48:07 openshell-supervisor-fence[166803]: OCSF CONFIG:DETECTED [INFO] Settings poll: config change detected [old_revision:17194639135693112131 new_revision:17194639135693112131 policy_changed:false provider_env_changed:true]
# workload, a model call open across one of them
model endpoint: Post "http://host.openshell.internal:12400/v1/chat/completions": EOF

Activity

  1. sdrahn commented on Nov 20, 2026

    @sdrahn
    Author

    Cause (v0.1.8). A credential refresh goes down the policy-reload path, and a reload closes every relay:

    1. crates/openshell-supervisor/src/config_runtime.rs:277-282: a new provider env revision sets provider_env_changed.
    2. config_runtime.rs:300-305: with a gateway-delivered policy, provider_env_changed alone makes policy_runtime_changed true.
    3. config_runtime.rs:431-453: that runs reload_gateway_configuration_runtime, which ends in OpaEngine::reload_configuration_from_proto_with_pid.
    4. crates/openshell-supervisor-network/src/opa.rs:808: that function calls advance_generation() on every reload, also for an identical policy.
    5. crates/openshell-supervisor-network/src/proxy/relay.rs:342 and l7/relay.rs:258/274/289: each relay selects on generation_guard.wait_until_stale() and closes when the generation moves. That includes the L4 passthrough relay, which injects nothing.

    The printed old_revision/new_revision are the config revision. The provider env revision, which did change, is not in the message.

  2. sdrahn commented on Nov 20, 2026

    @sdrahn
    Author

    Proposed fix: install a value refresh without reloading the policy.

    1. Detect a value refresh. Treat a change as a value refresh when provider_env_changed is set, the policy hash is unchanged, and the prepared environment has the same static credential bindings as the installed one. ProviderCredentialState::install_prepared already compares static_credential_identities.
    2. Install it without a reload. For a value refresh, run ctx.provider_credentials.install_prepared(...) and provider_readiness.credentials_installed(...), as the non-reload branch does at config_runtime.rs:639-646, and skip the reload. The generation stays, and no relay is woken. install_prepared keeps existing clones on the live state.
    3. Resolve per request in L7 tunnels. An L7 tunnel keeps the SecretResolver it was opened with (l7/relay.rs:66, :970). It should take it from the live ProviderCredentialState for each request, as l7/relay.rs:4586 does with state.resolver(). Otherwise an open tunnel goes on injecting the old value. I haven't traced every use of the captured resolver.

    A smaller first step, if (3) needs more design: keep the reload, but have relays that inject nothing (L4 passthrough, proxy/relay.rs:342) wait on a policy generation that a credential-only change does not advance. That alone fixes uninspected endpoints such as the model server above.

  3. mr-miles commented on Nov 20, 2026

    @mr-miles
    Contributor

    I also found this breaking workloads - it's fixed in #5361 . Anything causing a policy reload (e.g. approving a policy update suggestion) caused all the connections to be dropped which is very confusing for the agent.

  4. sdrahn commented on Nov 20, 2026

    @sdrahn
    Author

    @mr-miles Thank you, I did not find #5361 during my search before filing the issue.

  5. mr-miles commented on Nov 20, 2026

    @mr-miles
    Contributor

    Hope it is useful! I investigated because it made the sandboxes unusable, agents were spending tokens investigating the network and getting confused. - which sounds like your experience. I found that the supervisor closed unchanged connections at every opportunity so fixing it was involved. But it has improved things massively for me.

    Was hoping to get more traction from the maintainers (I don't understand how anyone could use it IRL as-is!) but I can only guess that their prod workloads use static policies that dont change and so they havent encountered these problems.

  6. sdrahn commented on Nov 20, 2026

    @sdrahn
    Author

    Yes, very useful. Thank you. I agree the current state is not really usable. I have actually put in some workarounds in my framework so I can actually work atm - essentially limiting the renewals and tuning things in a way they do not collide with other timeouts, including extending token lifetimes. I would love to remove the workarounds quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions