feat(server): support more than one sandbox ServiceAccount per gateway

[bjw123]

Summary

Bootstrap auth compared the TokenReview principal against a single value — [openshell.drivers.kubernetes] service_account_name — which is also the name the driver stamps onto every sandbox pod. One field served both purposes, so every sandbox on a gateway necessarily shared one Kubernetes identity, and where something other than the driver assigns the pod's ServiceAccount, bootstrap failed outright.

This implements both parts of the issue.

Part 1 makes bootstrap auth accept a configured set. additional_bootstrap_service_account_names adds identities for bootstrap only; service_account_name stays the one name the driver puts on pods.

Part 2 lets a caller choose the account for a single sandbox through SandboxTemplate.service_account_name, constrained to selectable_service_account_names plus the driver default. Without part 2 the driver still stamps one name on every pod it creates, so a gateway that owns its pods still has one Kubernetes identity — and therefore one cloud identity — for every sandbox.

Both lists are empty by default, so a gateway that sets neither behaves exactly as before.

Related Issue

Refs #2806, which I filed (state:accepted, area:gateway, area:compute, test:e2e-kubernetes). @elezar gave the go-ahead to pick it up after vouching me.

What the issue asks for

#2806 proposes two complementary parts, and is explicit that they are separable:

1. Bootstrap auth accepts a configured set of sandbox service accounts, with a single value staying the default and current behaviour unchanged.
2. The pod's ServiceAccount becomes selectable per sandbox — e.g. platform_config.service_account_name — constrained to the set from (1) and defaulting to the existing single value.

This PR implements both, and covers all six of the issue's Definition-of-Done items (checked off at the bottom of this description) plus the part 2 capability.

Part 1 is the prerequisite: a per-sandbox selector without an accepted set cannot authenticate, because bootstrap would still compare the presented principal against the one pod-default name. It is also independently useful on its own, and that is the case that motivated the issue — where something other than the gateway assigns the pod's ServiceAccount (a mutating admission policy, or an external controller that owns the sandbox pods), bootstrap fails outright today. The issue quotes the failure as

K8s TokenReview principal is not the configured sandbox service account

which this PR replaces with ... is not an accepted sandbox service account, since "the configured account" is no longer a single thing. Worth noting for anyone searching logs or docs for the old string.

Part 1 also follows the design constraint the issue is emphatic about: additive, not a widening of service_account_name. That field feeds spec.serviceAccountName on the pod, so a list or delimited string there would render invalid pods. The pod default and the accepted-for-auth set are kept as separate settings, which is the shape #2656 established for namespace validation and which this mirrors — ServiceAccountValidator sits next to the existing NamespaceValidator and works the same way.

How part 2 works

SandboxTemplate.service_account_name (new field 12 on the public message) is forwarded into platform_config, and the Kubernetes driver resolves the pod's account in validate_sandbox_create:

• unset → the driver's service_account_name, exactly as before
• a name in selectable_service_account_names (or the driver default) → that name
• anything else, including a value that is present but blank → InvalidArgument

It fails rather than falling back deliberately. A sandbox silently running as a different identity than the caller asked for does not surface at create time; it surfaces later as a confusing cloud IAM denial.

Resolving in validate_sandbox_create rather than in create_sandbox matters: that hook runs before the gateway persists the sandbox record and before it mints the sandbox JWT, and before the driver creates a workspace namespace or copies image-pull and TLS secrets. A rejected request leaves nothing behind.

Validation is layered. The gateway rejects a requested name that Kubernetes could not issue (DNS-1123 subdomain, ≤253 chars) and caps its length with the other template strings, so an over-long value cannot be echoed back into a gRPC status. The driver validates its own configured lists at startup, so a typo fails the gateway instead of surfacing per-request. And only the Kubernetes driver supports the field: the gateway rejects a request that sets it under another driver, because Podman would ignore it — acknowledging a request to confine a sandbox to an identity and then not doing it — while Docker and VM would fail with a message about platform_config, which the caller never set.

The rejection names the account the caller asked for and nothing else. The selectable set goes to the gateway log, matching how the bootstrap rejection already behaves.

Selectability is a second, separate opt-in, and that is the crux of the design. additional_bootstrap_service_account_names means "accept this identity when something outside the driver assigned it to the pod". selectable_service_account_names means "any caller who can create a sandbox may run as this identity". Those are different decisions, so enrolling a privileged account so an externally-owned pod can bootstrap does not put that account on the menu for everyone:

Configured in	Assigned by	A caller may request it
service_account_name	the driver, on every pod	yes
additional_bootstrap_service_account_names	something outside the driver	no
selectable_service_account_names	the driver, when requested	yes

Both lists feed the accepted-for-auth set, since an account a sandbox can run as has to be able to bootstrap. KubernetesComputeConfig owns both set builders so the authenticator and the driver cannot drift.

One limitation worth being explicit about: there is no per-caller restriction within the selectable set. Everything in selectable_service_account_names is available to every caller who can create a sandbox, so its members should be scoped accordingly. If you would rather that were policy-driven per user or per workspace, that is a larger change and I would rather agree the shape first.

On the triage recommendations

The triage assessment on the issue asked for unit coverage of legacy single-account behaviour, multiple allowed identities, non-member rejection, and selector validation, plus Kubernetes E2E coverage for bootstrap with a non-default allowed ServiceAccount.

• The first three are covered — see Testing below.
• Selector validation is covered: a request naming an account outside the selectable set is rejected, and the test that matters asserts a bootstrap-only account is not selectable.
• E2E is not in this PR. Rationale and the closest existing template are in the Testing section; happy to add it here instead of as a follow-up.

The triage also asked to "keep namespace binding explicit for any multi-namespace deployment". Namespace validation is untouched by this PR and still runs separately per workspace_mode, but the accepted set is keyed on the bare ServiceAccount name, so a name is accepted in every namespace that validator accepts. That is called out in the config docs, the chart values and the reference docs — and if you would rather the setting took qualified namespace:name principals, I am happy to change it; see the design notes below.

Changes

• auth/k8s_sa.rs — new ServiceAccountValidator, a closed BTreeSet with an accepts() check, built either directly or from the driver config via from_kubernetes_config. token_review_identity consults it instead of comparing one string, and now carries the presented account out of validation so the success log names the identity that actually authenticated rather than the configured default.
• config.rs (kubernetes driver) — additional_bootstrap_service_account_names and selectable_service_account_names, both #[serde(default, skip_serializing_if = "Vec::is_empty")] and empty by default, plus the three functions that give both consumers one source of truth: accepted_bootstrap_service_account_names(), selectable_pod_service_account_names() and resolve_pod_service_account().
• proto/openshell.proto — SandboxTemplate.service_account_name (field 12). Go SDK bindings regenerated; sdk/go proto:check passes.
• compute/mod.rs — build_platform_config forwards the requested account to the driver, which is the only layer that knows what is selectable.
• driver.rs — create_sandbox resolves the effective account before building SandboxPodParams, so the rendered pod and the bootstrap authenticator agree on one name, and a rejected request never reaches the apiserver.
• lib.rs — builds the validator from the bootstrap config and logs the accepted set once when the authenticator is enabled.
• Helm — server.drivers.kubernetes.additionalBootstrapServiceAccountNames and selectableServiceAccountNames, rendered into the [openshell.drivers.kubernetes] table directly beneath service_account_name and only when non-empty. A blank or non-string entry fails at render time rather than rendering away silently (sprig quote drops a nil, so [null] would otherwise produce = []). Chart README regenerated.
• Docs — a new ServiceAccount Bootstrap Identities section in docs/reference/sandbox-compute-drivers.mdx, the driver-key table row, and the gateway-config.mdx example.
• grpc/validation.rs — the requested name joins the other template strings under MAX_TEMPLATE_STRING_LEN and must be a name Kubernetes could issue.
• compute/mod.rs (third commit) — create_sandbox forwarded AlreadyExists and FailedPrecondition from the driver and collapsed everything else into Internal, so a driver-rejected request reached the caller as a server fault that clients retry. InvalidArgument is now preserved, the four near-identical arms are one, and the compensating store delete no longer discards its error — that failure orphans a sandbox row, so it is logged. This is not required by the selector, which is refused in validate_sandbox_create before anything is persisted; it is the same class of problem one layer down.
• architecture/gateway.md and .agents/skills/debug-openshell-cluster/SKILL.md — both described the gateway as accepting the configured service account, which is no longer accurate. Updated per the AGENTS.md requirements for architecture docs and for changes to Helm values/templates.

One incidental line: adding use std::collections::BTreeSet makes a pre-existing fully-qualified std::collections::BTreeSet::from in an unrelated test redundant, which unused_qualifications rejects under -D warnings, so that call is now unqualified.

Design notes

• Additive, not a widening. service_account_name is untouched, as the issue asks: a pod spec has a single serviceAccountName field, so a list there has no valid rendering. The pod default and the accepted-for-auth set are separate settings.
• Names are trimmed; blank entries are dropped. A ServiceAccount name never carries surrounding whitespace, so a padded config entry would otherwise sit in the set as a member no TokenReview username can ever match — an enrolment that looks correct in the values file and silently does nothing.
• Matching is exact and namespace-independent. The namespace is validated separately by NamespaceValidator, so under managed and operator workspace modes a name is accepted in every namespace that validator accepts. Adding a name does not widen which namespaces may bootstrap, but it does mean a generic name (default being the obvious one) is a much larger surface than it looks. This is documented in the config field, the chart values and the docs section. If you would rather the setting took qualified namespace:name principals, say so — that is a bigger change than part 1 and I did not want to pre-empt it.
• Where the setting lives. It sits in the driver table because that is where the gateway already reads namespace and service_account_name for bootstrap (kubernetes_config_for_k8s_sa_bootstrap). feat(server): support more than one sandbox ServiceAccount per gateway #2806 notes the overlap with refactor(server): decouple K8s ServiceAccount bootstrap from selected compute driver config #2023 on bootstrap-config ownership; this follows the existing placement rather than pre-empting that decision, and it moves with service_account_name if refactor(server): decouple K8s ServiceAccount bootstrap from selected compute driver config #2023 relocates it.
• Security properties are unchanged. Still TokenReview, still pod-bound extras required, still the live pod UID matched against the token, still validated against the pod's owning Sandbox CR. The set is operator-configured and closed.
• No format validation on the entries. service_account_name is not format-validated either, and a well-formed but wrong name fails exactly as silently as a malformed one, so the check would buy little. The rejection log names the presented principal alongside the accepted set, which is what actually makes a typo diagnosable. Happy to add is_dns_1123_label enforcement at startup if you want the stricter posture.

There is a natural bound on this worth stating: Kubernetes refuses to mint a pod-bound token whose ServiceAccount differs from the pod's own — cannot bind token for serviceaccount "X" to pod running with different serviceaccount name. An accepted identity can therefore only ever be presented by a pod actually running that ServiceAccount.

Testing

• cargo test -p openshell-server -p openshell-driver-kubernetes — green (1,409 and 225). Part 1 unit tests: pod default accepted alone (legacy behaviour), every configured name accepted, non-member rejected, blank entries dropped and padded entries trimmed, duplicates collapsed, an additional account authenticating through token_review_identity, and the presented account reported (with presented ≠ pod default, so the assertion distinguishes the token from the config). The pre-existing single-account accept/reject tests still pass unchanged.
• Config ingest — k8s_sa_bootstrap_reads_additional_service_accounts_from_driver_table parses the exact TOML the chart renders through kubernetes_config_for_k8s_sa_bootstrap and builds the validator from it. KubernetesComputeConfig denies unknown fields, so this is what pins the Helm key against the Rust field; a mismatch would refuse to start every gateway in a fleet.
• Part 2 unit tests: the bootstrap set is the union of both lists (trimmed, deduped); a bootstrap-only account is not selectable while still authenticating — the test for the escalation this design exists to prevent; resolution falls back to the driver default when absent or blank; a selectable request is honoured (including padded); a non-selectable or unknown request is rejected rather than silently defaulted; build_platform_config forwards a requested account and omits an unset one; the validator accepts both lists.
• Wiring tests, which the units did not cover: requested_pod_service_account reads the platform_config key (and ignores a non-string value), the read and the config resolve together, and the TOML-ingest test now pins both keys — a rename of either would otherwise refuse to start every gateway in a fleet with nothing red in CI.
• Startup validation: both lists and the pod default accept valid names and reject capitals, spaces, underscores, leading hyphens and over-long names.
• create_sandbox_preserves_invalid_argument_from_the_driver asserts the code, the unwrapped message, and that no sandbox record survives the rejection.
• driver_template_rejects_a_service_account_request_for_other_drivers covers Podman, Docker and VM.
• helm unittest deploy/helm/openshell — 8 new cases (each key omitted by default; each rendered; both rendered together; blank and non-string entries fail the render for both lists). Passing goes 97 → 105. The assertions are line-anchored, because a containment regex silently stops matching once a rendered array literal sits between the table header and the second key. The 6 failures in credential_drivers_test.yaml and gateway_config_test.yaml are pre-existing — I get the identical 6 against an unmodified origin/main worktree.
• mise run pre-commit passes, including clippy -D warnings, helm:lint, helm:docs:check and license:check.

Verified on a kind cluster

kind v1.34.0, agent-sandbox v0.5.0, Kubernetes compute driver, sandbox namespace os-sa, gateway built from this branch. Both parts are covered. The driver-assigned sandbox ServiceAccount is sa-openshell-sandbox; sa-openshell-sandbox-2 stands in for an identity assigned out-of-band.

To reproduce the issue's scenario I created a pod running sa-openshell-sandbox-2, annotated with a real sandbox id and owner-referenced to that sandbox's Sandbox CR — a sandbox pod whose ServiceAccount the driver did not choose — then called IssueSandboxToken with a pod-bound token for that account.

Accepted set	Presented account	Result
{sa-openshell-sandbox} (default)	sa-openshell-sandbox-2	PermissionDenied: SA token is not from an accepted sandbox service account
{sa-openshell-sandbox} (default)	sa-openshell-sandbox	bootstrap succeeded, sandbox JWT minted
+ sa-openshell-sandbox-2	sa-openshell-sandbox-2	bootstrap succeeded, sandbox JWT minted
+ sa-openshell-sandbox-2	sa-openshell-sandbox	bootstrap succeeded, sandbox JWT minted
+ " sa-openshell-sandbox-2 " (padded in values)	sa-openshell-sandbox-2	bootstrap succeeded — the ConfigMap keeps the padding, the accepted set holds the trimmed name

The rejection record names both sides, as the Definition of Done asks:

WARN openshell_server::auth::k8s_sa: K8s TokenReview principal is not an accepted
  sandbox service account username=system:serviceaccount:os-sa:sa-openshell-sandbox-2
  service_account=sa-openshell-sandbox-2 accepted_service_accounts={"sa-openshell-sandbox"}

Startup, with a name added:

INFO openshell_server: K8s ServiceAccount bootstrap authenticator enabled
  namespace=os-sa accepted_service_accounts={"sa-openshell-sandbox", "sa-openshell-sandbox-2"}

The pod default is genuinely a separate setting — sandboxes created after the additional name was configured still get exactly one ServiceAccount, the driver's:

POD                        SA
default--sa-probe          sa-openshell-sandbox
default--sa-probe-2        sa-openshell-sandbox
externally-owned-sandbox   sa-openshell-sandbox-2   # the out-of-band pod

And a blank list entry fails at render instead of silently vanishing:

$ helm upgrade ... --set 'server.drivers.kubernetes.additionalBootstrapServiceAccountNames[1]='
Error: UPGRADE FAILED: execution error at (openshell/templates/statefulset.yaml:18:8):
  server.drivers.kubernetes.additionalBootstrapServiceAccountNames entries must be non-empty strings

Part 2: selection and the escalation boundary

Gateway configured with additional_bootstrap_service_account_names = ["sa-openshell-sandbox-external"] and selectable_service_account_names = ["sa-openshell-sandbox-2"], so the two lists are distinguishable. Startup:

INFO openshell_server: K8s ServiceAccount bootstrap authenticator enabled
  namespace=os-sa accepted_service_accounts={"sa-openshell-sandbox", "sa-openshell-sandbox-2", "sa-openshell-sandbox-external"}

CreateSandbox with SandboxTemplate.service_account_name:

Requested	Result
(unset)	created; pod runs sa-openshell-sandbox
sa-openshell-sandbox-2 (selectable)	created; pod runs sa-openshell-sandbox-2
sa-openshell-sandbox-external (bootstrap-only)	InvalidArgument: service_account_name 'sa-openshell-sandbox-external' is not selectable on this gateway; selectable accounts are {"sa-openshell-sandbox", "sa-openshell-sandbox-2"}
nope-not-a-sa	InvalidArgument, same shape

POD                    SA
default--sel-default   sa-openshell-sandbox
default--sel-ok        sa-openshell-sandbox-2

The third row is the escalation this design prevents: an account enrolled so an externally-owned pod can bootstrap is not on the menu for callers. And bootstrap still works independently of selectability:

Pod's account	Bootstrap
sa-openshell-sandbox-2 (selectable, driver-assigned via the request)	JWT minted
sa-openshell-sandbox-external (bootstrap-only, assigned out-of-band)	JWT minted
sa-openshell-sandbox-nope (in neither list)	PermissionDenied: SA token is not from an accepted sandbox service account

The first run of this returned Internal rather than InvalidArgument for the two rejections, which is what prompted the third commit; re-verified after the fix.

Part 2

Verified against the exact commit on this branch: the deployed binary's SHA-256 matches the local build from 666c6b8b with a clean tree. Gateway configured with additional_bootstrap_service_account_names = ["sa-openshell-sandbox-external"] and selectable_service_account_names = ["sa-openshell-sandbox-2"], so the two lists are distinguishable.

INFO K8s ServiceAccount bootstrap authenticator enabled namespace=os-sa
  accepted_service_accounts={"sa-openshell-sandbox", "sa-openshell-sandbox-2", "sa-openshell-sandbox-external"}

CreateSandbox with SandboxTemplate.service_account_name:

Requested	Result
(unset)	created; pod runs sa-openshell-sandbox
sa-openshell-sandbox-2 (selectable)	created; pod runs sa-openshell-sandbox-2
sa-openshell-sandbox-external (bootstrap-only)	InvalidArgument: service_account_name '...' is not selectable on this gateway
sa-openshell-sandbox-nope (unknown)	InvalidArgument, same shape
Openshell_Sandbox (malformed)	InvalidArgument: template.service_account_name must be a valid Kubernetes ServiceAccount name (DNS-1123 subdomain, at most 253 characters)
" " (present but blank)	rejected, not defaulted
2000 characters	InvalidArgument: template.service_account_name exceeds maximum length (2000 > 1024)

The third row is the escalation the two-list split exists to prevent. Only the two successful creates produced a Sandbox CR — the five rejections left nothing behind, which is what moving the check into validate_sandbox_create buys. The caller gets its own requested name and no more; the selectable set goes to the gateway log:

WARN rejected a sandbox ServiceAccount request that is not selectable
  sandbox_id=a52a3b34-… requested_service_account=Some("sa-openshell-sandbox-nope")
  selectable_service_accounts={"sa-openshell-sandbox", "sa-openshell-sandbox-2"}

Bootstrap still works independently of selectability:

Pod's account	Bootstrap
sa-openshell-sandbox-2 (selectable, assigned because the request asked for it)	JWT minted
sa-openshell-sandbox-external (bootstrap-only, assigned out-of-band)	JWT minted
sa-openshell-sandbox-nope (in neither list)	PermissionDenied: SA token is not from an accepted sandbox service account

Startup validation is a new way for a gateway to refuse to boot, so it was worth confirming: selectableServiceAccountNames: [Bad_Name] leaves the pod in CrashLoopBackOff with selectable_service_account_names entry 'Bad_Name' is not a valid Kubernetes ServiceAccount name.

Two limitations the cluster run confirmed rather than fixed, both now documented: the driver provisions only its own account, so a selected account must already exist in every namespace the gateway uses — which rules out workspace_mode = "managed", where namespaces are created on demand — and a separately deployed driver process has no flag for the selectable list, so it refuses every non-default request.

The issue carries test:e2e-kubernetes, and the triage suggested E2E coverage for bootstrap with a non-default accepted ServiceAccount. I have not added an e2e/rust test. The fixture needs a pod whose ServiceAccount the driver does not assign, which is a new shape for that harness — workspace_namespace_operator.rs is the closest template, since it exercises the namespace half of the same function and already hand-creates a ServiceAccount. Happy to add it in this PR rather than as a follow-up if you'd prefer.

Definition of Done

• Bootstrap auth accepts a configured set; a single value remains the default and is unchanged in behaviour
• Pod-default account and accepted-for-auth set are distinct settings; the driver still stamps exactly one name
• Rejection logging identifies the presented principal and the accepted set
• Tests: single configured account (accept + reject), multiple configured accounts (each accepted, non-member rejected)
• Helm chart can express the set, with the existing single-value setting still supported
• Gateway configuration documentation updated
• Beyond the DoD — part 2: the pod's ServiceAccount is selectable per sandbox, constrained to an operator-configured set

Checklist

• Conventional commit, signed off (DCO)
• Default behaviour unchanged
• Unit and Helm chart tests added
• E2E tests — see the note above

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[bjw123]

I have read the DCO document and I hereby sign the DCO.

[bjw123]

recheck