Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -439,10 +439,16 @@ helm -n openshell get values openshell | grep sandboxNamespace

Then inspect sandbox resources in that namespace.

Check the configured sandbox service account when TokenReview bootstrap or
Check the accepted sandbox service accounts when TokenReview bootstrap or
sandbox registration fails. Helm creates a dedicated sandbox service account by
default and writes it to `[openshell.drivers.kubernetes].service_account_name`;
the gateway rejects projected tokens from other service accounts.
the gateway rejects projected tokens from any service account outside that name
plus `additional_bootstrap_service_account_names` and
`selectable_service_account_names`. A create that fails with
`service_account_name '<name>' is not selectable` is asking for an account
missing from `selectable_service_account_names`; the gateway log records the
requested name and the selectable set. The `gateway.toml` dump below shows both
settings.

```bash
helm -n openshell get values openshell | grep -A3 sandboxServiceAccount
Expand Down
2 changes: 1 addition & 1 deletion architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ Sandbox secrets are gateway-signed JWTs bound to a single sandbox ID. Docker,
Podman, and VM drivers deliver the initial token through supervisor-only
runtime material; Kubernetes supervisors exchange a projected ServiceAccount
token through `IssueSandboxToken`. The gateway validates that projected token
with Kubernetes `TokenReview`, requires the configured sandbox service account,
with Kubernetes `TokenReview`, requires an accepted sandbox service account,
checks the returned pod binding against the live pod UID, and verifies the pod's
controlling `Sandbox` ownerReference against the live Sandbox CR UID and
sandbox-id label before minting the gateway JWT. The bootstrap path accepts
Expand Down
3 changes: 2 additions & 1 deletion crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,8 @@ The driver injects gateway callback configuration, sandbox identity, TLS client
material, and the supervisor SSH socket path into the workload. Driver-owned
values must override image-provided environment variables.

Sandbox pods run as `service_account_name` and keep
Sandbox pods run as `service_account_name`, or as an account the caller
selected from `selectable_service_account_names` and keep
`automountServiceAccountToken: false`. The only Kubernetes token exposed to the
supervisor is an explicit, audience-bound projected token mounted at
`/var/run/secrets/openshell/token` for the one-shot `IssueSandboxToken`
Expand Down
418 changes: 416 additions & 2 deletions crates/openshell-driver-kubernetes/src/config.rs

Large diffs are not rendered by default.

140 changes: 138 additions & 2 deletions crates/openshell-driver-kubernetes/src/driver.rs
Original file line number Diff line number Diff line change
Expand Up @@ -486,6 +486,9 @@ impl KubernetesComputeDriver {
config
.validate_upstream_proxy_config()
.map_err(KubernetesDriverError::Precondition)?;
config
.validate_service_account_names()
.map_err(KubernetesDriverError::Precondition)?;
let base_config = match kube::Config::incluster() {
Ok(c) => c,
Err(_) => kube::Config::infer()
Expand Down Expand Up @@ -782,7 +785,10 @@ impl KubernetesComputeDriver {
let sa_api: Api<ServiceAccount> = Api::namespaced(self.client.clone(), namespace);
let sa = ServiceAccount {
metadata: ObjectMeta {
name: Some(self.config.service_account_name.clone()),
// Trimmed to match the name pods reference: the accepted set
// and the resolved pod value are both trimmed, so creating the
// untrimmed spelling would provision an account nothing uses.
name: Some(self.config.service_account_name.trim().to_string()),
labels: Some(BTreeMap::from([(
LABEL_MANAGED_BY.to_string(),
LABEL_MANAGED_BY_VALUE.to_string(),
Expand Down Expand Up @@ -1194,6 +1200,26 @@ impl KubernetesComputeDriver {
}))
}

/// Resolve the `ServiceAccount` for a sandbox's pod, logging a rejection.
///
/// The log carries the selectable set and the rejection does not: an
/// operator needs to see which name was asked for and what the gateway
/// offers, while a caller only needs to know its own request was refused.
fn resolve_requested_service_account(&self, sandbox: &Sandbox) -> Result<String, String> {
let requested = requested_pod_service_account(sandbox);
self.config
.resolve_pod_service_account(requested.as_deref())
.inspect_err(|err| {
warn!(
sandbox_id = %sandbox.id,
requested_service_account = ?requested,
selectable_service_accounts = ?self.config.selectable_pod_service_account_names(),
error = %err,
"rejected a sandbox ServiceAccount request that is not selectable"
);
})
}

pub async fn validate_sandbox_create(&self, sandbox: &Sandbox) -> Result<(), tonic::Status> {
let _ = self
.validate_driver_config_for_sandbox(sandbox)
Expand All @@ -1207,6 +1233,11 @@ impl KubernetesComputeDriver {
.map_err(tonic::Status::invalid_argument)?;
}
}
// Reject an unselectable ServiceAccount here, before the gateway
// persists the sandbox record or mints its JWT, so nothing has to be
// rolled back. `create_sandbox` resolves the name again for the value.
self.resolve_requested_service_account(sandbox)
.map_err(tonic::Status::invalid_argument)?;
let gpu_requirements = sandbox
.spec
.as_ref()
Expand Down Expand Up @@ -1352,6 +1383,14 @@ impl KubernetesComputeDriver {
validate_kubernetes_dns1123_label(&sandbox.name, "sandbox name")
.map_err(KubernetesDriverError::InvalidArgument)?;

// Resolved before any namespace or secret is created, so a request
// rejected here leaves nothing behind. The resolved name is always a
// member of the bootstrap authenticator's accepted set, because the
// selectable set is a subset of it.
let service_account_name = self
.resolve_requested_service_account(sandbox)
.map_err(KubernetesDriverError::InvalidArgument)?;

let name = sandbox.name.as_str();
let workspace = sandbox.workspace.as_str();
self.validate_workspace_namespace(workspace)?;
Expand Down Expand Up @@ -1417,7 +1456,7 @@ impl KubernetesComputeDriver {
proxy_auth_secret_key: self.config.proxy_auth_secret_key.as_deref(),
proxy_auth_allow_insecure: self.config.proxy_auth_allow_insecure == Some(true),
proxy_connect_by_hostname: self.config.proxy_connect_by_hostname == Some(true),
service_account_name: &self.config.service_account_name,
service_account_name: &service_account_name,
sandbox_id: &sandbox.id,
sandbox_name: &sandbox.name,
grpc_endpoint: &self.config.grpc_endpoint,
Expand Down Expand Up @@ -3267,6 +3306,19 @@ impl Default for SandboxPodParams<'_> {
}
}

/// The `ServiceAccount` a sandbox's template asked for, if it asked for one.
///
/// Split out from `create_sandbox` so the read is testable without a cluster:
/// it is the only thing standing between a caller's request and the identity
/// its pod runs as.
fn requested_pod_service_account(sandbox: &Sandbox) -> Option<String> {
sandbox
.spec
.as_ref()
.and_then(|spec| spec.template.as_ref())
.and_then(|template| platform_config_string(template, "service_account_name"))
}

fn validate_sidecar_proxy_identity(
params: &SandboxPodParams<'_>,
) -> Result<(), KubernetesDriverError> {
Expand Down Expand Up @@ -7064,6 +7116,90 @@ mod tests {
);
}

fn sandbox_requesting_service_account(value: Option<Value>) -> Sandbox {
Sandbox {
id: "sandbox-123".to_string(),
spec: Some(SandboxSpec {
template: Some(SandboxTemplate {
platform_config: value.map(|v| Struct {
fields: std::iter::once(("service_account_name".to_string(), v)).collect(),
}),
..Default::default()
}),
..Default::default()
}),
..Default::default()
}
}

/// This read is the only thing between a caller's request and the identity
/// its pod runs as, so it is worth pinning on its own.
#[test]
fn requested_pod_service_account_reads_the_platform_config_key() {
let sandbox = sandbox_requesting_service_account(Some(Value {
kind: Some(Kind::StringValue("openshell-sandbox-3".to_string())),
}));

assert_eq!(
requested_pod_service_account(&sandbox).as_deref(),
Some("openshell-sandbox-3")
);
}

#[test]
fn requested_pod_service_account_is_none_when_unset() {
assert_eq!(requested_pod_service_account(&Sandbox::default()), None);
assert_eq!(
requested_pod_service_account(&sandbox_requesting_service_account(None)),
None
);
}

/// A non-string value must not read as a request, or a malformed one would
/// silently take the driver default.
#[test]
fn requested_pod_service_account_ignores_a_non_string_value() {
let sandbox = sandbox_requesting_service_account(Some(Value {
kind: Some(Kind::NumberValue(42.0)),
}));

assert_eq!(requested_pod_service_account(&sandbox), None);
}

/// The join between the read and the config: a selectable request lands on
/// the pod, a bootstrap-only one is refused.
#[test]
fn requested_service_account_resolves_against_the_selectable_set() {
let config = KubernetesComputeConfig {
service_account_name: "openshell-sandbox".to_string(),
additional_bootstrap_service_account_names: vec![
"openshell-sandbox-external".to_string(),
],
selectable_service_account_names: vec!["openshell-sandbox-3".to_string()],
..Default::default()
};
let requested = |name: &str| {
let sandbox = sandbox_requesting_service_account(Some(Value {
kind: Some(Kind::StringValue(name.to_string())),
}));
config.resolve_pod_service_account(requested_pod_service_account(&sandbox).as_deref())
};

assert_eq!(
requested("openshell-sandbox-3").unwrap(),
"openshell-sandbox-3"
);
assert!(requested("openshell-sandbox-external").is_err());
assert_eq!(
config
.resolve_pod_service_account(
requested_pod_service_account(&Sandbox::default()).as_deref()
)
.unwrap(),
"openshell-sandbox"
);
}

#[test]
fn platform_config_bool_extracts_value() {
let template = SandboxTemplate {
Expand Down
3 changes: 2 additions & 1 deletion crates/openshell-driver-kubernetes/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ pub use config::{
AppArmorProfile, DEFAULT_GATEWAY_ID, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesSidecarConfig,
ManagedSshIngressConfig, OperatorNamespaceAllowlist, SupervisorSideloadMethod,
SupervisorTopology, WorkspaceMode, managed_namespace_prefix,
SupervisorTopology, WorkspaceMode, is_service_account_name, managed_namespace_prefix,
service_account_name_set,
};
pub use driver::{KubernetesComputeDriver, KubernetesDriverError};
pub use grpc::ComputeDriverService;
9 changes: 9 additions & 0 deletions crates/openshell-driver-kubernetes/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,15 @@ async fn main() -> Result<()> {
operator_namespace_label: args.operator_namespace_label,
operator_namespace_file: args.operator_namespace_file,
service_account_name: args.sandbox_service_account,
// Bootstrap TokenReview runs in the gateway, which reads its own
// `[openshell.drivers.kubernetes]` table for the accepted set.
additional_bootstrap_service_account_names: Vec::new(),
// This binary has no config file and no flag for the selectable
// set, and resolution runs in the driver, so per-sandbox selection
// is unavailable out-of-process: any account other than
// --sandbox-service-account is rejected. Selection needs the
// gateway's built-in driver, which reads the TOML table.
selectable_service_account_names: Vec::new(),
default_image: args.sandbox_image.unwrap_or_default(),
image_pull_policy: args.sandbox_image_pull_policy.unwrap_or_default(),
image_pull_secrets: args.sandbox_image_pull_secrets,
Expand Down
Loading
Loading