Skip to content

fix(gateway): handle delayed Podman bridge listeners - #2874

Draft
elezar wants to merge 3 commits into
mainfrom
codex/fix-fedora-release-canary
Draft

fix(gateway): handle delayed Podman bridge listeners#2874
elezar wants to merge 3 commits into
mainfrom
codex/fix-fedora-release-canary

Conversation

@elezar

@elezar elezar commented Aug 21, 2026

Copy link
Copy Markdown
Member

Summary

Reproduce and fix the Fedora Release Canary startup failure in the nested Docker-to-Fedora-to-rootful-Podman topology. When Podman reports its future bridge gateway before netavark assigns the address, the gateway now uses a narrowly gated wildcard listener whose non-loopback traffic remains restricted to sandbox-callable gRPC methods.

Scope: This fallback is limited to nested containers: the OpenShell gateway must itself be running inside an outer Linux container, with the built-in rootful Podman driver operating inside that container. It does not apply when the gateway runs directly on a host, to rootless Podman, to explicit host_gateway_ip values, or to external compute drivers named podman.

Related Issue

No issue required: this is a focused fix for a reliably failing release-canary regression. Long-term driver-owned callback relay work remains tracked in #2540, including the nested-rootful use case documented in #2540 (comment).

Release Canary reference: https://github.com/NVIDIA/OpenShell/actions/runs/32448768823/job/96673135310

Initial PR E2E failure: https://github.com/NVIDIA/OpenShell/actions/runs/32484361009/job/96781677362

The initial PR job failed before gateway startup because the raw-artifact harness did not create /usr/share/openshell. The corrected harness subsequently reproduced and validated the listener path in the local Nix VM topology.

Changes

  • Add an opt-in PR E2E lane using PR-built CLI, gateway, and supervisor artifacts in the Fedora/rootful-Podman canary topology.
  • Confirm the future Podman bridge gateway is absent before gateway startup and assert that the scoped wildcard fallback is active.
  • Fall back only when the gateway is nested inside a Linux container, the trusted built-in rootful Podman driver reports its discovered private managed-bridge address, the primary listener is IPv4 loopback, and the exact bind fails with EADDRNOTAVAIL.
  • Default the wildcard socket to callback-only authorization while preserving primary scope for connections addressed to loopback.
  • Preserve loopback-only plaintext service routing by passing the accepted destination address through connection dispatch.
  • Fix raw-artifact E2E setup for Nix VM execution, including stable image staging and a valid sandbox name.
  • Document the security tradeoff and update Release Canary and gateway troubleshooting guidance.

Security impact

The fallback exposes the sandbox-callable gRPC surface on every IPv4 interface in the outer container namespace for the lifetime of that gateway process. It does not expose user, administrator, health, reflection, non-callback inference, or HTTP routes on non-loopback interfaces. Sandbox mTLS, JWT authentication, the generated RPC allowlist, and the surrounding container-network boundary remain required defenses.

Testing

  • mise run pre-commit
  • cargo test -p openshell-server gateway_listener --lib (20 passed)
  • cargo clippy -p openshell-server --lib -- -D warnings
  • Nix VM E2E: Ubuntu VM -> Docker -> Fedora systemd container -> rootful Podman; confirmed missing 10.89.0.1, healthy fallback startup, bridge creation, supervisor callback, and successful sandbox exec
  • bash -n e2e/nested-podman-callback.sh
  • git diff --check

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 21, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@elezar elezar added the test:e2e Requires end-to-end coverage label Aug 21, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied, but pull-request/2874 does not exist yet. A maintainer needs to comment /ok to test e2e39b416207323bc6ef34480f6134efc0ddf2f1 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

@elezar

elezar commented Aug 21, 2026

Copy link
Copy Markdown
Member Author

/ok-to-test e2e39b4

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar elezar changed the title test(ci): add nested Podman callback E2E fix(gateway): handle delayed Podman bridge listeners Aug 21, 2026
@github-actions

Copy link
Copy Markdown

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar

elezar commented Aug 21, 2026

Copy link
Copy Markdown
Member Author

/ok-to-test fa02b7d

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant