Skip to content

feat(gateway): ContainerLab egress canary and RBAC fix - #386

Open
privateip wants to merge 1 commit into
feat/865-egress-phase-dfrom
feat/865-egress-phase-e
Open

feat(gateway): ContainerLab egress canary and RBAC fix#386
privateip wants to merge 1 commit into
feat/865-egress-phase-dfrom
feat/865-egress-phase-e

Conversation

@privateip

Copy link
Copy Markdown
Contributor

Summary

Phase E of #865, per docs/plans/865-edge-gateway-nat66-egress.md §6/§8.
Stacked on #385 (Phase D). Final phase in the #865 stack.

Fixed first, found by trying to wire this up: config/gateway/rbac.yaml
and config/router/rbac.yaml never granted access to
networkegresspolicies (or, for galactic-router, networkgateways)
-- Phase D's two new reconcilers would hit RBAC-forbidden errors in
any real deployment, not just this lab.

Lab wiring

  • GALACTIC_GATEWAY_EGRESS_ADDRESS/_SID on both gateway nodes.
    egress_sid reuses each node's own ingress locator (only the top 64
    bits are ever compared). masq_addr is a new, dedicated loopback
    address, not a reuse of the node's existing BGP-session address --
    reusing it would make the fail-closed egress-return dispatch also
    claim and drop the node's own inbound BGP TCP traffic.
  • NetworkEgressPolicy for ns10's iad attachment (not ns60, the
    ingress canary -- ns60 runs nginx, no ping; ns10 runs
    netshoot). Only ns10's iad pod can egress in this lab: each site
    is a separate Kind cluster, so NetworkGateway/NetworkEgressPolicy
    only exist where iad's own cluster can see them.
  • scripts/verify-egress.sh + task verify:egress: pings tr1's own
    loopback (already BGP-announced, outside any SRv6/uSID space) from
    ns10's iad pod.
  • deploy-system.sh's CRD-install list was missing
    networkegresspolicies entirely -- fixed.

Telemetry/quota

Confirmed, not added. edgemetrics.Collector iterates
DropReasonCount/DropReasonNames generically, so Phase B's new
egress drop reasons are already exposed with no changes here.
QuotaEnforcer/TelemetryEmitter stay Noop per the plan's own Phase E
deferral.

Not run live

Bringing up three Kind clusters plus the full FRR/BGP mesh is a
multi-minute operation not attempted in this session. Validated
instead with kubectl kustomize against config/gateway/,
config/router/, and (after simulating the deploy-time docker cp
step) the affected lab overlays -- all build clean. The lab also can't
fully deploy until datum-cloud/network#15 merges and tags:
deploy-system.sh pulls CRDs from the tagged upstream repo, not this
session's local checkout.

Testing

  • kubectl kustomize against config/gateway/, config/router/, and
    the affected lab overlays -- all build clean, env vars land
    correctly.
  • task lint (0 issues), shell syntax checks (bash -n) on both
    edited/new scripts.
  • No Go code changed in this phase.

🤖 Generated with Claude Code

@privateip
privateip requested a review from a team as a code owner August 13, 2026 21:13
@privateip
privateip requested review from ecv and removed request for a team August 13, 2026 21:13
@privateip privateip linked an issue Aug 13, 2026 that may be closed by this pull request
@privateip
privateip force-pushed the feat/865-egress-phase-d branch from a336660 to 4a9a0ff Compare August 14, 2026 01:16
Phase E of #865, per docs/plans/865-edge-gateway-nat66-egress.md §6/§8.

Fixed first, found by trying to wire this up: config/gateway/rbac.yaml
and config/router/rbac.yaml never granted access to
networkegresspolicies (or, for galactic-router, networkgateways) --
Phase D's two new reconcilers would hit RBAC-forbidden errors in any
real deployment, not just this lab.

Lab wiring:
- GALACTIC_GATEWAY_EGRESS_ADDRESS/_SID on both gateway nodes.
  egress_sid reuses each node's own ingress locator (only the top 64
  bits are ever compared). masq_addr is a new, dedicated loopback
  address, not a reuse of the node's existing BGP-session address --
  reusing it would make the fail-closed egress-return dispatch also
  claim and drop the node's own inbound BGP TCP traffic.
- NetworkEgressPolicy for ns10's iad attachment (not ns60, the
  ingress canary -- ns60 runs nginx, no ping; ns10 runs netshoot).
  Only ns10's iad pod can egress in this lab: each site is a separate
  Kind cluster, so NetworkGateway/NetworkEgressPolicy only exist
  where iad's own cluster can see them.
- scripts/verify-egress.sh + task verify:egress: pings tr1's own
  loopback (already BGP-announced, outside any SRv6/uSID space) from
  ns10's iad pod.
- deploy-system.sh's CRD-install list was missing
  networkegresspolicies entirely -- fixed.

Telemetry/quota: confirmed, not added. edgemetrics.Collector iterates
DropReasonCount/DropReasonNames generically, so Phase B's new egress
drop reasons are already exposed with no changes here.
QuotaEnforcer/TelemetryEmitter stay Noop per the plan's own Phase E
deferral.

Not run live: bringing up three Kind clusters plus the full FRR/BGP
mesh is a multi-minute operation not attempted in this session.
Validated instead with `kubectl kustomize` against config/gateway/,
config/router/, and (after simulating the deploy-time docker cp step)
the affected lab overlays -- all build clean. The lab also can't
fully deploy until datum-cloud/network#15 merges and tags: deploy-
system.sh pulls CRDs from the tagged upstream repo, not this
session's local checkout.
@privateip
privateip force-pushed the feat/865-egress-phase-e branch from da9d2c5 to 41aa3c1 Compare August 14, 2026 01:16
@ecv

ecv commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Approved, but one thing to sort before this merges: the canary sends the one protocol the egress datapath drops.

scripts/verify-egress.sh proves the path with ping_pod ... -6 (ICMP echo, via lib.sh:98). Phase B's handle_egress_forward drops any inner protocol that isn't TCP or UDP before it ever reaches the conn-table claim:

// internal/plumbing/ebpf/edgeprog/edgenat.c:1116
if (inner->nexthdr != EDGE_IPPROTO_TCP && inner->nexthdr != EDGE_IPPROTO_UDP) {
	count_drop(DROP_REASON_MALFORMED_EGRESS_FORWARD);
	return XDP_DROP;
}

So task verify:egress fails 3/3 on the first real run, and since it's now in the verify aggregate, it takes the whole lab verification with it. The "Not run live" note in the description is exactly why this survived.

Two follow-on notes while you're in there:

The netshoot-over-nginx reasoning in the README and the policy manifest is on the wrong axis — it's not "netshoot has ping", it's "netshoot has curl/nc". netshoot is still the right pick, just for a different reason.

Reachability alone doesn't prove masquerade happened. A TCP connection to tr1 that succeeds tells you packets got there; it doesn't tell you they were SNAT'd to fc00:0:9::2/fc00:0:a::2 rather than leaking out some other way. Something that observes the source address on the far side (an nc -l on tr1, or the egress conn-table / drop counters after the run) would close the same gap #368 describes for the ingress canary. That would also be the first real confirmation of #381's flagged backend_usid-read-off-the-wire assumption, which Phase D was supposed to prove and this phase was going to be the proof of.

Separately, the ICMPv6 fail-closed drop on the return side is filed as its own issue — that one is a deliberate deferral, not a bug in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Design: ContainerLab egress canary and RBAC gap (865 Phase E)

3 participants