feat(gateway): ContainerLab egress canary and RBAC fix - #386
Conversation
a336660 to
4a9a0ff
Compare
Phase E of #865, per docs/plans/865-edge-gateway-nat66-egress.md §6/§8. Fixed first, found by trying to wire this up: config/gateway/rbac.yaml and config/router/rbac.yaml never granted access to networkegresspolicies (or, for galactic-router, networkgateways) -- Phase D's two new reconcilers would hit RBAC-forbidden errors in any real deployment, not just this lab. Lab wiring: - GALACTIC_GATEWAY_EGRESS_ADDRESS/_SID on both gateway nodes. egress_sid reuses each node's own ingress locator (only the top 64 bits are ever compared). masq_addr is a new, dedicated loopback address, not a reuse of the node's existing BGP-session address -- reusing it would make the fail-closed egress-return dispatch also claim and drop the node's own inbound BGP TCP traffic. - NetworkEgressPolicy for ns10's iad attachment (not ns60, the ingress canary -- ns60 runs nginx, no ping; ns10 runs netshoot). Only ns10's iad pod can egress in this lab: each site is a separate Kind cluster, so NetworkGateway/NetworkEgressPolicy only exist where iad's own cluster can see them. - scripts/verify-egress.sh + task verify:egress: pings tr1's own loopback (already BGP-announced, outside any SRv6/uSID space) from ns10's iad pod. - deploy-system.sh's CRD-install list was missing networkegresspolicies entirely -- fixed. Telemetry/quota: confirmed, not added. edgemetrics.Collector iterates DropReasonCount/DropReasonNames generically, so Phase B's new egress drop reasons are already exposed with no changes here. QuotaEnforcer/TelemetryEmitter stay Noop per the plan's own Phase E deferral. Not run live: bringing up three Kind clusters plus the full FRR/BGP mesh is a multi-minute operation not attempted in this session. Validated instead with `kubectl kustomize` against config/gateway/, config/router/, and (after simulating the deploy-time docker cp step) the affected lab overlays -- all build clean. The lab also can't fully deploy until datum-cloud/network#15 merges and tags: deploy- system.sh pulls CRDs from the tagged upstream repo, not this session's local checkout.
da9d2c5 to
41aa3c1
Compare
|
Approved, but one thing to sort before this merges: the canary sends the one protocol the egress datapath drops.
// internal/plumbing/ebpf/edgeprog/edgenat.c:1116
if (inner->nexthdr != EDGE_IPPROTO_TCP && inner->nexthdr != EDGE_IPPROTO_UDP) {
count_drop(DROP_REASON_MALFORMED_EGRESS_FORWARD);
return XDP_DROP;
}So Two follow-on notes while you're in there: The netshoot-over-nginx reasoning in the README and the policy manifest is on the wrong axis — it's not "netshoot has Reachability alone doesn't prove masquerade happened. A TCP connection to Separately, the ICMPv6 fail-closed drop on the return side is filed as its own issue — that one is a deliberate deferral, not a bug in this PR. |
Summary
Phase E of #865, per docs/plans/865-edge-gateway-nat66-egress.md §6/§8.
Stacked on #385 (Phase D). Final phase in the #865 stack.
Fixed first, found by trying to wire this up:
config/gateway/rbac.yamland
config/router/rbac.yamlnever granted access tonetworkegresspolicies(or, forgalactic-router,networkgateways)-- Phase D's two new reconcilers would hit RBAC-forbidden errors in
any real deployment, not just this lab.
Lab wiring
GALACTIC_GATEWAY_EGRESS_ADDRESS/_SIDon both gateway nodes.egress_sidreuses each node's own ingress locator (only the top 64bits are ever compared).
masq_addris a new, dedicated loopbackaddress, not a reuse of the node's existing BGP-session address --
reusing it would make the fail-closed egress-return dispatch also
claim and drop the node's own inbound BGP TCP traffic.
NetworkEgressPolicyforns10's iad attachment (notns60, theingress canary --
ns60runsnginx, noping;ns10runsnetshoot). Onlyns10's iad pod can egress in this lab: each siteis a separate Kind cluster, so
NetworkGateway/NetworkEgressPolicyonly exist where iad's own cluster can see them.
scripts/verify-egress.sh+task verify:egress: pingstr1's ownloopback (already BGP-announced, outside any SRv6/uSID space) from
ns10's iad pod.deploy-system.sh's CRD-install list was missingnetworkegresspoliciesentirely -- fixed.Telemetry/quota
Confirmed, not added.
edgemetrics.CollectoriteratesDropReasonCount/DropReasonNamesgenerically, so Phase B's newegress drop reasons are already exposed with no changes here.
QuotaEnforcer/TelemetryEmitterstay Noop per the plan's own Phase Edeferral.
Not run live
Bringing up three Kind clusters plus the full FRR/BGP mesh is a
multi-minute operation not attempted in this session. Validated
instead with
kubectl kustomizeagainstconfig/gateway/,config/router/, and (after simulating the deploy-timedocker cpstep) the affected lab overlays -- all build clean. The lab also can't
fully deploy until datum-cloud/network#15 merges and tags:
deploy-system.shpulls CRDs from the tagged upstream repo, not thissession's local checkout.
Testing
kubectl kustomizeagainstconfig/gateway/,config/router/, andthe affected lab overlays -- all build clean, env vars land
correctly.
task lint(0 issues), shell syntax checks (bash -n) on bothedited/new scripts.
🤖 Generated with Claude Code