docs: refine BEP-1053 ROUTER frontend mode after design review

[achimnol]

Refines BEP-1053: ROUTER Frontend Mode across two design-review rounds (2026-06-21, 2026-07-15) and restructures it into the segmented BEP format so a first-time reader can follow intent → key decisions → detailed trade-offs.

Document restructure

The monolithic BEP is now a main overview + four sub-documents (per the BEP segmentation guide), with mermaid diagrams throughout:

Document	Contents
main	Motivation & design intent, architecture at a glance, Decision Log (15 entries), Open Questions, phased plan
architecture	Hierarchical routing model, per-level weight normalization, KV-cache tier scoping, per-replica-group instance lifecycle + in-memory vs subprocess trade-off, HA
coordinator	FrontendMode.ROUTER, registration validation, per-node liveness, desired-state tables, /v2/routers/*, snapshot + events, failure model, key custody
manager	Publication / model API key entities, Manager-side data model, RBAC shrink reconciler, BEP-1049 interplay
migration	Adding a ROUTER surface to an existing AppProxy deployment, version gates, rollback

Round 1 (2026-06-21): resolved the four original open questions

• Placement & HA: endpoint placement via existing scaling-group routing; per-node liveness (ephemeral node_id, valkey liveness set, derived nodes) exposed via REST + Prometheus + fleet view.
• Keys & visibility: explicit-at-issuance / RBAC-validated / shrink-only visibility with a periodic shrink reconciler; two-tier revocation + opt-in strict all-live-nodes confirmation; hash-only key custody (plain SHA-256, plaintext shown once).
• Data model & propagation: per-authority publication scope (multi-authority = caller composition); single per-authority revision + conditional snapshot polling; first-class aliases gated per key; atomic key rotation; control_mode to prevent a BEP-1049 dual-writer conflict; per-node best-effort rate_limit; ratio as relative weight (0 = drain); authority discovery via list_workers().

Round 2 (2026-07-15): hierarchy, fail-fast, and the Manager side

• Hierarchical (chained) routing adopted; the flattened weighted pool is superseded. Realized inside the ROUTER worker as per-replica-group router instances, so continuum-router's replica-level KV machinery (prefix-aware routing, KV index, disaggregated serving) is preserved. Instance substrate (in-memory scopes vs subprocess pool) is recorded as an Open Question with a recommendation (in-memory first, behind a substrate-agnostic seam).
• Replica-group id / traffic_weight propagate over the existing circuit wire (replica_groups on the circuit payload, replica_group_id per route) — folds the BA-6233 "weight never reaches AppProxy" follow-up into this proposal; rollout ramps become ordinary route updates.
• Colocation fail-fast: registration rejects mixing a ROUTER worker with stock inference workers on one coordinator (review request); the migration document covers the dedicated-coordinator path for existing deployments.
• Manager-side data model specified: normalized model_publications / model_publication_names / model_publication_endpoints / model_api_keys, CASCADE backstops + service-layer lifecycle hooks — vs the coordinator's deliberately wire-shaped JSONB mirror (rationale recorded).
• Deterministic mixed-node_id liveness (nodes = liveness-set count + legacy counter) so rolling upgrades of stock workers stay correct; coordinator schema recorded as freely evolvable (BEP-1005 has no work plan).
• All four review threads answered in-line; set-operator symbols replaced per review.

Pairing / cross-repo status

The worker-side counterpart is lablup/continuum-router#748 (docs/en/architecture/appproxy-worker.md). The shipped router implementation (epic lablup/continuum-router#804) implements the now-superseded flattened selection; its rework to the hierarchical model — and the doc re-sync — is tracked on #804.

https://claude.ai/code/session_01DRHPAAYiQxQyMRkzic4Dda
https://claude.ai/code/session_01P38DWyutrMv8kuC8XKjyux

[Copilot]

Pull request overview

This PR updates the BEP-1053 design document to reflect post–design-review decisions for ROUTER frontend mode, clarifying routing placement/HA, key custody/visibility, and desired-state propagation semantics across Manager ↔ coordinator ↔ router worker.

Changes:

• Refines terminology and core entities (publications, aliases, authority/node) and documents per-authority scoping.
• Updates key issuance/visibility and revocation semantics (explicit-at-issuance shrink-only visibility, two-tier revocation, optional strict all-live-nodes confirmation).
• Expands coordinator/worker design details (per-node liveness via node_id, conditional snapshot polling via known_revision, and hash-only key custody).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jopemachine]

Could we fail fast instead of silently misrouting? For example, when an app proxy worker is registered, could we raise an error if a ROUTER and an app proxy worker type that can receive legacy inference traffic are registered at the same time?

[achimnol]

Agreed. Adopted as-is in the second review round (2026-07-15). Clarified that:

• Registration now rejects with an explicit error any combination that would make one coordinator host both a ROUTER worker and a stock worker whose accepted_traffics include inference, in either registration order. Interactive stock workers still colocate freely.

Doc relocation:

• Since the doc was restructured in 4d3f1ef, this now lives in BEP-1053/coordinator.md ("FrontendMode.ROUTER and registration") with a Decision Log entry in the main doc, and BEP-1053/migration.md documents the operational path for existing deployments (a dedicated coordinator for the ROUTER authority; rollback steps included).

[jopemachine]

Is there a reason to keep this in a JSONB column instead of normalizing it into a separate table?

[achimnol]

Reasoning for keeping aliases as an embedded column rather than a child table, on the coordinator specifically:

• Every read is whole-entity. The only consumers (the snapshot builder and the model-updated event) always emit the full publication. A child table would add a join/aggregate to every snapshot and event build, only to reassemble the exact array we stored.
• Every write is whole-entity. Mutations arrive as full-state upserts (PUT with complete state, last-writer-wins), so a column makes an upsert a single-row write; a child table turns each upsert into a delete-and-reinsert of child rows inside a transaction, for the same end state.
• Separation of concern. The heavy burden of validation (e.g., uniqueness of alias) is already handled by the Manager.
  • model_publication_names stores primary + aliases as rows with an is_primary flag and UNIQUE (authority, name) (BEP-1053/manager.md, 4d3f1ef).
• This coordinator-internal design can be updated whenever we want in the future, though, without impacting the wire protocol. Even we proceed with BEP-1005 (merging the coordinator into the manager), the coordinator-specific mirror could be migrated as a "cache" layer.

[jopemachine]

For extensibility, I think it would be good to introduce a middleware layer and align the implementation with it.

[achimnol]

Recorded the trade-off as an Open Question (BEP-1053/coordinator.md "Snapshot endpoint" since 4d3f1ef).

The catch with a generic conditional-GET middleware: it can only compare after the handler has already built the response body, so it saves response bytes but not the DB read. Skipping the read requires a per-route "current version" resolver — which is the known_revision check under another name, plus indirection.

Alternative way:

• Adopt the standard ETag/If-None-Match header form, implemented as a small handler-level helper (the authority revision as a strong ETag, early 304 before the state query) rather than a blanket middleware.
  • It keeps standard HTTP semantics and lets future snapshot-style endpoints reuse the helper, without pretending the version check can be route-agnostic. The router-side client change is a one-line header swap, which we've confirmed is acceptable.

Checks on ecosystem compatibility:

• The ETag header design is compatible with any client. None of the public LLM APIs (OpenAI, OpenRouter, Anthropic) use HTTP conditional caching on their data planes (they cache by request-body keys such as prompt caching, OpenRouter's X-OpenRouter-Cache-*) and our ETag use would sit only on the private coordinator↔worker control-plane endpoint, which the end-user clients never touch.

[jopemachine]

Will the same table be added to the manager database with the same schema as well?

[achimnol]

Yes on persistence, no on "same schema".

The difference is intentional (see the sibling JSONB thread for the coordinator half). The stack splits state three ways:

• The Manager owns the normalized source of truth and validation (model_publications + model_publication_names with UNIQUE (authority, name) as a DB constraint model_publication_endpoints with CASCADE-backstop FKs + model_api_keys, specified in BEP-1053/manager.md since 4d3f1ef).
• The coordinator owns a wire-shaped denormalized mirror whose only job is entity-granular replay to stateless router nodes.
• The router holds everything in memory only (its key store is explicitly never persisted) and rebuilds from the coordinator's snapshot on (re-)registration.