compute: two-runtime read isolation (interactive runtime)

[antiguru]

Prerequisite: #37884 (ComputeRuntimeRole + role-labeled metrics). The role enum, the role metric label, and the serve/register_with role threading now live there with every call site passing Solo. Its commit appears in this diff until it merges. After it merges, rebase onto main and this PR is left flipping the call sites to Maintenance/Interactive.

Stacks on #37880 (typed s/u/t-prefixed global ids in clusterd-test-driver specs), which makes the driver's bare transient export flag moot: a transient export is now just a t-prefixed id. #37880's commits appear in this diff until it merges; rebase onto main after.

A second in-process "interactive" compute runtime that serves reads against the arrangements the maintenance runtime maintains, isolating latency-sensitive reads from CPU-bound maintenance. Implements the two-runtime design (#37747).

The cross-runtime arrangement-sharing primitive lives entirely in Materialize (mz-compute::shared_trace, mz-row-spine::ArcBatch), so it builds against released differential-dataflow with no fork or [patch.crates-io]. The diff also carries the production Rc→Arc spine migration that cross-thread sharing needs. The publisher follows the controller's compaction through AllowCompaction rather than reading trace internals.

How it works:

• Maintained indexes publish their oks/errs arrangements into a per-process ArrangementSharingRegistry. A ComputeRuntimeRole { Solo, Maintenance, Interactive } distinguishes runtimes; Solo (default, single-runtime) is byte-behaviorally unchanged, metrics included.
• A process-level multiplexer presents one controller endpoint, routing peeks and one-shot work to interactive, maintained work to maintenance, exactly one PeekResponse per uuid, and each collection's frontiers only from the runtime that owns it (both runtimes install the internal logging dataflows, so this keeps the interactive runtime's empty copies from regressing the controller's per-collection frontier).
• The interactive runtime serves everything through the registry: fast-path index peeks read the published arrangement directly; slow-path query dataflows (e.g. introspection views) import the maintenance arrangements as real ArrangementFlavor::SharedTrace arrangements (not re-derived collections) and render joins/reduces over them; the query's own transient output is itself published so its result peek is served the same way. A spike verified a join over an imported shared arrangement is correct (the join cuts each trace at its own acknowledged batch boundary, so the shared handle's batches_through is always batch-aligned).
• Introspection reads come from maintenance. Maintenance publishes its logging/introspection indexes into the registry too (gated strictly on the Maintenance role, since those indexes bypass the normal export_index publish path). The interactive runtime runs with logging disabled and serves introspection peeks from maintenance's published copies, so introspection queries during hydration return promptly (possibly stale) instead of blocking behind maintenance.
• Notification-driven, no polling. The registry carries a per-interactive-worker coalescing SyncActivator + dirty-id inbox. A read whose dependency is not yet published/sealed is enqueued, not blocked; publication (insert) and seal (note_frontier, fired from the maintenance export's frontier probe) mark the id dirty and wake the interactive worker, which re-examines only the affected pending work. A query dataflow whose imported deps aren't published yet is deferred (not built) and built once all deps publish. The per-step process_peeks scan is removed on the interactive runtime.
• Controller plumbing: ENABLE_TWO_RUNTIME_COMPUTE (dyncfg, off by default in production) makes the controller launch replicas with a second interactive runtime (--interactive-compute-timely-config, its own worker ports). Enabled by default in the variable CI system parameters so the suite exercises the two-runtime path broadly.
• Failure model: shared-fate abort covers both runtimes (subprocess test); dropping a still-deferred interactive dataflow cancels cleanly rather than panicking.

Acceptance: a clusterd-test-driver workflow drives an interactive query dataflow that imports an unpublished maintenance index, scheduled before the index publishes (mirroring controller ordering), and its result peek resolves correctly only after publication — proving the defer→build→resolve read path is served off the maintenance worker.

A TwoRuntimeReadIsolation parallel-benchmark scenario measures read latency while the maintenance runtime is saturated by hydration churn. With two runtimes on, point-read p50 stays flat (~11ms) through the churn; the single-runtime baseline backlogs to tens of seconds.

Design and per-task plans: #37747, doc/developer/design/20260720_two_runtime_compute/stage2-detailed-plan.md.

Scope / follow-ups:

• Subscribe is out of scope: all interactive work is single-time, so the shared import applies no until/as_of coalescing (correct for one-shot peeks; a future subscribe migration must add it).
• A full SQL-level "introspection during hydration" acceptance test (versus the protocol-level clusterd-test-driver one here) is a follow-up; the mechanism it would exercise is in place.
• The interactive serving loop is a single step loop, so its read throughput has a ceiling (heavy scans and light point reads share it); the read-isolation win holds below that drain rate. Raising it is future work.
• Minor: the inline interactive peek records no index_peek_total_seconds; pre-existing println! debug lines in render.rs None arms.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DAfFVVJ8BwahdaAHkRtbf6

[antiguru]

This is an architectural commitment, not just a feature

Before we land this, it's worth being explicit that two-runtime is not a "sure, let's do it" change. It commits the compute layer to a new shape and is effectively a one-way door. This comment lays out the trade-off honestly so the decision is made with eyes open.

What it actually is

Two runtimes do not add CPU and do not magically "isolate" reads — both runtimes share the same cores. What the second runtime buys is one precise thing: a separate, OS-preemptible run loop, so an interactive read is no longer trapped behind a long, run-to-completion maintenance operator step on the same timely worker.

The right way to frame it is separation of concerns, not isolation:

• The maintenance runtime stays a pure run-to-completion batch engine. Operators consume their inputs fully; the only sanctioned yield is pre-exchange, for downstream memory reduction. Yielding for interactivity is an anti-pattern we do not want in that runtime.
• The interactive runtime is a pure, preemptible low-latency reader that reads the maintained arrangements zero-copy.

A single runtime cannot be both without compromising one of them. Two runtimes let each be pure. That is the real argument.

Why the obvious alternatives don't apply

"Just use a read replica." Fails on the use case that matters most: introspection. A replica's introspection describes that replica, so it fundamentally cannot be served from another one — and you need it exactly when the maintenance runtime is pinned (hydration, batchy work), which is when it blocks. Today we fly dark precisely then. Nothing but an in-process second runtime fixes that. Separately, replicas mostly redline on memory, not CPU, so a second replica doubles the binding resource; the shared-arrangement approach here duplicates no memory. And because those boxes are memory-bound, the spare CPU the interactive runtime needs is exactly what is already idle — the CPU-saturated case (where two runtimes help least) is not the common one.

"Just yield more finely in the one runtime." That is the anti-pattern above. It would degrade the maintenance runtime's core contract to buy interactivity it shouldn't be responsible for.

The costs we are accepting

1. A permanent cross-runtime concurrency surface. Seal signals, lost-wakeups, placeholder/adopt races, delayed-capability panics, read-hold lifetimes. This class of bug is subtle and does not go away; several were found and fixed on the way here. This is the real price, not "more threads."
2. The control plane remains a parallel, unsolved bottleneck. For non-introspection reads under load, peeks still serialize behind DDL on the single coordinator thread. Two-runtime fixes the data plane; it does not touch that. Landing this must not let the coordinator serialization problem fall off the roadmap — two-runtime is necessary but not sufficient for that second use case.

Why it's a one-way door

• The off switch is not a clean exit. The Solo path keeps single-runtime deployments byte-identical, but once users are on the isolated low-latency reads, turning the feature off is a visible query-latency regression, not a no-op.
• The capability we depend on will atrophy. Once reads live in the interactive runtime, the maintenance runtime no longer needs to accommodate interactivity at all — and it will be built to be maximally batchy because it was freed to be. Single-runtime interactive-read behavior will rot from disuse and hardened assumptions. Getting it back later is not a revert; it is a capability we will have to rebuild.

That irreversibility is the crux. It is acceptable only because the end-state — maintenance as a pure batch runtime, interactive as a pure reader — is the architecture we would choose deliberately, given the points above. The bar for merging is therefore not "we can back out." It is "we would design it this way on purpose." On the introspection-during-hydration and memory-bound-fleet facts, we would.

Bottom line

Worth doing, but as a deliberate architectural commitment. The durable, defensible assets are (a) zero-copy cross-thread arrangement sharing as a primitive and (b) introspection that survives hydration — both survive every counter-argument. The framing to avoid leaning on is "reads get faster," which a CPU-bound box or the unsolved coordinator cap can each undercut. Reviewers should weigh this as a commitment to the two-runtime shape, not just as an isolated feature.

[antiguru]

Alternative would be to allow setting transient global ids based on the name, using the canonical GloablId string representation.

[antiguru]

Could this function be generic over traces surfacing rows?

[antiguru]

Spiked. This is the same reduce_abelian higher-ranked output-key normalization wall documented on threshold_local: the bound only normalizes when the input and output trace types are concrete through the function signature, so a generic Tr: TraceReader does not compile. The three helpers (_local, _trace, _shared_trace) share identical bodies but must stay concrete for that reason.

Unifying them needs mz_reduce_abelian's key-container bounds reworked (the same class of fix as differential #797/#798's named key-container parameter), which is a broader reduce-plumbing change than this file. A macro could remove the textual duplication but adds indirection for three ~10-line functions without touching the type constraint. Leaving concrete and tracking the genericization with the reduce_abelian work rather than here.

[antiguru]

This is a limitation that we shouldn't have (and don't document).