Multi-level orchestration: depth-capable client core and drill-down UI

[danielpeng2]

Description

Client side of multi-level (configurable-depth) agent orchestration. Today only the root "lead" agent can start child agents; the server is making depth configurable via OrchestrationMaxDepth (root = depth 0, a run may orchestrate iff depth < max, default 1 = today's behavior, dogfood = 2). This PR makes the client behave correctly when a tree is more than one level deep:

• Mid-tree parents stream their children. A child conversation blocked on wait_for_events now confirms parent status against the server and registers for the ancestor event stream, instead of being short-circuited by the old one-level-tree invariant (still behind the WaitForEventsParentRegistration dogfood flag).
• Children auto-execute run_agents. Child conversations live in hidden panes where a confirmation card would be invisible and hang the run. Their run_agents calls auto-execute; interactive policy blocks (never-allow profile, missing harness key, disapproved config) still surface as a graceful Denied tool result. The multi-level surfaces are gated on a new MultiLevelOrchestration dogfood flag: with the flag off, a child's run_agents call is denied outright ("Multi-level orchestration is not enabled on this client.") instead of falling through to the interactive card path. Run-wide settings carry over via the existing per-surface execution-profile inheritance. The server gates tool exposure by depth, so this path is unreachable at depth 1.
• The pill bar becomes a drill-down. Each conversation's bar renders only its DIRECT children; a child that is itself an orchestrator renders as a group pill with a rolled-up subtree badge (count and tint computed in a single walk over the loaded subtree, matching exactly what the aggregated status covers). Breadcrumbs lead back to the tree root and, when 2+ levels deep, to the direct parent. At depth 1 this renders exactly what the flattened bar rendered before.
• Navigation is depth-aware. ESC and the back button go one level up (repeated ESC walks up the tree) and the label names the direct parent; ESC now also works when the parent lives in a hidden off-tree pane by swapping it in place instead of the unreachable workspace-focus path. Next/prev cycling and the agents-panel root card roll up the whole subtree via the recursive topology walkers. Parent resolution (explicit id → run-id index → legacy server-token index) is unified in one history-model resolver shared by indexing, breadcrumbs, the root walk, and the back button.
• Disclosure on the confirmation card. When children may orchestrate, the run_agents card notes "These agents may start their own child agents" (gated on the MultiLevelOrchestration flag, since the client cannot cheaply know the server depth budget).

Everything is inert at orchestration depth 1: the server never grants children the run_agents tool, trees never exceed one level, and the drill-down anchor resolution matches the old root-anchored behavior exactly.

Linked Issue

Part of the multi-level orchestration work (client side); pairs with the warp-server OrchestrationMaxDepth changes.

Testing

• Unit tests: mid-tree wait registration (streamer, including the wait-time eligibility predicate), child auto-execute policy under both MultiLevelOrchestration flag states, recursive rooting / spawn-order cycling with 3-level fixtures, direct-children pill ordering, drill-down anchor resolution, breadcrumb target resolution (including token-only parent linkage), loaded-subtree rollup counts, and back-button labels at depth (truncation is pixel-based via the button's label clip).
• cargo check -p warp --lib, cargo clippy -p warp --all-targets --tests -- -D warnings, and ./script/format are clean at this stack state.
• I have manually tested my changes locally with ./script/run (depth-2 trees against a server with OrchestrationMaxDepth=2)

https://www.loom.com/share/beffa1541c5e4032914442baf93bde1b

Agent Mode

• Warp Agent Mode - This PR was created via Warp's AI Agent Mode

Co-Authored-By: Warp Agent agent@warp.dev

[danielpeng2]

• Multi-level orchestration: pull-based remote grandchild visibility #14763
• Multi-level orchestration: depth-capable client core and drill-down UI #14762 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[oz-for-oss]

@danielpeng2

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[oz-for-oss]

Overview

This PR adds client support for multi-level orchestration: depth-aware run-agents behavior, mid-tree event streaming, recursive status rollups, drill-down pill-bar navigation, breadcrumbs, and back-button behavior for nested agents.

Concerns

• The change is user-facing (pill bar drill-down, breadcrumbs, back-button labeling/navigation, and run-agents card copy), but the provided PR description does not include screenshots or a screen recording. For this user-facing change, please include screenshots or a screen recording demonstrating it working end to end.

Verdict

Found: 0 critical, 1 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

[cephalonaut]

Ordered by priority.

Daniel - looks good overall. Reviewed with an agent, so feel free to push back on
any of these or skip altogether.

Highest value

1. Should the multi-level work get its own flag? — run_agents.rs:438, run_agents_card_view.rs:1546, warp_features/src/lib.rs:706
Today the disclosure is gated on WaitForEventsParentRegistration, whose doc covers only the
streamer registration, and the child auto-execute isn't gated at all. QUALITY-919 plans to
promote and then remove that flag, which would ship the disclosure without anyone deciding to.
A separate flag would cover both, and would keep the auto-execute from running ahead of the
server's depth budget.

2. What happens on restore from disk? — orchestration_pill_bar.rs:684, :1789, terminal/view.rs:28645
The badge counts descendants from the children index, including ones not loaded yet. The child
list filters those out, and the bar returns None when the filtered list is empty — so a pill
could show "3" and drilling in makes the bar disappear. There are now three separate fallbacks
for an unloaded ancestor (this count, the breadcrumb label, the back-button label). Is an
unloaded ancestor actually reachable here? If not, those fallbacks may be hiding a bug.

3. Two different parent lookups — orchestration_pill_bar.rs:735 and :714
drill_down_anchor_id resolves via the links helper (agent-id + server-token); breadcrumb_ids
and the root walk use the history model's (agent-id only). With server-token-only linkage you
can drill in and have no breadcrumb back. Use the same one in both?

4. The bar and keyboard nav can now disagree — orchestration_pill_bar.rs:637, orchestration_topology.rs:214, :288
The comments still say these can't drift. The row now shows one level while next/prev cycles the
whole tree, so cycling can land on an agent that isn't in the bar and the bar re-anchors under
you. Comments need a tweak either way — and do you want cycling to reach agents that aren't
visible?

Smaller

5. Back-button truncation — terminal/view.rs:28625
ActionButton already ellipsizes once max_label_width is set (action_button.rs:369, :789);
the back button sets neither. Counting characters misses wide glyphs, so a CJK name still
overflows. with_max_label_width would replace the constant, the branch, and its test.

6. A test still asserts the removed short-circuit — wait_for_events_tests.rs:84
It expects zero get_ambient_agent_task calls for a child. It passes only because App::test
never drives the spawned future, so times(1) wouldn't work either. Pulling the guard chain in
register_parent_on_wait into a plain predicate would make it testable — and would cover the
child case, which the new streamer test doesn't (it calls finish_register_parent_on_wait
directly).

7. Leaf children now refetch on every wait — orchestration_event_streamer.rs:556
Without the short-circuit a leaf never becomes a parent, so it refetches each time it waits.
Heads up that #14473/#14480 are in review and should remove the need for this fetch — worth
keeping an eye on so the two don't collide.

8. drill_down_anchor_id isn't tested — orchestration_pill_bar_tests.rs:318
The new tests cover breadcrumb_ids only. build_three_level_tree would cover the anchor rules
in a few asserts, including the depth-1 equivalence the PR claims.

9. One more label case — view_tests.rs:8496
"for parent agent" (nested parent, empty name) isn't covered. One assert on the existing
fixture.

Nits

10. Telemetry — orchestration_pill_bar.rs:1834, :816
Breadcrumb clicks look identical to pill switches, and source_conversation_id now means the
anchor rather than the root. Probably wants a look before anyone reads these numbers.

11. Double walk — orchestration_pill_bar.rs:684
Count and aggregated status walk each subtree separately, per child, per render. Could be one
walk if it's easy.

12. Breadcrumb pill re-derives the pill chrome — orchestration_pill_bar.rs:1762
Will drift next time pills get restyled.

13. Leftover? — orchestration_topology.rs:300
history.conversation(&active_conversation_id)?; discards its result. Still needed?

14. Parent resolved twice — terminal/view.rs:11336
is_child_agent is just label != "for terminal".

15. "sub-agents" — run_agents_card_view.rs:1550
Elsewhere we say "child agents", including the denial message on this same path.

[danielpeng2]

Thanks for the thorough review — addressed in 4942833. Point-by-point:

1. Flag split — done. Added a MultiLevelOrchestration dogfood flag gating the client-side multi-level surfaces. The confirmation-card disclosure now gates on it (not WaitForEventsParentRegistration, which stays scoped to streamer registration — independently useful at depth 1). Child conversations still always auto-execute run_agents (a card in a hidden pane would hang the run), but with the flag off, prepare_request_for_execution denies the call outright with "Multi-level orchestration is not enabled on this client." instead of falling through to the interactive path. Tests cover both flag states end-to-end through execute().
2. Badge vs. row mismatch — done. New loaded_subtree_rollup topology helper does a single walk over loaded descendants and returns count + aggregated status together; aggregated_orchestrator_status shares the same walker. The badge count now covers exactly the set the aggregated status inspects (unloaded index-only ids excluded — test pins this), and the double traversal per group pill is gone (also resolves 11).
3. Parent resolution unified — done. It turned out resolved_parent_conversation_id_from_refs already covered tokens via conversation_id_for_agent_id (run-id index → server-token index), so the real fix was deleting the duplicated fallback chain: orchestration_conversation_links::parent_conversation_id is now a thin wrapper over the history resolver, and the redundant links-local conversation_id_for_agent_id is gone. Token-linked children index under their parent at restore (desired); new tests pin token-only linkage → child indexing + breadcrumb targets, and → the "for Orchestrator" back label.
4. Whole-tree cycling — kept deliberately. Keyboard next/prev cycles the whole tree while the bar re-anchors via drill_down_anchor_id, which is correct follow-the-selection behavior — the bar always shows the level containing the selection. The two stale comments ("cannot drift" / "single ordering source") are rewritten to describe reality: per-level ordering is shared; cycling is tree-wide; the bar follows by re-anchoring.
5. Back-button truncation — done. Dropped BACK_BUTTON_PARENT_NAME_MAX_CHARS and the char-count branch; the button now sets with_max_label_width (160px) and ellipsizes pixel-based via its label clip. The char-truncation test went with it.
6. Stale wait test — done. Extracted the guard chain into should_register_parent_on_wait (flag → not-remote-view → not-already-parent; no child check) with four focused unit tests, including "child conversation IS eligible". The App-based executor test is renamed to execute_invokes_parent_registration_for_child_conversations, now drives the spawned fetch and asserts times(1) on the mock — no more vacuous times(0) or false short-circuit comments.
7. Leaf refetch on every wait — accepted cost, no change. Children can appear out-of-band (Oz CLI / web API) at any time, so a leaf can become a parent between waits; the wait-time fetch is the cheapest correct check. The structural fix is the unified event-streaming work in QUALITY-928: Orchestration unified stack — core tracker + family stream (M1) #14473/QUALITY-928: Orchestration unified stack — pane path + transcript (M2) #14480.
8. Anchor tests — done. Three drill_down_anchor_id tests: leaf anchors its parent's level, node-with-children anchors itself, and depth-1 equivalence with the historical root-anchored behavior.
9. Done. The existing fixture now also asserts "for parent agent" for a child of an empty-named nested parent (and "for Orchestrator" for the empty-named mid itself).
10. Telemetry — done. Breadcrumb clicks now report a dedicated PillBarPillKind::Breadcrumb, and PillBarInteractionEvent carries root_conversation_id alongside source_conversation_id (documented as the drill-down anchor; the two coincide at depth 1 and in the TUI's root-anchored tab bar).
11. Done via 2 (single-walk rollup).
12. Breadcrumb pill chrome extraction — deferred. The breadcrumb pill already shares the PILL_* constants; folding it into render_pill needs a small render-config refactor that we'd rather do as a follow-up than grow this PR.
13. Done. Kept the existence guard with a one-line comment marking the intentional bail.
14. Done. update_agent_view_back_button_state derives is_child_agent from the label result (label != "for terminal"), dropping the duplicate parent resolution.
15. Done. "sub-agents" → "child agents".

Validation: ./script/format clean, both presubmit clippy invocations clean, 168 targeted tests green (pill bar, topology, run_agents, wait_for_events, streamer, back-button labels). PR description updated for the new flag gating.

(Posted by Warp Agent on Daniel's behalf.)