Skip to content

fix: wrap replay selector-miss as REPLAY_DIVERGENCE (repair-loop regression)#1223

Merged
thymikee merged 3 commits into
mainfrom
fix/replay-selector-miss-divergence
Jul 12, 2026
Merged

fix: wrap replay selector-miss as REPLAY_DIVERGENCE (repair-loop regression)#1223
thymikee merged 3 commits into
mainfrom
fix/replay-selector-miss-divergence

Conversation

@thymikee

Copy link
Copy Markdown
Member

Summary

  • Regression: a replay action dispatch failure that was THROWN (e.g. a raw AppError from a press/click/fill/longpress selector-miss) escaped runReplayScriptFile's per-action if (!response.ok) divergence wrapping and fell through to the outer catch, returning a bare COMMAND_FAILED with the legacy diagnostics shape ({replayPath, step, action, positionals, snapshotDiagnostics}) instead of the ADR 0012 REPLAY_DIVERGENCE report — no resume, no screen refs, no suggestions. This breaks the interactive repair loop for the most common drift case (a renamed/removed selector). Target-binding verification failures (decision 3's own pre-action classification) were unaffected — they already return {ok:false} and were never thrown.
  • Root cause: invokeReplayAction (src/daemon/handlers/session-replay-action-runtime.ts) called invokeResolvedReplayAction(...) unguarded. DaemonInvokeFn is typed (req) => Promise<DaemonResponse> — every caller (the top-level replay loop, invokeReplayRetryBlock's per-attempt check, and nested Maestro runFlow invocations) only ever branches on response.ok, never on a rejection. A thrown AppError therefore had no safety net at this boundary and propagated raw past all of them.
  • Why CI missed it: every existing divergence-shape test (session-replay-target-verification-runtime.test.ts, session-replay-runtime-failure-response.test.ts) drives the failure through a returned {ok:false} invoke response — that path was never broken. No test exercised a thrown dispatch failure, so the gap shipped silently (surfaced by the refactor: consolidate architecture ownership and client results #1210 architecture-consolidation merge).

Fix layer

Considered two layers:

  1. Wrap the per-action loop body in runReplayScriptFile (session-replay-runtime.ts) in a try/catch.
  2. Wrap the dispatch call inside invokeReplayAction (session-replay-action-runtime.ts).

Went with (2): invokeReplayAction is the single choke point every replay action dispatch funnels through, regardless of nesting depth — the top-level loop, invokeReplayActionBlock/invokeReplayRetryBlock (retry blocks), and invokeMaestroRunFlowWhenControl (nested Maestro runFlow) all recurse back into this same function via its invokeNestedReplayAction closure. None of those nested callers had throw protection either — a throw inside a retry-block sub-action or a nested Maestro action would have escaped just as raw. Fixing at (2) closes the gap for all of them uniformly and needs no changes to runReplayScriptFile's loop or control-flow logic. It also makes invokeReplayAction honor DaemonInvokeFn's own contract (never reject) at its own boundary, rather than relying on every caller to defend itself.

-  const response = await invokeResolvedReplayAction({ ... });
+  let response: DaemonResponse;
+  try {
+    response = await invokeResolvedReplayAction({ ... });
+  } catch (dispatchErr) {
+    const appErr = asAppError(dispatchErr);
+    response = errorResponse(appErr.code, appErr.message, appErr.details);
+  }

The existing if (!response.ok) { ...; return await failStep(...); } in runReplayScriptFile then wraps it into REPLAY_DIVERGENCE exactly as it already does for any other returned failure — no other code changed.

Tests

New src/daemon/handlers/__tests__/session-replay-dispatch-selector-miss.test.ts, end-to-end through runReplayScriptFile with a throwing invoke mock (mirroring the real resolution.ts throw shape: AppError('COMMAND_FAILED', ..., {hint: selectorFailureHint(...)})), asserting REPLAY_DIVERGENCE (not COMMAND_FAILED) with resume{allowed,from,planDigest} and non-empty screen.refs:

  • (a) an annotated press whose target-v1 evidence verifies (matchCount ≥ 1, guard minted) but dispatch itself throws a selector-miss — proves verification passing does not suppress a thrown dispatch failure.
  • (b) an unannotated press whose dispatch throws a selector-miss.
  • (c) a fill selector-miss thrown at dispatch — also asserts the typed text never leaks into the divergence.

All three fail with error.code === 'COMMAND_FAILED' on the pre-fix code and pass after the fix (verified by temporarily reverting the fix and re-running).

Full existing replay suite (24 files / 309 tests, including retry-block and Maestro runFlow tests) stays green, confirming the fix doesn't change control-flow behavior for those nested callers.

Live verification (iPhone 17 Pro simulator, Metro :8082, real ADR-0012 target-v1 fixture)

Used ~/.agent-device-bench/replay-runs/derisk2/prevent-remove.ad, broke a press selector (renamed label to a nonexistent one), ran replay --json against the locally built daemon.

After the fix — real device, REPLAY_DIVERGENCE with resume + populated screen.refs:

{
  "kind": "action-failure",
  "cause": {
    "code": "COMMAND_FAILED",
    "message": "Selector did not match: label=\"Push Article ZZZ_RENAMED_MISSING\"",
    "hint": "Selector text/label values match exactly (quote multi-word values: text=\"Sign in\"). Run snapshot -i to see current elements and refs, or use find <text> for contains matching."
  },
  "screen": {
    "state": "available",
    "refsGeneration": 784727,
    "refs": [
      { "ref": "e1", "role": "application", "label": "React Navigation Example" },
      { "ref": "e2", "role": "window", "label": "Next keyboard" },
      { "ref": "e3", "role": "other", "label": "Next keyboard" },
      { "ref": "e4", "role": "other", "label": "Padding-Left" },
      "...16 more"
    ],
    "truncated": true
  },
  "resume": { "allowed": true, "from": 4, "planDigest": "2a77bbb16042adbfd5588e39d860f8a6e19db62677bcfc09fcbac4e58e02cab5" },
  "suggestionCount": 0
}

Note on the live "before" comparison: I also rebuilt and ran the unfixed daemon against the same broken fixture and it produced the same correct REPLAY_DIVERGENCE shape — because interaction-touch.ts's own dispatch paths (dispatchRuntimeInteraction, dispatchDirectIosSelectorInteraction) and the Maestro tapOn resolution path already catch a resolution-time selector-miss internally and return {ok:false} before it ever reaches invokeReplayAction, so this specific live trigger was already shielded by an unrelated layer. The regression is real and reachable at the invokeReplayAction boundary itself — proven directly by the unit-level fault injection above (a mocked invoke that throws, matching the exact resolution.ts throw shape), which reliably reproduces bare COMMAND_FAILED pre-fix and REPLAY_DIVERGENCE post-fix:

// before (unit-level fault injection, pre-fix)
{ "ok": false, "error": { "code": "COMMAND_FAILED", "message": "No element matched selector label=\"Missing\"" } }

// after (same injection, post-fix)
{ "ok": false, "error": { "code": "REPLAY_DIVERGENCE", "message": "Replay failed at step 1 (...): No element matched selector label=\"Missing\"", "details": { "divergence": { "kind": "action-failure", "resume": { "allowed": true, "from": 1, "planDigest": "..." }, "screen": { "state": "available", "refs": [] } } } } }

The fix still closes a real, currently-reachable gap: any caller of invokeReplayAction whose underlying dispatch throws (present or future command handlers, retry-block sub-actions, nested Maestro runFlow actions) is now guaranteed a wrapped REPLAY_DIVERGENCE instead of an unwrapped throw, matching DaemonInvokeFn's documented contract.

Gate

pnpm check (lint, typecheck, layering, production-exports, mcp-metadata, build, bundle-owner-files, fallow, unit tests, smoke) — all green: 417 test files / 3819 tests passed, 0 failures.

Test plan

  • npx vitest run src/daemon/handlers/__tests__/session-replay-dispatch-selector-miss.test.ts (new regression tests, fail pre-fix / pass post-fix)
  • npx vitest run src/daemon/handlers/__tests__/session-replay*.test.ts (full replay suite, 24 files / 309 tests)
  • npx tsc --noEmit -p .
  • pnpm check (full gate)
  • Live simulator verification with before/after JSON pasted above

…ession)

A dispatch failure thrown by a replay action (e.g. a selector-miss during
press/click/fill/longpress) escaped runReplayScriptFile's `if (!response.ok)`
divergence wrapping and fell through to the outer catch, returning a bare
COMMAND_FAILED with the legacy diagnostics shape instead of the ADR 0012
REPLAY_DIVERGENCE report (no resume, no screen refs, no suggestions) —
breaking the interactive repair loop for the common drift case.

Fix at the invocation layer: invokeReplayAction (the single choke point every
replay action dispatch funnels through — the top-level loop, retry blocks,
and nested Maestro runFlow invocations) now normalizes a thrown AppError into
a `{ok:false}` response before returning, matching DaemonInvokeFn's own
never-reject contract. This is more localized than wrapping the per-action
loop body in session-replay-runtime.ts and additionally closes the same gap
for retry/control-flow and Maestro nested dispatch, which funnel through this
exact function and previously had no throw protection either.

Adds regression coverage for the exact gap: end-to-end replay tests asserting
REPLAY_DIVERGENCE (not COMMAND_FAILED) with resume + screen refs for an
annotated press, an unannotated press, and a fill selector-miss thrown at
dispatch.
@github-actions

github-actions Bot commented Jul 12, 2026

Copy link
Copy Markdown

Size Report

Metric Base Current Diff
JS raw 1.7 MB 1.7 MB +69 B
JS gzip 536.7 kB 536.7 kB +40 B
npm tarball 646.3 kB 646.3 kB +41 B
npm unpacked 2.3 MB 2.3 MB +69 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 28.0 ms 27.9 ms -0.1 ms
CLI --help 57.5 ms 57.4 ms -0.2 ms

Top changed chunks:

Chunk Raw diff Gzip diff
dist/src/session.js +69 B +40 B

@thymikee

Copy link
Copy Markdown
Member Author

Review at 5ee04cd found two blockers:

  1. High: the catch wraps all failures from invokeResolvedReplayAction, including plain TypeError and programmer bugs in replay control or Maestro handling, into UNKNOWN and then a repairable REPLAY_DIVERGENCE. Narrow the catch to expected AppError dispatch failures, or move it around the rejecting invoke call, and retain the outer internal-error path for unexpected exceptions.
  2. Medium: errorResponse with appErr.details bypasses normalizeError. Later code hoists hint, diagnosticId, and logPath, but not retriable or supportedOn. Thrown AppErrors can therefore lose top-level recovery signals. Add thrown-path coverage and preserve the normalized fields.

The new tests are non-vacuous for the catch, but the live before and after did not exercise it because the production selector route was already shielded. No fixer dispatched.

thymikee added a commit that referenced this pull request Jul 12, 2026
Addresses three protocol blockers:
- Blocker 1: add repairHint to Decision 4's details.divergence field list and
  spec it as a single bounded enum (record-and-heal|state-repair|caution|manual),
  present on every divergence, carried at every level, surviving all four
  projections (text/JSON/client/MCP).
- Blocker 2: make R3's mapping total — no recorded targetEvidence (reachable for
  unannotated action-failure per #1223, or any kind on a legacy script) => manual,
  generalizing the sparse-capture fail-safe.
- Blocker 3: correct capture-timing — target-binding kinds use their PRE-action
  tree; action-failure uses its POST-response tree (adequate for the container
  presence test); no new pre-action tree is stored for action-failure.
Validation extended for the projection-survival and no-evidence/post-response cases.
…eError

invokeReplayAction's catch previously coerced ANY thrown dispatch failure
(via asAppError) into a {ok:false} response so the replay loop's divergence
wrapping applies — including plain TypeError/ReferenceError from programmer
bugs, which asAppError silently reclassified as a repairable
REPLAY_DIVERGENCE ("heal a selector"), masking a real crash.

Narrow the catch to only wrap actual AppError dispatch failures; anything
else re-throws to the outer internal-error path. Switch the wrap itself from
errorResponse(appErr.code, appErr.message, appErr.details) to
{ ok: false, error: normalizeError(dispatchErr) } — the canonical daemon
thrown->response pattern (request-generic-dispatch.ts, selector-runtime.ts)
— so retriable/supportedOn and hint/diagnosticId/logPath hoisting survive
on the thrown path instead of being dropped.
@thymikee thymikee merged commit 70bb9b6 into main Jul 12, 2026
22 checks passed
@thymikee thymikee deleted the fix/replay-selector-miss-divergence branch July 12, 2026 12:28
@github-actions

Copy link
Copy Markdown
PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-07-12 12:28 UTC

thymikee added a commit that referenced this pull request Jul 12, 2026
* docs: ADR-0012 Decision 7 — agent-supervised re-record repair

Adds "heal-by-doing": when replay diverges on selector drift, the agent
performs the failed step's intent with ordinary interactive commands
against blessed refs, and the CLI emits the healed .ad from the session's
actual successful execution path (session.actions) instead of hand-edited
selector text. Folds in five normative protocol rules (R1-R5) from an
external design review (verdict: SOUND-WITH-FIXES) covering record-arming
timing, --from continuation semantics, mechanical repairHint routing, the
writer's bare-@ref fail-close, and the recorded-open requirement. Makes
explicit that this reintroduces an EXPLICIT, opt-in heal and is therefore
consistent with (not a reversal of) decision 1's retirement of --update's
SILENT auto-rewrite.

* docs: renumber to Decision 6 and make repairHint the mechanical primary router

- Rename 'Decision 7' -> 'Decision 6' (ADR has decisions 1-5; new one is the
  6th, placed after 5. Mandatory validation; existing decisions unchanged).
- Reframe the two-sub-flows router: the CLI-computed repairHint (mechanical,
  in-scope, ships with this decision per R3) is the PRIMARY router; the agent
  follows the hint and uses screen.refs only as an ambiguity override, not as
  the default router. Removes the agent-judgment-vs-R3 contradiction.

* docs: ADR-0012 Decision 6 — daemon-side repairHint (4 kinds) + repair-run boundary (R6)

Addresses two P1 review gaps:
- P1-A: state that repairHint is computed daemon-side at divergence time from
  the recorded targetEvidence + the daemon's own full pre-action capture (only
  the enum crosses the wire, so the flat/capped screen.refs never gate routing);
  define repairHint for all four divergence kinds (selector-miss, identity-
  mismatch, identity-unverifiable, action-failure) with the sparse-capture
  fail-safe to manual.
- P1-B: add R6 — --save-script records a boundary watermark (session.actions.
  length at invocation); the healed script serializes only the post-watermark
  slice, so a reused session's earlier actions don't pollute it.
Clarifications: R4 fails loudly (non-zero exit, never swallowed); exact default
output path = <original-stem>.healed.ad sibling; arming sets recordSession AND
the watermark before step 1.

* docs: ADR-0012 — declare repairHint in the wire contract + make R3 total

Addresses three protocol blockers:
- Blocker 1: add repairHint to Decision 4's details.divergence field list and
  spec it as a single bounded enum (record-and-heal|state-repair|caution|manual),
  present on every divergence, carried at every level, surviving all four
  projections (text/JSON/client/MCP).
- Blocker 2: make R3's mapping total — no recorded targetEvidence (reachable for
  unannotated action-failure per #1223, or any kind on a legacy script) => manual,
  generalizing the sparse-capture fail-safe.
- Blocker 3: correct capture-timing — target-binding kinds use their PRE-action
  tree; action-failure uses its POST-response tree (adequate for the container
  presence test); no new pre-action tree is stored for action-failure.
Validation extended for the projection-survival and no-evidence/post-response cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant