feat(run): name the criteria a task failed in the summary, and fail CI with --fail-under

[earakely-scale]

Summary

Two changes make agent-env run's results useful without opening the store, and one makes it pipe quietly.

1. The summary says why a task scored below 1. Under each such task, one line per failed criterion: the step, the criterion, and the reason its verifier recorded (its justification, message or description), cut to one line. At most five per task, then "and N more".

Tasks:
  tasks/hello.json v1: scored below 1 (hello: 0), 0.1s, instance …/hello-opi80zxj
    hello: greets: does not contain 'hello'
    hello: check passes: exit=3; stderr=missing config: settings.toml

Before, the summary stopped at scored below 1 (hello: 0), and the reasons were only in the stored instance.

2. --fail-under SCORE lets CI fail on a miss. It takes a score from 0 to 1 and exits 1 when a task scored below it or recorded no score, which can't show it passed:

$ agent-env run hello --fail-under 1
…
Below --fail-under 1: tasks/hello.json (0)
$ echo $?
1

Without the flag the exit status is unchanged: only a run that raised, or a failed step, exits 1. That keeps a large eval tolerant by default.

3. A reader that leaves no longer breaks the run. With agent-env run … | head, the run used to log progress stopped: on_progress raised BrokenPipeError(32, 'Broken pipe') and exit 1. With --keep | tee and Ctrl-C, which stops tee too, it exited 1 and left the kept sandboxes up. Now, once a stream's reader has gone, the rest of that stream's lines go to /dev/null, and the run finishes and tears down as usual. The "Still up" warning goes to stderr, so it still reaches the user when stdout is piped away.

How

• task.failed_criteria(steps, context) sits beside recorded_scores and shares its loop over the recorded verifications (_verifications), so a criterion's step is named as its score is. It reads each verification scoring below 1. A result fails when its result is false, or, when it records no result, when its score is below 1; skipped results never fail. The criterion is named by criterion, id or name, else <type> #<index>. The reason is the first of justification, message, reason and description. That covers the shapes verify_sandbox, the env-outcome, prompt-response, unit-test and rubrics verifiers write.
• TaskRun.failed_criteria and BundleRun.below(threshold) give library callers the same information the CLI prints.
• cli/run.py:
  • _criteria_lines formats the summary lines and _report_below the closing line.
  • Every line goes through _echo, which catches BrokenPipeError per line and redirects only the stream that broke, so no block stops at its first failed write: --keep still waits and tears down, and --verbose tracebacks still print when only stdout is piped away.
  • --fail-under needs a BUNDLE, and changes nothing a --dry-run shows. Without a BUNDLE, the usage error now names the flags that were given (--fail-under needs a BUNDLE to run) instead of a hand-kept list of every flag.

Tests

• Unit:
  • failed_criteria_test.py covers naming by step or key, verifications at or above 1, results without result, skipped results, and the reason's precedence.
  • run_test.py covers the summary lines across result shapes, with truncation and the five-line cap, and --fail-under below, at and with no threshold. It also covers a task with no score failing the gate, and the usage and dry-run cases.
  • installed_test.py gets the new usage text.
  • run_teardown_test.py checks the "Still up" warning is on stderr.
  • Unit tier: 8,014 passed. Two unrelated cases each failed once across runs and pass alone: a test_trajectory_follower case and a gateway time-trigger case.
• Integration (fast tier, run_bundle_test.py):
  • The rerun that breaks hello now checks its criterion line, and a --fail-under 1 run that exits 1 with the "Below" line.
  • A new test closes the pipe after one line, as head does. The run must exit 0 with no BrokenPipeError on stderr, and the fixture checks no sandbox was left behind. On main the same run exits 1 with the warning.
  • A --keep run closes the pipe at "Holding", then gets Ctrl-C, and must exit 0 and tear its sandbox down. Before the per-line guard it exited 1 and left the sandbox folder.
  • A closed stderr drops only its own lines; stdout keeps printing.
  • The whole file passes: 10 passed, 1 skipped (the GNU stat case on a Mac).

Limits

• Only verifications that record per-criterion results get lines. A verification scoring below 1 with no failed result recorded shows its score alone, as before.
• --fail-under gates each task's lowest score. An eval-level gate, such as a pass rate, is still the "tolerant eval runs" item.

Not hot path (the CLI, plus a new reader beside recorded_scores), so no release of its own.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no blocking issue remains.

Summary

The PR names failed criteria in task summaries, adds --fail-under for CI, and lets runs finish when a pipe reader leaves.

• Task summaries name the checks that lowered a score.
• agent-env run can fail when a task misses a score cutoff.
• A closed output pipe no longer stops a bundle run.

_{Reviews (3) · Last reviewed commit: "refactor(run): derive the BUNDLE usage e..." · Reviewed by Greptile}