Skip to content

feat(run_code): pass verifier records to the script as input["verifiers"] - #180

Closed
tejas-latupalli wants to merge 2 commits into
mainfrom
tejaslatupalli/run-code-input-verifications
Closed

tejas-latupalli wants to merge 2 commits into
mainfrom
tejaslatupalli/run-code-input-verifications

Conversation

@tejas-latupalli

@tejas-latupalli tejas-latupalli commented Oct 13, 2026 •

Copy link
Copy Markdown

What and why

A run_code script receives {"args", "results"}, where results is context.metadata["script_results"]. Verifier steps record their outcome under context.metadata["verifications"], which a script can't see. A pipeline that wants to act on a verifier's findings, such as repairing a task after a Lighthouse audit, could only do it if each verifier also copied its record into script_results.

RunCodeTaskStep._build_input now also passes "verifiers": context.metadata.get("verifications", {}), and the module docstring documents the new key. Only the script input key is named verifiers; the context metadata key stays verifications. This change is additive: existing scripts that read only args and results behave the same. The new key holds an empty dict when no verifier has run.

The PR also fixes a race in tst/unit/store/mongo_document_store_test.py::test_a_unique_index_is_awaited_however_long_it_builds, which failed CI on this PR (assert 0.2989 >= 0.3). The test started the 0.3s release timer before reading time.monotonic(), so on a slow runner the measured wait could come in just under 0.3s. The test now reads the clock first.

Related: FD-3919. This replaces scaleapi/scaleapi#167303, which mirrored the Lighthouse audit record into script_results. Once this releases, the SDK only needs its agentenv-framework floor raised.

No size cap is added for the staged input.json. The verifier records already live in the persisted context, so they are JSON-serializable and bounded by what the context already carries.

How it was tested

  • pytest tst/unit/task_step/test_run_code.py tst/unit/store/mongo_document_store_test.py: 87 passed. Two new tests use the real runner path (_real_run):
    • test_script_reads_earlier_verifier_records: the script reads inp["verifiers"]["audit"]["state"], and inp["results"] still comes through.
    • test_script_gets_empty_verifiers_when_no_verifier_ran: inp["verifiers"] == {}.
  • I ran the fixed index-wait test 15 times in a row, and it passed every time.
  • make unit-test on the first commit: 7657 passed, 7 skipped, 1 failed. The failure is tst/unit/store/local_grant_server_test.py::test_the_platform_default_is_served_on_loopback_too_where_a_vm_on_this_machine_reaches_it, with httpx.ConnectError: Connection reset by peer. It fails the same way on main without this change; my sandbox's network proxy blocks that loopback request.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D6YRwGQbj6RUzhErtTpXVV


Generated by Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues remain.

Summary

run_code scripts now receive verifier records through input["verifiers"].

  • Run-code scripts can read verifier records alongside their other inputs.
  • The Mongo index test starts its release timer after its stopwatch.

Reviews (2) · Last reviewed commit: "fix(run_code): name the script input key..." · Reviewed by Greptile

…ations"]

A run_code script got only args and earlier script_results, so a pipeline
could not act on what verifier steps recorded under
context.metadata["verifications"] (e.g. a Lighthouse audit) without each
verifier duplicating its record. The script input now carries that dict too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6YRwGQbj6RUzhErtTpXVV
@tejas-latupalli
tejas-latupalli requested a review from a team as a code owner October 13, 2026 16:54
…ait test clock first

The verifier records reach the script as input["verifiers"]. In
test_a_unique_index_is_awaited_however_long_it_builds the release timer
started before the clock was read, so the measured wait could come in
just under 0.3s on a slow runner; read the clock first.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D6YRwGQbj6RUzhErtTpXVV
@tejas-latupalli tejas-latupalli changed the title feat(run_code): pass verifier records to the script as input["verifications"] feat(run_code): pass verifier records to the script as input["verifiers"] Oct 13, 2026

@agutmanstein-scale agutmanstein-scale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Breaks run_code design

@tejas-latupalli

Copy link
Copy Markdown
Author

Closing in favor of the SDK-side fix (scaleapi/scaleapi#167303, reopened). The Lighthouse step will mirror its record into script_results, so run_code stays unchanged here.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants