Repository navigation
feat(run_code): pass verifier records to the script as input["verifiers"] - #180
Closed
tejas-latupalli wants to merge 2 commits into
Closed
tejas-latupalli wants to merge 2 commits into
tejas-latupalli wants to merge 2 commits into
Conversation
…ations"] A run_code script got only args and earlier script_results, so a pipeline could not act on what verifier steps recorded under context.metadata["verifications"] (e.g. a Lighthouse audit) without each verifier duplicating its record. The script input now carries that dict too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6YRwGQbj6RUzhErtTpXVV
…ait test clock first The verifier records reach the script as input["verifiers"]. In test_a_unique_index_is_awaited_however_long_it_builds the release timer started before the clock was read, so the measured wait could come in just under 0.3s on a slow runner; read the clock first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D6YRwGQbj6RUzhErtTpXVV
agutmanstein-scale
approved these changes
Oct 13, 2026
agutmanstein-scale
requested changes
Oct 13, 2026
agutmanstein-scale
left a comment
Collaborator
There was a problem hiding this comment.
Breaks run_code design
Author
|
Closing in favor of the SDK-side fix (scaleapi/scaleapi#167303, reopened). The Lighthouse step will mirror its record into Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
A
run_codescript receives{"args", "results"}, whereresultsiscontext.metadata["script_results"]. Verifier steps record their outcome undercontext.metadata["verifications"], which a script can't see. A pipeline that wants to act on a verifier's findings, such as repairing a task after a Lighthouse audit, could only do it if each verifier also copied its record intoscript_results.RunCodeTaskStep._build_inputnow also passes"verifiers": context.metadata.get("verifications", {}), and the module docstring documents the new key. Only the script input key is namedverifiers; the context metadata key staysverifications. This change is additive: existing scripts that read onlyargsandresultsbehave the same. The new key holds an empty dict when no verifier has run.The PR also fixes a race in
tst/unit/store/mongo_document_store_test.py::test_a_unique_index_is_awaited_however_long_it_builds, which failed CI on this PR (assert 0.2989 >= 0.3). The test started the 0.3s release timer before readingtime.monotonic(), so on a slow runner the measured wait could come in just under 0.3s. The test now reads the clock first.Related: FD-3919. This replaces scaleapi/scaleapi#167303, which mirrored the Lighthouse audit record into
script_results. Once this releases, the SDK only needs itsagentenv-frameworkfloor raised.No size cap is added for the staged
input.json. The verifier records already live in the persisted context, so they are JSON-serializable and bounded by what the context already carries.How it was tested
pytest tst/unit/task_step/test_run_code.py tst/unit/store/mongo_document_store_test.py: 87 passed. Two new tests use the real runner path (_real_run):test_script_reads_earlier_verifier_records: the script readsinp["verifiers"]["audit"]["state"], andinp["results"]still comes through.test_script_gets_empty_verifiers_when_no_verifier_ran:inp["verifiers"] == {}.make unit-teston the first commit: 7657 passed, 7 skipped, 1 failed. The failure istst/unit/store/local_grant_server_test.py::test_the_platform_default_is_served_on_loopback_too_where_a_vm_on_this_machine_reaches_it, withhttpx.ConnectError: Connection reset by peer. It fails the same way onmainwithout this change; my sandbox's network proxy blocks that loopback request.🤖 Generated with Claude Code
https://claude.ai/code/session_01D6YRwGQbj6RUzhErtTpXVV
Generated by Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no actionable issues remain.Summary
run_codescripts now receive verifier records throughinput["verifiers"].Reviews (2) · Last reviewed commit: "fix(run_code): name the script input key..." · Reviewed by Greptile