Repository navigation
feat(hub): add agent visual critic - #325
Merged
Merged
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical consent and credit-accounting flaws can allow revoked-model inference and spending beyond the shared monthly budget.
20 open issues
Preserve the complete validated question for runtime inference · New Require the candidate's active consent grant for evaluation · New Enforce candidate revocation before reuse, inference, and further dispatch · New Include currentRoute in the evaluation cache key · New Include guidance environment in the cache identity · New Enforce or separate inference cost from the shared spending cap · New Recheck promotion before returning cached findings · New Sort screenshots and provide matching viewport descriptions · New Retain charges for completed or uncertain inference calls · New Preserve consumed credits when expiring cached evaluation payloads · New Use one locked budget calculation for training and agent reservations · New Require matched runtime environment metadata before enabling guidance · New Authenticate Canvas credentials before parsing large request bodies · New Bound the total request deadline across sequential route evaluations · New Test Canvas grants and repository access through the real router · New Validate complete PNG uploads before reserving work · New Test learning-text redaction and promotion matching with findings · New Restrict recalled learnings to the promoted candidate's contract · New Assert each forbidden column is absent independently · New Keep repository imports out of the domain layer · New
What changed in this PR
Adds an explicit visual-critic action so authoring agents can request advisory feedback from a repository’s promoted decision model through the existing Canvas connection. Results remain unverified guidance and do not change code, publish Builds, or record reviewer verification.
Changes:
- Adds
evaluate_prototypewith bounded screenshot inputs and structured findings. - Connects Hub authorization, learning eligibility, runtime inference, rate limits, and durable evaluation reservations.
- Adds database schema, focused tests, and visual-critic documentation.
| File | Description |
|---|---|
hub/test/database/decision-agent-evaluation-migration/index.db.test.ts |
Tests migration application and storage constraints. |
hub/server/start-server-processes/index.ts |
Accepts visual-critic dependencies. |
hub/server/run-api/index.ts |
Passes dependencies into the API app. |
hub/server/routes/types.ts |
Extends route dependencies. |
hub/server/routes/index.ts |
Wires the Canvas evaluation dependencies. |
hub/server/routes/api/canvas/index.ts |
Registers the grant-authorized evaluation endpoint. |
hub/server/routes/api/canvas/decision-model/types.ts |
Defines the evaluation route contract. |
hub/server/routes/api/canvas/decision-model/evaluations/post/index.ts |
Parses and dispatches evaluation requests. |
hub/server/routes/api/canvas/decision-model/evaluations/post/index.test.ts |
Tests request validation and authority forwarding. |
hub/server/routes/api/canvas/decision-model/create-agent-visual-critic-route-dependencies/index.ts |
Composes runtime, learning, limits, and cache adapters. |
hub/server/repositories/decision-agent-evaluations/reserve-decision-agent-evaluation/index.ts |
Adds transactional cache and budget reservations. |
hub/server/repositories/decision-agent-evaluations/reserve-decision-agent-evaluation/index.db.test.ts |
Tests caching, expiry, and concurrent reservations. |
hub/server/repositories/decision-agent-evaluations/delete-decision-agent-evaluation/index.ts |
Deletes evaluation reservations. |
hub/server/repositories/decision-agent-evaluations/complete-decision-agent-evaluation/index.ts |
Persists completed results. |
hub/server/repositories/canvas/read-evaluation-context/index.ts |
Resolves prototype and matching Build context. |
hub/server/repositories/canvas/read-evaluation-context/index.db.test.ts |
Tests checkout-context lookup and exclusions. |
hub/server/index.ts |
Creates and supplies visual-critic services. |
hub/server/domains/decision-model/types.ts |
Adds evaluation contracts and candidate eligibility. |
hub/server/domains/decision-model/evaluate-prototype-with-decision-model/index.ts |
Implements policy checks and route evaluation. |
hub/server/domains/decision-model/evaluate-prototype-with-decision-model/index.test.ts |
Tests findings, caching, and policy denials. |
hub/server/domains/decision-model/errors.ts |
Adds evaluation-specific error codes. |
hub/server/domains/decision-model/create-hub-decision-taskset-source/index.ts |
Reads learning questions and training consent. |
hub/server/domains/decision-model/create-hub-decision-taskset-source/index.test.ts |
Tests the learning-authority projection. |
hub/server/domains/decision-model/create-decision-model-store/index.ts |
Exposes candidate production eligibility. |
hub/server/domains/decision-model/create-agent-evaluation-limits/index.ts |
Adds session and repository rate limits. |
hub/server/domains/decision-model/create-agent-evaluation-limits/index.test.ts |
Tests both rate-limit scopes. |
hub/server/domains/canvas/connections/index.ts |
Derives evaluation authority from Canvas grants. |
hub/server/create-app/index.ts |
Forwards visual-critic route dependencies. |
hub/schema/repository.prisma |
Links repositories to agent evaluations. |
hub/schema/migrations/20261219120000_decision_agent_evaluations/migration.sql |
Creates evaluation storage and indexes. |
hub/schema/decision-candidate.prisma |
Links candidates to agent evaluations. |
hub/schema/decision-agent-evaluation.prisma |
Defines reservation and cached-result fields. |
hub/docs/prototype-canvas.md |
Documents the action and advisory boundary. |
hub/docs/decision-model-control-plane.md |
Documents evaluation policy and runtime integration. |
.github/extensions/replica-prototype/start-canvas-server/index.mts |
Connects the canvas server to evaluation requests. |
.github/extensions/replica-prototype/start-canvas-server/index.mjs |
Updates generated server wiring. |
.github/extensions/replica-prototype/register-prototype-canvas/index.mts |
Registers evaluate_prototype and its input schema. |
.github/extensions/replica-prototype/register-prototype-canvas/index.mjs |
Updates generated action registration. |
.github/extensions/replica-prototype/README.md |
Explains visual-critic usage and limitations. |
.github/extensions/replica-prototype/evaluate-prototype/index.test.mts |
Tests request projection and response validation. |
.github/extensions/replica-prototype/evaluate-prototype/index.test.mjs |
Adds generated extension tests. |
.github/extensions/replica-prototype/evaluate-prototype/index.mts |
Validates screenshots and projects Hub findings. |
.github/extensions/replica-prototype/evaluate-prototype/index.mjs |
Adds the generated evaluation adapter. |
Review effort: Max (adaptive)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Why
Prototype authors need a safe way to ask the repository's promoted decision model whether the current rendered work follows accepted learnings before requesting human review. The first integration must remain explicit and advisory so model scores cannot silently change code, publish Builds, promote candidates, or become verified review outcomes.
What changes
The Replica canvas adds an
evaluate_prototypeaction that accepts bounded route screenshots for desktop or narrow viewports in light or dark themes. Screenshot upload is blocked unless the local Copilot process explicitly setsREPLICA_ALLOW_PROTOTYPE_SCREENSHOT_UPLOAD=true. The action then uses the approved Canvas grant and sends checkout context and image bytes without the user's prompt, Copilot session ID, preview origin, or credentials.Hub derives the repository, prototype, matching published Build, membership, and rate-limit identity from the Canvas grant. It rereads the promoted candidate's exact consent grant and pinned question digest, deletion state, repository settings, candidate eligibility, and live runtime promotion before cached results, each inference request, and the final response. The runtime environment must be explicitly configured as fixture or production.
Durable reservations prevent duplicate in-flight work and retain consumed request units after failures or cache expiry. A separate monthly quota bounds fresh agent inference by evaluated route; it is not presented as runtime credit or cost accounting. Cached records store learning IDs and probabilities without screenshots or learning text.
Results identify the learning, route, evaluated views, follows and breaks probabilities, candidate generation, and cached or fresh status. Fixture results cannot become production guidance. The action never changes the prototype or records reviewer verification.