Skip to content

feat(hub): add agent visual critic - #325

Merged
alliscode merged 4 commits into
build/hub/newfrom
alliscode-agent-visual-critic
Nov 20, 2026
Merged

alliscode merged 4 commits into
build/hub/newfrom
alliscode-agent-visual-critic

Conversation

@alliscode

@alliscode alliscode commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Why

Prototype authors need a safe way to ask the repository's promoted decision model whether the current rendered work follows accepted learnings before requesting human review. The first integration must remain explicit and advisory so model scores cannot silently change code, publish Builds, promote candidates, or become verified review outcomes.

What changes

The Replica canvas adds an evaluate_prototype action that accepts bounded route screenshots for desktop or narrow viewports in light or dark themes. Screenshot upload is blocked unless the local Copilot process explicitly sets REPLICA_ALLOW_PROTOTYPE_SCREENSHOT_UPLOAD=true. The action then uses the approved Canvas grant and sends checkout context and image bytes without the user's prompt, Copilot session ID, preview origin, or credentials.

Hub derives the repository, prototype, matching published Build, membership, and rate-limit identity from the Canvas grant. It rereads the promoted candidate's exact consent grant and pinned question digest, deletion state, repository settings, candidate eligibility, and live runtime promotion before cached results, each inference request, and the final response. The runtime environment must be explicitly configured as fixture or production.

flowchart LR
  policy[Local screenshot opt-in] --> agent[Authoring agent]
  agent -->|explicit evaluate_prototype| canvas[Replica canvas]
  canvas -->|approved grant and bounded screenshots| hub[Hub authorization and policy]
  hub -->|pinned questions and promoted generation| runtime[Repository decision runtime]
  runtime --> findings[Unverified findings]
  findings --> agent
Loading

Durable reservations prevent duplicate in-flight work and retain consumed request units after failures or cache expiry. A separate monthly quota bounds fresh agent inference by evaluated route; it is not presented as runtime credit or cost accounting. Cached records store learning IDs and probabilities without screenshots or learning text.

Results identify the learning, route, evaluated views, follows and breaks probabilities, candidate generation, and cached or fresh status. Fixture results cannot become production guidance. The action never changes the prototype or records reviewer verification.

Ben Thomas and others added 2 commits November 20, 2026 07:57
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@alliscode
alliscode requested review from a team and nshki as code owners November 20, 2026 16:11
Copilot AI adaptive review requested due to automatic review settings November 20, 2026 16:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical consent and credit-accounting flaws can allow revoked-model inference and spending beyond the shared monthly budget.

20 open issues
What changed in this PR

Adds an explicit visual-critic action so authoring agents can request advisory feedback from a repository’s promoted decision model through the existing Canvas connection. Results remain unverified guidance and do not change code, publish Builds, or record reviewer verification.

Changes:

  • Adds evaluate_prototype with bounded screenshot inputs and structured findings.
  • Connects Hub authorization, learning eligibility, runtime inference, rate limits, and durable evaluation reservations.
  • Adds database schema, focused tests, and visual-critic documentation.
File Description
hub/​test/​database/​decision-agent-evaluation-migration/​index.db.test.ts Tests migration application and storage constraints.
hub/​server/​start-server-processes/​index.ts Accepts visual-critic dependencies.
hub/​server/​run-api/​index.ts Passes dependencies into the API app.
hub/​server/​routes/​types.ts Extends route dependencies.
hub/​server/​routes/​index.ts Wires the Canvas evaluation dependencies.
hub/​server/​routes/​api/​canvas/​index.ts Registers the grant-authorized evaluation endpoint.
hub/​server/​routes/​api/​canvas/​decision-model/​types.ts Defines the evaluation route contract.
hub/​server/​routes/​api/​canvas/​decision-model/​evaluations/​post/​index.ts Parses and dispatches evaluation requests.
hub/​server/​routes/​api/​canvas/​decision-model/​evaluations/​post/​index.test.ts Tests request validation and authority forwarding.
hub/​server/​routes/​api/​canvas/​decision-model/​create-agent-visual-critic-route-dependencies/​index.ts Composes runtime, learning, limits, and cache adapters.
hub/​server/​repositories/​decision-agent-evaluations/​reserve-decision-agent-evaluation/​index.ts Adds transactional cache and budget reservations.
hub/​server/​repositories/​decision-agent-evaluations/​reserve-decision-agent-evaluation/​index.db.test.ts Tests caching, expiry, and concurrent reservations.
hub/​server/​repositories/​decision-agent-evaluations/​delete-decision-agent-evaluation/​index.ts Deletes evaluation reservations.
hub/​server/​repositories/​decision-agent-evaluations/​complete-decision-agent-evaluation/​index.ts Persists completed results.
hub/​server/​repositories/​canvas/​read-evaluation-context/​index.ts Resolves prototype and matching Build context.
hub/​server/​repositories/​canvas/​read-evaluation-context/​index.db.test.ts Tests checkout-context lookup and exclusions.
hub/​server/​index.ts Creates and supplies visual-critic services.
hub/​server/​domains/​decision-model/​types.ts Adds evaluation contracts and candidate eligibility.
hub/​server/​domains/​decision-model/​evaluate-prototype-with-decision-model/​index.ts Implements policy checks and route evaluation.
hub/​server/​domains/​decision-model/​evaluate-prototype-with-decision-model/​index.test.ts Tests findings, caching, and policy denials.
hub/​server/​domains/​decision-model/​errors.ts Adds evaluation-specific error codes.
hub/​server/​domains/​decision-model/​create-hub-decision-taskset-source/​index.ts Reads learning questions and training consent.
hub/​server/​domains/​decision-model/​create-hub-decision-taskset-source/​index.test.ts Tests the learning-authority projection.
hub/​server/​domains/​decision-model/​create-decision-model-store/​index.ts Exposes candidate production eligibility.
hub/​server/​domains/​decision-model/​create-agent-evaluation-limits/​index.ts Adds session and repository rate limits.
hub/​server/​domains/​decision-model/​create-agent-evaluation-limits/​index.test.ts Tests both rate-limit scopes.
hub/​server/​domains/​canvas/​connections/​index.ts Derives evaluation authority from Canvas grants.
hub/​server/​create-app/​index.ts Forwards visual-critic route dependencies.
hub/​schema/​repository.prisma Links repositories to agent evaluations.
hub/​schema/​migrations/​20261219120000_decision_agent_evaluations/​migration.sql Creates evaluation storage and indexes.
hub/​schema/​decision-candidate.prisma Links candidates to agent evaluations.
hub/​schema/​decision-agent-evaluation.prisma Defines reservation and cached-result fields.
hub/​docs/​prototype-canvas.md Documents the action and advisory boundary.
hub/​docs/​decision-model-control-plane.md Documents evaluation policy and runtime integration.
.github/​extensions/​replica-prototype/​start-canvas-server/​index.mts Connects the canvas server to evaluation requests.
.github/​extensions/​replica-prototype/​start-canvas-server/​index.mjs Updates generated server wiring.
.github/​extensions/​replica-prototype/​register-prototype-canvas/​index.mts Registers evaluate_prototype and its input schema.
.github/​extensions/​replica-prototype/​register-prototype-canvas/​index.mjs Updates generated action registration.
.github/​extensions/​replica-prototype/​README.md Explains visual-critic usage and limitations.
.github/​extensions/​replica-prototype/​evaluate-prototype/​index.test.mts Tests request projection and response validation.
.github/​extensions/​replica-prototype/​evaluate-prototype/​index.test.mjs Adds generated extension tests.
.github/​extensions/​replica-prototype/​evaluate-prototype/​index.mts Validates screenshots and projects Hub findings.
.github/​extensions/​replica-prototype/​evaluate-prototype/​index.mjs Adds the generated evaluation adapter.

Review effort: Max (adaptive)

Prototype Author and others added 2 commits November 20, 2026 10:39
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@alliscode
alliscode merged commit e520c5a into build/hub/new Nov 20, 2026
51 checks passed
@alliscode
alliscode deleted the alliscode-agent-visual-critic branch November 20, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants