feat(eval): query_logs tool selection on hosted-shaped platform - #100
Draft
jordienr wants to merge 3 commits into
Draft
feat(eval): query_logs tool selection on hosted-shaped platform#100jordienr wants to merge 3 commits into
jordienr wants to merge 3 commits into
Conversation
…hosted) Two MCP-mode evals asserting description-driven log tool selection: - hosted project -> query_logs - self-hosted project -> get_logs From supabase/mcp#333 review feedback (Rodriguespn). Blocked on query_logs shipping in a released @supabase/mcp-server-supabase version and bumping MCP_SERVER_VERSION in this repo; not runnable until then.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Rodriguespn
added
the
run-evals-changed
Rodriguespn
removed
the
run-evals-changed
6 tasks
The self-hosted scenario assumed the evals harness could simulate a platform without ClickHouse-backed query_logs support, but supabaseMcpServer() always spawns @supabase/mcp-server-supabase against api-platform.ts, which implements queryLogs unconditionally regardless of --api-url. There is no way to exercise the get_logs-only arm through this harness; that behavior is already covered by supabase/mcp's own unit tests (server.test.ts, debugging-tools.test.ts). Keep and rename the hosted scenario, since query_logs is now the only listed logs tool on the platform-lite fixture (get_logs is hidden), so this verifies query_logs works end-to-end rather than a two-tool choice.
jordienr
changed the title
Rodriguespn
changed the base branch from
main
to
fix/claude-code-mcp-tool-endpoint
August 7, 2026 14:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One MCP-mode eval verifying the agent successfully uses
query_logsto investigate edge function logs on a hosted-shaped platform-lite fixture:investigate-logs-002-query-logs-tool-selection— assertsquery_logswas called (andget_logswas not), scored deterministically onctx.toolCalls[].endpoint.Why
Review feedback on supabase/mcp#333 (Rodriguespn): "curious if the agent will correctly call
query_logswhen working with the hosted version ... we should have an eval scenario." That PR now hidesget_logsfromtools/listwhenever the platform implementsqueryLogs(registration-driven, not description-driven), so this exercisesquery_logsend-to-end on the fixture where it's the only logs tool listed.Dropped: the self-hosted arm
The original version of this PR also had
investigate-logs-003-tool-selection-self-hosted, asserting the agent usesget_logs(notquery_logs) on a self-hosted-shaped platform. That scenario is untestable through this harness and has been removed:supabaseMcpServer()(packages/core/src/index.ts) always spawnsnpx @supabase/mcp-server-supabase@<version> --api-url <backend>, which always constructs its platform viacreateSupabaseApiPlatform()(packages/mcp-server-supabase/src/platform/api-platform.ts). That implementation definesqueryLogsunconditionally — it doesn't vary by--api-url, including when pointed at platform-lite. The actual self-hosted/CLI behavior (get_logs-only) comes from a different, externalSupabasePlatformimplementation (the CLI's own adapter) that this harness never spawns. There's no way to make the spawned server lackqueryLogsfrom evals today.That registration behavior is already covered by
supabase/mcp's own unit tests (server.test.ts: "debugging tools show get_logs when the platform does not implement query_logs";debugging-tools.test.ts). If we want a real evals-level self-hosted check in the future,@supabase/mcp-server-supabasewould need some way to force aqueryLogs-less platform for testing (e.g. a flag or a local-stack-shaped runtime), which is out of scope here.Still blocked — not runnable yet
query_logsonly exists in unmerged supabase/mcp#333.MCP_SERVER_VERSIONhere is pinned to a version without it, so this eval can't pass until:@supabase/mcp-server-supabaseversion shipsquery_logs, andMCP_SERVER_VERSIONis bumped to that version inpackages/core/src/index.ts.Validated with
pnpm eval:dry(loads, parses, plans asmode=tools runtime=platform-lite). Not executed — needs the version bump above.