fix(core): normalize claude-code MCP tool names to bare endpoint - #178
Open
Rodriguespn wants to merge 1 commit into
Open
fix(core): normalize claude-code MCP tool names to bare endpoint#178Rodriguespn wants to merge 1 commit into
Rodriguespn wants to merge 1 commit into
Conversation
Claude Code names MCP tools `mcp__<server>__<tool>`, so the parser recorded `originalName` (the scorer-facing endpoint) as e.g. `mcp__supabase-mcp__query_logs`, while the Codex parser records the bare `query_logs`. A scorer doing `tc.endpoint === "query_logs"` therefore matches Codex but silently never matches Claude Code, so tool-selection evals false-fail on Claude Code for behavior that actually happened. Strip the `mcp__<server>__` prefix in the claude-code parser so the endpoint is agent-agnostic, consistent with Codex. The raw name is still passed to `normalizeToolName`, so `name` stays `tool_use`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The claude-code parser recorded MCP tool calls'
originalName(the scorer-facingendpoint) as Claude Code's rawmcp__<server>__<tool>(e.g.mcp__supabase-mcp__query_logs), while the Codex parser records the barequery_logs. So a scorer doingtc.endpoint === "query_logs"matches Codex but never matches claude-code — tool-selection evals false-fail on claude-code for behavior that actually happened.This strips the
mcp__<server>__prefix in the claude-code parser so the endpoint is agent-agnostic, matching Codex. The raw name is still passed tonormalizeToolName, sonamestaystool_use.Why it's in the parser
Each agent parser owns normalizing its own tool names; Codex already emits the bare MCP tool name, claude-code was the outlier.
Verification
investigate-logs-002-query-logs-tool-selectionwithclaude-code-sonnet-5against the mcp#333 preview build went FAIL (1/2) → PASS (2/2); Showing a false negative (agent was called the tool but the eval check ignored it)