Most MCP tools can't distinguish "found nothing" from "couldn't look". I measured it, starting with my own #838
ogasurfproject-jpg
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Pre-submission Checklist
What would you like to share?
I built a read-only probe that asks every tool one question from its declared outputSchema alone: can a consumer separate a failed read from an empty one? It calls tools/list and nothing else. No tools/call, no installs, no account.
Why it matters. If a tool returns the same shape for "the query ran and matched nothing" and "the query threw," an agent cannot tell your outage from your silence. It will confidently report absence during a failure. Federico Blanco Sanchez-Llanos named this "The Mould, Not the Letter."
What I found. Across a spread sample of the 12,429 active public endpoints (18 spoke MCP, 335 tools): 96.7% (a strict structural ruler) to 99.1% (a lexical one) of tools declare no way to hold the difference. I report a band because the rulers err in opposite directions, and I publish the field names because the structural ruler's passes are mostly incidental booleans (one endpoint passed on remote: boolean from a job board, not a read-state).
I measured myself first. 85.7% of my own tools failed it, the gate and ledger included. Better than the field, still most of mine. Fixed to 2.9%; the whole diff, the four runtime bugs, and the reproduce commands are frozen in a public append-only ledger (mould-gh-12).
Check yours (read only, executes nothing):
The honest limit: a schema is a declaration, not behaviour. This says what your contract can express, not what your handler does at runtime; that needs tools/call, which I won't do to servers I don't own. Method, both rulers, and the population hash are all public.
Relevant Links
Ledger record (recompute the hash yourself): https://gate.horizonshield.dev/mould/mould-gh-12
Method and per-condition guide: https://shield.the-horizons-innovation.com/verify-directory/operators/
Survey data, CC0: https://github.com/ogasurfproject-jpg/mcp-registry-survey
All reactions