docs: add tester quick-start smoke test guide - #158
Conversation
A fast end-to-end smoke test for verifying a solr-mcp release JAR across Claude Code, Claude Desktop, and MCP Inspector. Covers client registration for all three transports, a list-collections sanity check, and a full build-and-query walkthrough against a self-contained 61-document demo collection with expected-output checkpoints and sample queries. Lives in dev-docs/ alongside the other contributor/release-verification guides. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: adityamparikh <aditya.m.parikh@gmail.com>
The demo URL is a shared public Solr, so the 'shows' collection may already exist from a prior tester's run. create-collection is not idempotent and errors on an existing name. Add guidance to use a unique collection name (keeping the doc-count checkpoints valid) with a reuse/delete fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: adityamparikh <aditya.m.parikh@gmail.com>
|
Thanks for this, I will be able tot eat this tomorrow |
|
@epugh any feedback? |
|
@adityamparikh I revamped the document to get down to just the essence... I wanted to simplify this so folks can more quickly consume the content. Thoughts? @janhoy --> This was inspired by, I think, your comment about the release email. |
janhoy
left a comment
There was a problem hiding this comment.
Thanks. This gives devs clear instructions on how to validate, not only that correct jars are produced, but that it also works for basic use case.
Perhaps later we are able to also include a simple smoke-mcp.py script that will perform all steps for you, using MCP-Inspector's --cli option to script some MCP tool calls.
|
Fiddled with the tool mcp-assert (https://github.com/blackwell-systems/mcp-assert) and these commands will assert the list-collections tool: JAR="./build/libs/solr-mcp-1.0.0-SNAPSHOT.jar"
export SOLR_URL="http://localhost:8983/solr/"
mkdir -p "evals"
cat > "evals/list-collections.yaml" <<YAML
name: list-collections returns the two seeded collections
server:
command: "java"
args: ["-jar", "$JAR"]
assert:
tool: list-collections
args: {}
expect:
not_error: true
not_empty: true
contains: ["books", "films"]
not_contains: ["nonexistent"]
min_results: 2
timeout: 30s
YAML
npx -y @blackwell-systems/mcp-assert run --suite "$REPO/evals/"The tool can also do an automated "audit", listing tools and checking that they are ok (read only) npx -y @blackwell-systems/mcp-assert audit --server "java -jar ./build/libs/solr-mcp-1.0.0-SNAPSHOT.jar" |
What
Adds
dev-docs/SMOKE_TEST.md— a fast, copy-paste end-to-end smoke test for verifying asolr-mcprelease JAR works across the three common MCP clients (Claude Code, Claude Desktop, MCP Inspector).Why
There's currently no single tester-facing checklist for confirming a release candidate (or a fresh local build) actually serves tools over stdio. This gives release verifiers a self-contained script with expected outputs at each step, so a broken JAR is obvious without a separate "expected results" reference.
What's in it
claude mcp add,claude_desktop_config.json,npx @modelcontextprotocol/inspector) — samejava -jar+SOLR_URLrecipe for each.list-collectionssanity check as the first smoke signal.create-collection→add-fields→index-json-documents→get-collection-stats→search) against a self-contained 61-document demo collection, with expected-output checkpoints (61 of 61 indexed,numDocs: 61,20 hits).pintfields don't reject them), and seven samplesearchqueries plus their natural-language equivalents.Placed in
dev-docs/alongside the existing contributor/release-verification guides (e.g.ATR_TESTING_GUIDE.md).Docs-only change — no source or build files touched.
🤖 Generated with Claude Code