Skip to content

docs: add tester quick-start smoke test guide - #158

Open
adityamparikh wants to merge 3 commits into
apache:mainfrom
adityamparikh:docs/smoke-test-guide
Open

docs: add tester quick-start smoke test guide#158
adityamparikh wants to merge 3 commits into
apache:mainfrom
adityamparikh:docs/smoke-test-guide

Conversation

@adityamparikh

Copy link
Copy Markdown
Contributor

What

Adds dev-docs/SMOKE_TEST.md — a fast, copy-paste end-to-end smoke test for verifying a solr-mcp release JAR works across the three common MCP clients (Claude Code, Claude Desktop, MCP Inspector).

Why

There's currently no single tester-facing checklist for confirming a release candidate (or a fresh local build) actually serves tools over stdio. This gives release verifiers a self-contained script with expected outputs at each step, so a broken JAR is obvious without a separate "expected results" reference.

What's in it

  • Client registration for all three clients (claude mcp add, claude_desktop_config.json, npx @modelcontextprotocol/inspector) — same java -jar + SOLR_URL recipe for each.
  • A list-collections sanity check as the first smoke signal.
  • A full build-and-query walkthrough (create-collectionadd-fieldsindex-json-documentsget-collection-statssearch) against a self-contained 61-document demo collection, with expected-output checkpoints (61 of 61 indexed, numDocs: 61, 20 hits).
  • Appendices with the schema, the 61-doc dataset (nulls pre-stripped so pint fields don't reject them), and seven sample search queries plus their natural-language equivalents.

Placed in dev-docs/ alongside the existing contributor/release-verification guides (e.g. ATR_TESTING_GUIDE.md).

Docs-only change — no source or build files touched.

🤖 Generated with Claude Code

adityamparikh and others added 2 commits June 19, 2026 12:41
A fast end-to-end smoke test for verifying a solr-mcp release JAR across
Claude Code, Claude Desktop, and MCP Inspector. Covers client registration
for all three transports, a list-collections sanity check, and a full
build-and-query walkthrough against a self-contained 61-document demo
collection with expected-output checkpoints and sample queries.

Lives in dev-docs/ alongside the other contributor/release-verification
guides.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: adityamparikh <aditya.m.parikh@gmail.com>
The demo URL is a shared public Solr, so the 'shows' collection may already
exist from a prior tester's run. create-collection is not idempotent and
errors on an existing name. Add guidance to use a unique collection name
(keeping the doc-count checkpoints valid) with a reuse/delete fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: adityamparikh <aditya.m.parikh@gmail.com>
@epugh

epugh commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Thanks for this, I will be able tot eat this tomorrow

@epugh epugh mentioned this pull request Jun 21, 2026
@adityamparikh

Copy link
Copy Markdown
Contributor Author

@epugh any feedback?

@epugh
epugh requested a review from janhoy July 23, 2026 17:03
@epugh

epugh commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

@adityamparikh I revamped the document to get down to just the essence... I wanted to simplify this so folks can more quickly consume the content. Thoughts?

@janhoy --> This was inspired by, I think, your comment about the release email.

@janhoy janhoy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. This gives devs clear instructions on how to validate, not only that correct jars are produced, but that it also works for basic use case.

Perhaps later we are able to also include a simple smoke-mcp.py script that will perform all steps for you, using MCP-Inspector's --cli option to script some MCP tool calls.

@janhoy

janhoy commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Fiddled with the tool mcp-assert (https://github.com/blackwell-systems/mcp-assert) and these commands will assert the list-collections tool:

JAR="./build/libs/solr-mcp-1.0.0-SNAPSHOT.jar"
export SOLR_URL="http://localhost:8983/solr/"

mkdir -p "evals"

cat > "evals/list-collections.yaml" <<YAML
name: list-collections returns the two seeded collections
server:
  command: "java"
  args: ["-jar", "$JAR"]
assert:
  tool: list-collections
  args: {}
  expect:
    not_error: true
    not_empty: true
    contains: ["books", "films"]
    not_contains: ["nonexistent"]
    min_results: 2
timeout: 30s
YAML

npx -y @blackwell-systems/mcp-assert run --suite "$REPO/evals/"

The tool can also do an automated "audit", listing tools and checking that they are ok (read only)

npx -y @blackwell-systems/mcp-assert audit --server "java -jar ./build/libs/solr-mcp-1.0.0-SNAPSHOT.jar"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants