{venue}
+ {title}
+{authors}
+ { + url && ( + + {url} ↗ + + ) + } + +
+
+
+
+ + BibTeX +
+{bibtex}{authors}
+ { + url && ( + + {url} ↗ + + ) + } + +{bibtex}+ QueryGym is split into four surfaces. They share the same data contract, + so a run from the dashboard or a third-party submitter lands in the same + leaderboard as the toolkit's own results. +
+ +{c.body}
++ QueryGym pairs a small, opinionated library with a contract-driven + reproducibility pipeline. The toolkit, the dashboard, and the leaderboard + all share the same data shape. +
+{f.body}
++ QueryGym is a toolkit for benchmarking and reproducing LLM-based query + rewriting methods across IR datasets. Open prompt bank, pluggable + searchers, frozen schemas, citable runs. +
+ +{m.id}
+ {m.tagline}
+ {variant === "full" && m.paper_url && ( + + {m.paper} ↗ + + )} ++ QueryGym is built by the LS3 Lab with collaborators + across the IR community. The aim is to make LLM-based query reformulation + research easy to reproduce: same prompt bank, same data shape, + same retrieval contracts, regardless of which LLM, dataset, or method + you're benchmarking. +
+ ++ Amin Bigdeli · Radin Hamidi Rad · Mert Incesu · Negar Arabzadeh · + Charles L. A. Clarke · Ebrahim Bagheri +
+ ++ Apache 2.0. Free to use commercially. Schema and reproducibility + artifacts are in the public domain (CC0). +
+ ++ New methods, datasets, and reproducibility runs are welcome. See the + + contributor guide + + and the + + reproducibility flow + . +
++ Two papers back QueryGym. Use the toolkit citation if you're using the + library; add the reproducibility citation if you're comparing against + the SIGIR 2026 leaderboard numbers. +
+ ++ Reformulate a query against any OpenAI-compatible endpoint. Pyserini and + BEIR are optional extras. +
++ Nine reformulation methods, each with a registered prompt, a paper + reference, and a reproducibility entry on the leaderboard. +
++ The QueryGym Dashboard is the hosted product layer — login, run + methods through a UI, compare results live, manage API keys, and + publish to the leaderboard. +
++ QueryGym runs on Python 3.9+. The default install is dependency-light; + add extras for HuggingFace datasets, BEIR, Pyserini, or the + reproducibility tooling. +
+ ++ Reformulate a single query with the GenQR Ensemble method and any + OpenAI-compatible endpoint. +
++ Full guide: + + querygym.readthedocs.io ↗ + +
+
+ Each method ships with a registered prompt in the prompt bank, an
+ @register_method(...) entry, and a paper
+ reference. Switch between them by name in the API or CLI.
+
+ Adding a new method? + + Contributor guide ↗ + +
++ Every QueryGym run produces a single JSON conforming to a public, + versioned schema. Submissions to the leaderboard carry the JSON, a + TREC-format run file, and the reformulated queries — together they + reconstruct the experiment from a fresh clone. +
+ ++ Browse the full SIGIR 2026 reproducibility leaderboard. Per-dataset, + per-method, per-LLM views with citable run files. +
++ Field-by-field documentation, validation rules, and a worked + example. Mirrors the canonical JSON Schema file. +
+
+ Run the example pipeline, then use submit_run.py
+ and open a PR. CI validates the JSON against the schema; a maintainer
+ verifies the numbers locally before merge.
+
+ QueryGym is backed by two papers: the toolkit demo and a multi-LLM + reproduction study. Both link directly to the committed corpus. +
+ +