A benchmark and open call for valuable, reproducible RL runs. Each task is a real training run that went wrong. The agent has to find the root cause (issue) and, where a repaired run exists, fix it and rerun (solution).
- 24 Oct 2026 — Call for contributions officially open, and Kimi-K3.1-Pro added to the leaderboard.
- 22 Oct 2026 — Leaderboard v0.1 released, covering 14 tasks.
- 19 Oct 2026 — GitHub repo and website released.
AutoRLBench Leaderboard 0.1, issue evaluation. First release; updated as new agents and tasks are added.
| # | Model | Agent | Issue score | |
|---|---|---|---|---|
| 1 | Claude-Fable-5.1 | Claude Code | 85.7 % | █████████████████░░░ |
| 2 | Grok-4.7 | Grok Build | 78.6 % | ████████████████░░░░ |
| 3 | Kimi-K3.1-Pro | Claude Code | 71.4 % | ██████████████░░░░░░ |
| 4 | GPT-6-Astra | Codex | 64.3 % | █████████████░░░░░░░ |
| 4 | GLM-5.3 | Claude Code | 64.3 % | █████████████░░░░░░░ |
| 6 | GPT-6.1-Sol | Codex | 57.1 % | ███████████░░░░░░░░░ |
| 6 | Qwen3.8-Max | Qwen Code | 57.1 % | ███████████░░░░░░░░░ |
| 8 | Kimi-K3 | Claude Code | 42.9 % | █████████░░░░░░░░░░░ |
| 8 | Gemini-4-Argon | Claude Code | 42.9 % | █████████░░░░░░░░░░░ |
| 10 | DeepSeek-V4-Pro | Claude Code | 21.4 % | ████░░░░░░░░░░░░░░░░ |
Solution evaluation (repair and rerun) is coming soon. Full results: website.
Six domains, one representative task each. The domains come from where in the RL pipeline the failure lives:
Configuration · issue · solutionAIME25 single-sample variance One sample per question makes a 30-question score too noisy to judge the run. |
Reward · issue · solutionLength bonus reward hack An unconditional length bonus teaches the policy to write longer, not better. |
Data · issueTraining data is too hard Train is TACO VERY_HARD while validation is EASY; held-out accuracy ends below its start.
|
Environment · issue · solutionZero query vector in retrieval The retriever's query vector is all zeros, so every query gets the same documents. |
Algorithm · issueGRPO uid assigned after rollout repeat The uid assignment lands after the rollout repeat step, making every GRPO group a singleton and silently degrading training to baseline-free REINFORCE. |
Infrastructure · issue · solutionMoE auxiliary gradients miss token normalization The per-token loss callback omits the token count, so MoE z-loss gradients are never normalized. |
prepare ──▶ agent ──▶ [rerun] ──▶ verifier
│ │ │ │
│ │ │ └─ issue: LLM judge on agent_answer.md vs. hidden oracle
│ │ │ solution: a metric threshold on the rerun's logs
│ │ └─ solution only: the harness re-runs the experiment on the agent's
│ │ workspace, per the task's run_spec.json
│ └─ reads instruction.md and a writable copy of workspace/
└─ fetches pinned models and datasets (the only stage with network)
Each stage runs in its own container from the same prebuilt image. Oracles and verifiers are mounted only into the verifier stage.
At the solution level the agent repairs the workspace and submits report.md — it does not launch training itself. The harness clears the task's log roots, re-runs the experiment on the repaired workspace, and grades only that fresh run. No report.md, no rerun and no grading: the task scores 0. A solution task without a run_spec.json (a pure protocol repair, say) skips the rerun but is bound by the same report gate.
| Path | Contents |
|---|---|
tasks/ |
Benchmark tasks, grouped by domain |
autorlbench/ |
Evaluation harness: task loader, validator, runtimes, agent adapters, CLI |
docker/ |
Shared environment images and in-container tooling (README) |
skills/package-task/ |
Agent skill that turns a raw task into a task |
tests/ |
Harness unit tests |
page/ |
Website source, deployed to GitHub Pages |
tasks/<domain>/<task_id>/
├── README.md # summary, difficulty, reference solution, verification
├── task.toml # metadata and environment image
├── workspace/ # the run as the agent sees it: code, config, logs, metrics
├── environment/prepare.sh # fetches pinned models and datasets (only stage with network)
└── evaluations/
├── issue/ # instruction.md · oracle.md · verifier/
└── solution/ # instruction.md · oracle/ · verifier/ (grader_spec.json · run_spec.json)
# Validate a task
python3 -m autorlbench validate tasks/reward/slime_retool_length_reward_bonus
# Prepare and run the verifier on a task in Docker (no agent)
python3 -m autorlbench evaluate tasks/reward/slime_retool_length_reward_bonus \
--evaluation-level issue --backend docker \
--judge-model <judge-model> --json
# Run an agent through Harbor (pip install harbor)
python3 -m autorlbench harbor-evaluate tasks/reward --agent codex --model <model>Run python3 -m autorlbench --help for every option. The issue verifier calls an OpenAI-compatible judge through OPENAI_BASE_URL and OPENAI_API_KEY. evaluate runs no agent, so at --evaluation-level solution it stops at the report gate and scores 0 unless the staged workspace already carries a report.md.
We welcome both problematic runs and healthy reference runs. Bring the issue materials, or add the repaired run when it exists; an agent does the packaging. Contribute two valid tasks and you become a co-author on the benchmark paper. See CONTRIBUTING.md and the contribution guide.
Apache-2.0. Each task's workspace keeps the license of its upstream code.