Report attainable coverage on tournament leaderboards

[cemreinanc]

Summary

This PR introduces the concept of "attainable coverage" to the scoring system, which represents the maximum coverage a forecaster could achieve on a question based on when it actually closed relative to its scheduled close time. This allows the system to fairly evaluate coverage on questions that resolve early.

Key Changes

• New get_attainable_coverage() method on Question model that calculates the fraction of the scheduled forecasting window during which a question was actually open:

  • Returns 1.0 for questions that stay open until/past their scheduled close time
  • Returns < 1.0 for questions that close early (e.g., resolve before scheduled close)
  • Calculated as: (effective_close_time - open_time) / (scheduled_close_time - open_time)

• Updated Contribution tracking to include attainable_coverage field alongside existing coverage field, populated only for successfully resolved questions

• Enhanced coverage metrics in project contributions display:

  • Individual question coverage now shows both actual and attainable coverage: "X% (Y%)"
  • Added three new summary metrics:
    • Total Coverage: Average coverage across resolved questions
    • Total Attainable Coverage: Average attainable coverage across resolved questions
    • Effective Coverage: Question-weighted ratio of actual to attainable coverage
  • Updated column header from "coverage" to "Coverage (max)" for clarity

• Updated leaderboard max_coverage calculation to use weighted attainable coverage instead of just question weights, ensuring coverage is measured against what was actually achievable

• Added comprehensive test coverage for the new get_attainable_coverage() method with multiple scenarios (full window, early close, exact scheduled close)

• Updated UI labels and help text in English translations to explain the new attainable coverage concept

Implementation Details

• The attainable_coverage is only populated for successfully resolved questions (excludes ambiguous/unknown resolutions)
• A helper function _get_attainable_coverage() ensures consistent logic across the codebase
• The effective coverage metric uses weighted sums to account for question importance
• Null/undefined coverage values are formatted as "-" for better UX

https://claude.ai/code/session_01NmzezEyuu7d2sX214hZWqZ

Summary by CodeRabbit

• New Features
  • Added maximum, attainable, and effective coverage metrics to contribution views.
  • Per-question coverage now reflects the portion of the forecasting window when questions were available.
  • Added explanations for question-level and tournament-level coverage calculations.
• Bug Fixes
  • Improved coverage calculations for resolved questions, including weighting and actual close times.
  • Updated leaderboard and contribution data to support attainable coverage when applicable.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

📝 Walkthrough

Walkthrough

Coverage calculations now account for early question closure. Scoring data exposes attainable coverage, and the contributions UI displays total, attainable, effective, and maximum coverage with updated explanations.

Changes

Coverage metrics

Layer / File(s)	Summary
Attainable coverage calculation questions/models.py, tests/unit/test_questions/test_models.py	Question.get_attainable_coverage() calculates the fraction of the scheduled window that a question remained open. Tests cover full and half-window availability.
Scoring data propagation scoring/utils.py, scoring/serializers.py	Contributions include nullable attainable coverage. Leaderboard maximum coverage uses weighted attainable coverage for successful resolutions.
Contribution coverage display front_end/src/types/scoring.ts, front_end/src/app/(main)/(leaderboards)/contributions/components/project_contributions.tsx, front_end/messages/en.json	The UI displays total, attainable, effective, and maximum coverage. Localization adds labels and explanations for the new metrics.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Question
  participant ScoringUtils
  participant LeaderboardSerializer
  participant ContributionsUI
  Question->>ScoringUtils: Calculate attainable coverage
  ScoringUtils->>LeaderboardSerializer: Include weighted coverage data
  LeaderboardSerializer-->>ContributionsUI: Serialize contribution metrics
  ContributionsUI->>ContributionsUI: Display total, attainable, effective, and maximum coverage

Loading

Possibly related issues

• Tournament leaderboards should report attainable coverage #3757 — The PR implements attainable, maximum, and effective coverage calculations and leaderboard reporting described by the issue.

Possibly related PRs

• Metaculus/metaculus#4537 — Both changes implement attainable or maximum coverage calculations and update coverage UI and localization.
• Metaculus/metaculus#4681 — Both changes modify LeaderboardSerializer.get_max_coverage to use question coverage weights.

Suggested reviewers: hlbmtc, lsabor

Poem

A rabbit watched the questions close,
And measured time through every rose.
“Now attainable scores are clear,
With effective coverage hopping near!”
The leaderboard shines bright—
Metrics sorted just right.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: reporting attainable coverage on tournament leaderboards.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/attainable-coverage-3757

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@front_end/messages/en.json`:
- Around line 927-929: Update the effectiveCoverageInfo translation to state
that both total coverage and total attainable coverage are weighted by each
question’s question_weight before division, while preserving the leaderboard
equivalence description.

In
`@front_end/src/app/`(main)/(leaderboards)/contributions/components/project_contributions.tsx:
- Around line 114-120: Update the attainable_coverage display branch in the
project contributions component to pass contribution.coverage directly to
formatPercent instead of coercing nullish values to 0. Preserve the existing
attainable coverage formatting and render null coverage as the formatter’s "-"
fallback.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 93321de6-2d38-40b5-bc0c-b2dec11b1b3b

📥 Commits

Reviewing files that changed from the base of the PR and between 983ccda and 48e4fcc.

📒 Files selected for processing (7)

• front_end/messages/en.json
• front_end/src/app/(main)/(leaderboards)/contributions/components/project_contributions.tsx
• front_end/src/types/scoring.ts
• questions/models.py
• scoring/serializers.py
• scoring/utils.py
• tests/unit/test_questions/test_models.py

[github-actions]

🚀 Preview Environment

Your preview environment is ready!

Resource	Details
🌐 Preview URL	https://metaculus-pr-5124-attainable-coverage-3757-preview.mtcl.cc
📦 Docker Image	ghcr.io/metaculus/metaculus:attainable-coverage-3757-558a4da
🗄️ PostgreSQL	NeonDB branch preview/pr-5124-attainable-coverage-3757
⚡ Redis	Fly Redis mtc-redis-pr-5124-attainable-coverage-3757

Details

• Commit: fc4e4aa51516453096ecdc0cb2710589450bd7bf
• Branch: claude/attainable-coverage-3757
• Fly App: metaculus-pr-5124-attainable-coverage-3757

---

ℹ️ Preview Environment Info

Isolation:

• PostgreSQL and Redis are fully isolated from production
• Each PR gets its own database branch and Redis instance
• Changes pushed to this PR will trigger a new deployment

Limitations:

• Background workers and cron jobs are not deployed in preview environments
• If you need to test background jobs, use Heroku staging environments

Cleanup:

• This preview will be automatically destroyed when the PR is closed

[cemreinanc]

@SylvainChevalier do you think this is working and ready?

[SylvainChevalier]

@claude the leaderboard scoring terminology section should mention that the coverage column is effective coverage, and define it.

[claude]

Claude finished @SylvainChevalier's task in 3m 52s —— View job

---

Done — Coverage defined in the scoring terminology section

Pushed as 1ab50d5.

• Read repo guidelines & PR context
• Find the leaderboard scoring terminology section
• Add an effective-coverage definition for the Coverage column
• Lint / format
• Commit

What changed

project_leaderboard_client.tsx — the Scoring Terminology toggle previously defined only Score. It now has a second Coverage entry, gated on isAdvanced since the Coverage column itself is only rendered in advanced mode (project_leaderboard_table/index.tsx:104-113). The link points at /help/scores-faq/#coverage, matching the anchor used elsewhere in the leaderboards code.

en.json — new leaderboardCoverageInfo key:

Your effective coverage: the question-weighted sum of your Coverages on all resolved questions in the tournament, divided by the question-weighted sum of the maximum coverage attainable on those questions. The maximum is less than 100% on questions that closed early, so this measures how much of the coverage that was actually available you captured. 100% means you had full coverage on every question.

That wording tracks what the column actually computes: coverage / max_coverage in table_row.tsx:45-48, where max_coverage is sum(attainable_coverage * question_weight) from LeaderboardSerializer.get_max_coverage (scoring/serializers.py:93-118), and it lines up with the effectiveCoverageInfo text on the contributions page, which already claims to match the leaderboard.

Notes

• The string was added to en.json only, consistent with the other keys this PR introduces (totalCoverageInfo, effectiveCoverageInfo, …) — the other locales are filled in by the translation pipeline rather than with English text.
• bun run lint:types passes clean. bun run lint:js exits 1 on a pre-existing react-hooks/static-components error in src/stories/utils/renderer/conditional-renderer.tsx, untouched by this change; no new warnings in the files I edited. bun run format reformatted the new JSX block only.
• I did not run a build; nothing outside these two files changed.
  • branch claude/attainable-coverage-3757

[SylvainChevalier]

Added in cb4ab96. The leaderboard's scoring terminology section now has a Coverage entry alongside Score:

Coverage — The Coverage column shows your effective coverage: the sum of your coverage over all resolved questions in the tournament, divided by the sum of the maximum coverage attainable on those questions, with both sums weighted by question weight. Questions you didn't forecast count as zero coverage. A question's attainable coverage is less than 100% when it closed early, so this measures you against what was actually achievable rather than against every question's full scheduled lifetime.

Two notes on how it's wired:

• It's gated behind the Advanced toggle, matching the Coverage column itself, which only renders in the advanced view. Defining a column that isn't on screen seemed more confusing than helpful — easy to ungate if you'd rather it always show.
• I left the column header as "Coverage" rather than renaming it to "Effective Coverage", since the ask was to explain it in the terminology section. Leaderboards can already retitle columns via display_config.column_renames if you want it renamed per-tournament.

---

Generated by Claude Code

[SylvainChevalier]

Build Docker Image failed on cb4ab96, and I've pushed a fix in 8422886.

The failure wasn't from this diff:

bash: .github/scripts/slugify_branch.sh: No such file or directory
##[error]Process completed with exit code 127.

The build workflow calls .github/scripts/slugify_branch.sh to derive the image tag. That script exists on main but was added after this branch last synced (Aug 8), so it wasn't in the checked-out tree and the step exited 127 before any build work started. Any sufficiently stale branch would hit this.

Fix was to merge main into the branch, which brings the script in. The merge was clean — no conflicts — and I verified the coverage changes survived it intact. That also clears the behind state the PR had been sitting in.

---

Generated by Claude Code

[SylvainChevalier]

Frontend Checks failed on 1ab50d5; fixed in 558a4da.

lint:types reported ~30 TS2345 errors, all missing translation keys referenced from midterms-2026/*, comment.tsx and private_note.tsx — files this PR never touches:

error TS2345: Argument of type '"midtermsHubTimelineUnavailable"' is not assignable
to parameter of type 'NamespacedMessageKeys<...>'

Cause: 1ab50d5 rewrote en.json from a working copy that predated the main merge in 8422886, so writing the file back reverted the 27 keys main had added since Aug 8. Key counts tell the story:

commit	keys in en.json
8422886 (merge of main)	2430
1ab50d5	2403
558a4da (this fix)	2430

558a4da restores the merged content and re-applies the leaderboardCoverageInfo wording from 1ab50d5 — the text is unchanged from what that commit set. Against main, en.json now has 8 additions and 0 removals, those 8 being exactly the keys this branch introduces (coverageMax, totalCoverage*, effectiveCoverage*, coverageMaxInfo, leaderboardCoverageInfo).

I edited the file textually rather than parsing and re-emitting it, because en.json carries two pre-existing duplicate keys on main (excludeBots, bots) that a JSON round-trip would silently collapse. Those duplicates are unrelated to this PR but probably worth a separate cleanup, since duplicates resolve last-one-wins.

---

Generated by Claude Code