Installing Dependencies

Make sure you have Python and Poetry installed (Poetry is a Python package manager).

If you don't have Poetry installed:

sudo apt update -y
sudo apt install -y pipx
pipx install poetry

# Optional
poetry config virtualenvs.in-project true

From the repo root:

poetry install

to install all required dependencies.

Input data

Forecast CSVs live under local/private_input_data/ (gitignored). For Summer 2026 the simulation script expects:

• local/private_input_data/pro_forecasts_2026_summer.csv
• local/private_input_data/bot_forecasts_2026_summer.csv

You can:

• If you have access, download prior analysis outputs (and sometimes inputs) from this Google Drive folder, or
• Request access to analysis data from the instructions shared here which will show you how to use the Metaculus Data Needs Form

Running the analysis

For the current (Summer 2026) tournament:

poetry run python aib_analysis/run_summer_simulations.py

That writes JSON artifacts to local/summer_2026_simulations/.

Check logs/latest_info.log for warnings (annulled questions, weight mismatches, unmatched questions, missing control bots, etc.).

For older seasons (Q1–Q4), use aib_analysis/run_q1_q2_simulations.py and edit the __main__ paths/folders as needed.

Viewing results

poetry run streamlit run aib_analysis/front_end.py

In the UI, set the tournaments folder to the simulation output path (e.g. local/summer_2026_simulations). Restart Streamlit or use Clear cache in the ⋮ menu after changing underlying JSON or loading/scoring code.

To get a markdown results summary (leaderboards, Pro vs bot head-to-head, per-type and team-size breakdowns, charts) from the same JSON outputs:

poetry run python aib_analysis/utils/summarize_season_results.py --simulations-folder local/summer_2026_simulations --season-name "Summer 2026"

That writes results_summary.md, CSV leaderboards, and PNG charts to local/summer_2026_simulations/results_summary/.

Pros vs bots write-up

To draft the season's results post from the same simulation outputs:

poetry run python -m aib_analysis.pro_vs_bot_writeup.run --season spring_2026 --html

It reads local/<season>_simulations/ and writes local/<season>_pro_vs_bot_writeup/: writeup.md (precommitted result sentences plus [===Fill me===] blocks with reminders), writeup_preview.md (the same text with charts embedded, for viewers that won't load local images), PNG charts, CSV leaderboards, and pro_name_key.csv (maps Pro aliases to usernames; keep it local). Add a new season, and anything the data can't provide, in aib_analysis/pro_vs_bot_writeup/season_config.py.

Bot-maker survey analysis

aib_analysis/survey_analysis/ joins the season's bot-maker survey to the bot leaderboard and writes a report with charts. Its inputs live in the folder named in config.py (local/private_input_data/summer_2026_survey_data/ for Summer 2026): the form responses (the response sheet downloaded as CSV), the payouts sheet's season tab as CSV (only the username, bot, in_aib and aib_prize columns are read), and the blank form printed to PDF (the run checks every option list against it with pdftotext). Survey answers are never edited by hand; write-ins count only through the committed rules in config.py, and the run stops on any research write-in they don't cover. Per-bot scores and answers stay in that folder, out of git. For a new season, start with SEASON_REVIEW_CHECKLIST in season_notes.py; the run refuses to start until REVIEWED_FOR_SEASON_START matches the season's start date.

poetry run python -m aib_analysis.survey_analysis.run [--refresh] [--blind]

--refresh rebuilds the leaderboard cache from the season's 2_bot_tournament.json. --blind withholds every result for the hypotheses in precommitted.py (and the alternatives listed there) until the picks have been reviewed. Outputs go to local/summer_survey_analysis/; aib_analysis/survey_analysis/render_html.sh renders them to HTML.

Bot maker advice post

To draft the season's "Advice from Bot Makers to Bot Makers" post from the bot maker survey (a CSV export of the form responses; see aib_analysis/bot_maker_advice/season_config.py for the path):

envh run --with OPENROUTER_API_KEY=<secret name>,METACULUS_TOKEN=<secret name> --reason "bot maker advice thematic analysis" -- \
    poetry run python -m aib_analysis.bot_maker_advice.run --season summer_2026

It runs a precommitted, LLM-assisted codebook thematic analysis (models, thresholds and selection rules are in method.py) and writes local/<season>_bot_maker_advice/: writeup.md (at most 5 themes in a fixed format with verbatim quotes, one-offs, open-source bots, every answer in full, and [===Fill me===] reminders), qa_review.md (every disagreement, tie-break, rejected citation and data-check match, to review before publishing), data/codes.csv, respondent_key.csv (maps answer IDs to bot names; keep it local), and llm_calls/ (every prompt and reply; reruns reuse them for free). Run --smoke-test first to check the plumbing with cheap models. If the season's survey analysis report exists (local/<season>_survey_analysis/), each theme is also checked against its significant results.

Structure

The project is built around SimulatedTournament (see data_models.py / simulated_tournament.py). It is initialized from Forecast objects and derives users, scores, etc. Most analyses are filtered, aggregated, or combined tournaments built from those forecasts.