[vahid-ahmadi]

Stacked on #221 (base vahid/r5-pipeline). Recovers the daily snapshots in #222. This PR adds no file under data/vintages/. It adds the tool and its tests; applying it is a separate step for Vahid (order below).

What it does

data/recover_orphaned_snapshots.py reads data/archive-orphaned-snapshots-2026-11 (77f3176). Its parents after the first are the 50 orphaned bot commits.

1. For each orphan it lists the vintage files that orphan added against its own parent (git diff-tree --diff-filter=A).
2. Per series, oldest first, it drops a file whose payload equals the snapshot before it, from the store or from files already kept. The comparison is fetch.same_data: observations, release stamp, next release, last period, title and units, not fetched_utc. The first day a release was seen is kept.
3. A path that already exists with identical bytes counts as present, so a re-run adds nothing. Different bytes stop the run. Nothing is overwritten.
4. --apply writes each kept file byte for byte from its orphan commit, so vintage and fetched_utc are the originals. It then rebuilds MANIFEST/latest with fetch.py's own functions and fails if any tracked vintage was modified, renamed or deleted.

Without --apply it is a dry run. --against REF plans against another commit's store, and --summary writes the table as markdown.

data/refresh.py change: the carry step now also drops a branch file whose release repeats the snapshot before it (drop_repeats). Without this, once the recovery is on main, the bot branch would carry its 5 Nov copies of releases main already holds.

Dry run on the real archive (6 Nov)

50 orphan commits, 818 added vintage files.

store it is applied to	add	repeat days dropped	already present	existing later file repeating a recovered release
main today (before #195)	270	548	0	0
#195 head 18442ea (after #195)	265	533	20	15

Files to add per series (before #195): bank rate, gilt 5y/10y/20y and US 10y Treasury 48 each (daily series); US real GDP and fed funds 3 each; US CPI, payrolls, unemployment 2 each; ONS AWE, core CPI, CPI, monthly GVA, PSNB, PSND, unemployment and vacancies 2 each; ONS GDP and business investment 1 each.

A scratch apply (not pushed) gave data/fetch.py --check OK with 346 files across 20 series, tests/test_data_store.py 2386 passed and 79 skipped, an empty MDR diff, and a re-run adding 0. The post-#195 case gave 361 files OK and 2486 passed.

How it interacts with #195, and the order

Two reasons for recovering before #195:

• Repeats. Refresh official data vintages #195's 15 non-daily 5 Nov files repeat releases first fetched earlier (CPI 21 Oct, GDP 30 Sep, and so on). If Refresh official data vintages #195 lands first, those 15 become permanent same-release repeats, because the store is append-only.
• Outturn vintage. ingest_outturns.py reads latest/. Recovered first, CPI 2026Q3 3.10 is recorded under vintage 2026-10-21 and GDP 2026Q2 1.37 under 2026-09-30, the first days we held them. If Refresh official data vintages #195 goes first, both are recorded under 2026-11-05. No published score changes either way, because the only scored round (29 Jul) predates both dates.

After #195 still works. The script inserts files behind 5 Nov, and the gates pass.

Order for Vahid:

1. Merge Data pipeline: fetch, ingest, score and open one checked PR; stop the nowcast's silent no-op #221, then this PR.
2. On a fresh branch from main, run python3 data/recover_orphaned_snapshots.py --apply --summary recovery.md. Commit the vintages, MANIFEST and latest first, with the table in the message.
3. Then run python3 data/refresh.py --skip-fetch. With the vintages committed it writes no notes for about 49 back-filled dates (which would be legacy per-series pages that conflict with Redesign the static site: Autumn Budget look, one fact per home, readable layouts #200). It ingests, scores, rebuilds pages and runs every gate. Commit and open the PR as yourself so CI runs on it.
4. Merge that before Refresh official data vintages #195, ideally before the 12 Nov GDP release.
5. Then the fetch dispatch from Data pipeline: fetch, ingest, score and open one checked PR; stop the nowcast's silent no-op #221's order updates Refresh official data vintages #195. Carry skips the 20 files main now has and the 15 repeats.

Nothing is merged, no workflow has been dispatched, and no vintage is written to any branch.

Tests: tests/test_recover_orphaned_snapshots.py has 11 tests. Most are pure planning tests. The rest are end to end on a synthetic repo with main, two orphans and an octopus archive: dry run writes nothing; apply writes the original bytes, passes fetch.py --check with an empty MDR diff, and is idempotent; a conflict is refused. There is also one test for refresh.drop_repeats. All of them fail on #221's head, where the module and function don't exist.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
macromod	Ready	Preview	Nov 6, 2026 3:51am UTC