Skip to content

Refresh the data to 2026Q2 and regenerate the committed artefacts - #24

Draft
vahid-ahmadi wants to merge 5 commits into
vahid/yoy-simple-ratesfrom
vahid/data-2026q2
Draft

vahid-ahmadi wants to merge 5 commits into
vahid/yoy-simple-ratesfrom
vahid/data-2026q2

Conversation

@vahid-ahmadi

@vahid-ahmadi vahid-ahmadi commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Merge prerequisite: #18 (base vahid/yoy-simple-rates). Merge #18 first.

Why

No new PolicyEngine Macro forecast round is possible while the hosted SVAR's data end in 2026Q1: the next round has to condition on 2026Q2, and the round machinery refuses a repeated data edge (PolicyEngine/macro#224). ONS has now published 2026Q2. This PR refreshes the packaged dataset to 2026Q2 from the same official sources scripts/download_data.py already uses. It then regenerates every committed artefact.

Stacked on #18 (base vahid/yoy-simple-rates); merge #18 first. On main, the staleness hash (evaluation_code_version) covers code only, so a data-only refresh would leave the gate green without proving that anything was regenerated. #18 adds the dataset to the hash. That is what makes "the staleness gate passes" below a real check: every evaluation JSON here was regenerated on the new CSV, or the gate would fail.

No methodology change. Estimation still ends at 1992Q1-2025Q1, with the same lags, priors, seeds, draw counts and Covid dummies. Only the conditioning edge (forecast origin T) moves, from 2026Q1 to 2026Q2. The rolling and coverage evaluations gain one origin (49 -> 50) because their last origin is T - 9, which is what those scripts do by design.

Data (vintage table now in data/README.md)

Source Series Release Last obs
ONS QNA (UKEA) ABMI 30 Sep 2026 2026Q2
ONS CPI (MM23) D7BT, D7CH, D7EC 21 Oct 2026 Sep 2026
ONS LMS MGSX (satellite) 20 Oct 2026 2026Q2
BoE IADB IUQABEDR, XUQABK67, XUQAUSS downloaded 6 Nov 2026 2026Q3
FRED Brent, US/EA/DE/JP GDP, EA HICP, US CPI (+ the unchanged China/Japan legs) downloaded 6 Nov 2026 2026Q2-Q3
  • The binding edge is 2026Q2, set by euro-area and Japanese GDP. The build is capped at 2026Q2 (download_data.py: 1990Q1:2026Q2; the IADB Dateto moves to 1 Jan 2027).
  • data/boe_var_data.csv and src/boe_var/_data/boe_var_data.csv are byte-identical, at 138 rows, 1992Q1-2026Q2.
  • uk_unemployment_mgsx.csv gains 2026Q2 = 4.9%. The 1971Q1-2026Q1 values are unchanged in the 20 Oct LMS release.
  • History was revised. ONS Blue Book 2026 (in the 30 Sep QNA) moved uk_gdp: the largest change to a quarterly growth rate is 0.18 log points, and the level is 2.8-4.3 log points higher. FRED revisions moved world_gdp by up to 0.14 log points in growth. Every other column moves by at most 0.003. So even with the coefficient window unchanged, the coefficients move slightly.
  • The 2026Q2 quarter is large: real oil +21.6 log points, CPI energy +6.2, world CPI +1.9 (EA HICP +2.1, US CPI +1.5), UK CPI +0.8, UK GDP +0.53. I checked these against the raw series. They are not a build error.

Regeneration (GitHub Actions, not the laptop)

Each script ran with its committed defaults on a throwaway branch, tmp/data-2026q2-regen (run 42154335977). That branch has been deleted. The workflow file is not in this PR, and it is not part of any hash.

Artefact Script Draws / accepted / ESS (old -> new)
results/rolling_evaluation.json run_rolling_evaluation.py origins 49 -> 50
results/coverage_evaluation.json run_coverage_evaluation.py origins 49 -> 50
results/unemployment_satellite_validation.json run_satellite_validation.py -
results/forecast_summary.md, fig1, fig7-9 run_forecast_revision.py --draws 6000 430 / ESS 197.9 -> 455 / ESS 181.4
results/summary.md, fig2-6 run_replication.py --draws 10000 10000 / 744 / ESS 356.1 -> 10000 / 765 / ESS 368.0
results/forecast_vs_boe_obr.md (new top section) run_external_comparison.py 449 / ESS 210.9 -> 473 / ESS 209.0 (pre-Covid 296 / 167 -> 260 / 118.9)

Replication (summary.md): same 1992Q1-2025Q1 sample, revised history

  • FEVD at 1 year, renormalised: UK GDP identified global/domestic/unidentified 42.8/37.7/19.5 -> 43.2/36.5/20.3%. UK CPI 49.5/34.0/16.5 -> 48.0/34.0/18.0%.
  • Per-draw group-share means (the paper-comparable statistic): UK GDP global 37.4 -> 38.7%, UK CPI global 42.3 -> 41.5%. The reading against the paper (about 40% GDP, about 50% CPI) does not change: it matches on GDP and falls short on CPI.

Forecast, old (T = 2026Q1) -> new (T = 2026Q2), production sample

CPI inflation, four-quarter %, simple rate:

Quarter Median 68% band
2026Q3 2.64 -> 3.00 1.90-3.37 -> 2.51-3.43
2027Q3 3.00 -> 3.44 1.68-4.32 -> 2.02-4.77
2028Q3 2.98 -> 3.14 1.67-4.51 -> 1.76-4.61
2029Q3 3.02 -> 3.17 1.60-4.49 -> 1.67-4.62

GDP growth, four-quarter %:

Quarter Median 68% band
2026Q3 1.37 -> 1.59 0.59-2.21 -> 1.04-2.17
2027Q3 1.47 -> 1.20 0.19-2.68 -> -0.06-2.40
2028Q3 1.46 -> 1.23 0.06-2.75 -> 0.02-2.50
2029Q3 1.53 -> 1.35 0.10-2.80 -> 0.03-2.65
Quarter Unemployment % Energy contribution pp Bank Rate %
2026Q3 4.96 -> 4.87 0.24 -> 0.68 3.76 -> 3.85
2027Q3 4.92 -> 4.84 0.08 -> 0.12 3.81 -> 3.92
2028Q3 4.88 -> 4.88 0.06 -> 0.05 3.87 -> 3.90
2029Q3 4.84 -> 4.89 0.06 -> 0.08 3.91 -> 3.91

2026Q3 is now one quarter ahead, where before it was two, so its band is narrower by construction. The direction of every change follows from the 2026Q2 energy quarter now being in the conditioning data: the identified shock at T is a world energy shock, with posterior weight 0.98 on the inflationary sign (forecast_summary.md). Near-term CPI and the energy contribution rise; GDP from 2027 is 0.2-0.3pp weaker. The composite-IRF peak CPI effect is +0.22 -> +0.66pp at h3, and the GDP effect +0.27pp at h0 -> -0.36pp at h6. The Bank comparison columns in forecast_vs_boe_obr.md (July MPR, March EFO) are unchanged. Parts 1-3 of that file are the 22 Sep analysis, and the new top section says which of its numbers are superseded.

Published numbers that move (pins in tests/test_committed_artifacts.py, no tolerance changed)

  • Rolling, h1 vs RW: world_gdp 1.0450 -> 1.0433, world_cpi 0.7329 -> 0.7334, oil 1.0301 -> 1.0276, bank_rate 0.8809 -> 0.8814, eri 1.0386 -> 1.0379, cpisa 0.6293 -> 0.6286, cpi_energy 0.9762 -> 0.9771, uk_gdp 1.0637 -> 1.0631.
  • Rolling, h1 vs drift: 1.0710/0.9433/1.0135/0.7933/1.0299/0.8306/0.9771/1.0635 -> 1.0716/0.9440/1.0102/0.7950/1.0287/0.8313/0.9730/1.0634 (same variable order).
  • Rolling, h1 vs AR(1): 0.7227/0.9713/1.0199/1.3822/1.0680/1.0284/0.9802/0.6984 -> 0.7223/0.9701/1.0156/1.3789/1.0675/1.0289/0.9813/0.6987.
  • Rolling, h8 vs RW: 0.7108/0.6741/1.0837/1.0280/1.4104/0.6731/1.1343/1.0581 -> 0.7003/0.6682/1.0848/1.0255/1.4122/0.6681/1.1342/1.0471.
  • Production spec vs drift, h1 (uk_gdp/cpisa/bank_rate): 0.9873/0.8232/0.7809 -> 0.9874/0.8237/0.7835; h8: 0.9469/1.0117/0.8298 -> 0.9461/1.0113/0.8319.
  • Coverage, 68% band mean by horizon: min 0.5663 -> 0.5800, max 0.7117 -> 0.7125; 90% band mean 0.770 -> 0.777.
  • Unchanged: the win counts against RW (4,4,4,3,3,3,3,3). The qualitative claims also all still hold: no result survives multiplicity control against drift, UK GDP is still not significantly better than drift, and the Covid dummies still help UK GDP.

Checks

What Vahid has to do for the macro round machinery to use this

  1. Merge Compare model YoY rates with BoE/OBR on the simple-rate basis; hash the data #18, then this PR, into main (merge commits, so the sha is stable). Note the new main sha S.
  2. In PolicyEngine/macro, change both pins to S in one commit:
    • integration/modal_app.py:133 BOE_REVISION = "S"
    • integration/pyproject.toml:53 boe_var @ git+https://github.com/PolicyEngine/boe-var-model@S
  3. Redeploy the Modal app (modal deploy integration/modal_app.py) so the hosted image rebuilds at S. Do this before the next due day (12 Nov per #224).
  4. Merge macro #221 (or have it merged) for the edge gate. forecasts/nowcast.py --edge-gate reads BOE_REVISION, fetches data/boe_var_data.csv at that sha and takes its last quarter (2026Q2). The required quarter is the one after the newest archived edge (2026Q1 -> 2026Q2), so the gate passes by itself and the round runs. data_edge in the round comes from the model's forecast_origin, so it will read 2026Q2.
  5. Cosmetic follow-ups in macro, which do not block the round: hardcoded "2026Q1" vintage strings in integration/src/policyengine_macro/capabilities.py:89-90 (data_edge, data_vintage), core.py:4362 (data_vintage of the committed-artefact path), core.py:1366 (comment), and the site copy in svar/index.html, svar/methodology/index.html, svar/validation/index.html and economy/topics/{growth,inflation}/index.html ("conditioning data through 2026Q1"). The hosted 2,000-draw accepted/ESS figures quoted on svar/methodology will also change after the redeploy.

Not done here: no forecast round created, nothing in macro edited, nothing deployed.

🤖 Generated with Claude Code

vahid-ahmadi and others added 4 commits November 6, 2026 03:04
…20 Oct)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the 2026Q2 data

Run on GitHub Actions (throwaway branch tmp/data-2026q2-regen, run 42154335977)
with each script's committed defaults. Pins in test_committed_artifacts move
with the new origin (49 -> 50) and the ONS revisions; no tolerance changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s_boe_obr.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_replication.py --draws 10000 on Actions run 42154335977: 744 -> 765
accepted, ESS 356.1 -> 368.0. Sample unchanged (1992Q1-2025Q1); the moves
come from the Blue Book 2026 revisions to UK GDP history.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….py (01/Jan/2027)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi

Copy link
Copy Markdown
Contributor Author

Independent verification of #24 (checked at 422bd5b; CI green)

Result: the refresh is correct. I found one stale doc URL and fixed it in d4ec3e7. Merging #24 changes the published replication figures (details below).

1. Scope. The only code change is two literals in scripts/download_data.py: the end cap moves 2026Q1 → 2026Q2, and the IADB Dateto moves to 01/Jan/2027. Everything else is docstrings, SPEC/methodology wording, test pins (values only, tolerances unchanged), data and regenerated artefacts. There is no method change. Fix pushed: data/README.md:20 still had Dateto=01/Jul/2026 and now matches the script.

2. The 2026Q2 values, against my own fetches from the ONS generator and BoE IADB:

Series Official Repo
ABMI 2026Q2 (30 Sep 2026) £744,453m exp(13.520405) = 744,453
IHYQ q/q 0.5% 0.53%
D7BT 2026Q2 / D7G7 142.3 / 2.8% 142.3 (raw), yoy 2.77%
MGSX 2026Q2 (20 Oct 2026) 4.9% 4.9; all 222 satellite rows match ONS
Bank Rate (IUDBEDR) 3.75 for all of 2026Q2 3.75

The energy shock is visible in the raw data: CPI motor fuels (D7EC) rose 17.8 log points in Q2, household energy (D7CH) fell 5.3, and Brent went from $77.8 to $97.0.

Note: Bank Rate rose to 4.00 on 5 Nov 2026. That is after the data edge. The model's median for 2027Q3 is 3.92.

3. Replication 744 → 765 is expected. The 1992Q1–2025Q1 window is unchanged, but the data inside it were revised:

  • uk_gdp moved by up to 4.42 log points in level and 0.18 in growth (Blue Book 2026).
  • world_gdp moved by up to 0.14 in growth (FRED).
  • cpisa and oil_price moved by about 0.003, because the seasonal factors are estimated on the full sample.

With the same seed, the posterior differs and so does the set of accepted draws. The figures the macro site quotes that move:

Figure Old New
Accepted draws 744 765
Acceptance rate 7.4% 7.6%
ESS 356.1 368.0
FEVD global share, GDP (per-draw mean) 37.4% 38.7%
FEVD global share, CPI (per-draw mean) 42.3% 41.5%
68% band, GDP [23.4, 52.2] [24.0, 53.1]
68% band, CPI [25.5, 60.3] [25.6, 56.3]
Rolling origins 49 50
Minimum q vs the drift benchmark 0.36 0.39

The qualitative readings hold: a match on GDP, roughly 8.5pp short on CPI, and no skill against drift.

Lines in macro main that quote these numbers:

  • papers/boe-svar/sections/appendix.tex:80, implementation.tex:19, comparison.tex:9,20-21,34,37,39, validation.tex:14,18,40-41,48,143
  • main.tex:31,35, results.tex:6,19, limitations.tex:6, source.tex:6
  • figures/make_comparison.py:107-110, papers/boe-svar/index.html:87
  • svar/validation/index.html:93,98,169-170,176,186,193,308,542,601, svar/index.html:83
  • models/index.html:262,328, index.html:278
  • integration/src/policyengine_macro/capabilities.py:267-268

Lines that only go stale once Modal is redeployed (they say "data through 2026Q1"):

  • index.html:69, svar/index.html:146, svar/methodology/index.html:251
  • economy/topics/growth/index.html:135, economy/topics/jobs/index.html:129, models/index.html:207
  • core.py:1248,4132, capabilities.py:89-90
  • economy/topics/inflation/index.html:131: its 2026Q2 forecast row is now an outturn of 2.8%.

Recommendation: the paper attributes these numbers to "the repository's production artifact". Either pin the paper to the pre-#24 sha or tag, or update every line above in one macro PR when this merges.

Pre-existing, not caused by #24: implementation.tex:19 quotes the forecast revision as 487 accepted / ESS 207.5. The repo had 430 / 197.9 before #24 and has 455 / 181.4 after.

4. The estimation sample still ends in 2025Q1, as intended. The --est-end 2025Q1 defaults and SPEC.md set this. Forecasts still condition on data through 2026Q2: run_external_comparison.py:119 sets y_full to the full frame, and the origin is df.index[-1].

One judgement call: the period held back from estimation is now five quarters (2025Q2–2026Q2), while docs/methodology.md says "last year reserved". Either reword that line, or roll the end of the estimation sample to 2025Q2 (which moves the replication again).

Outturn check. CPI for 2026Q3 is already published at 3.1% (D7G7, 21 Oct). That is inside the new one-quarter-ahead 68% band of 2.51–3.43 (median 3.00). Bank Rate for 2026Q3 was 3.75, against a median of 3.85.

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown
Contributor Author

Merging this PR leaves the PolicyEngine Macro site's BoE SVAR replication figures stale: 744 accepted draws, ESS 356.1, FEVD 37.4 / 42.3, 49 origins, smallest q 0.36. I've prepared both options as draft PRs:

One item from (b) belongs in this PR: docs/methodology.md still says the last year is reserved. The held-out period is now five quarters (2025Q2–2026Q2).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant