UK wave 1: ONS population source packages — mid-year estimates 2023+2024 and 2024-based projections (#132)

[juaristi22]

Part of #132 — deliberately not Closes: this PR delivers one of wave 1's three families. The issue's completion map:

• Region×age (108 fixture rows) — delivered here, plus the mid-2024 estimates and the 2024-based national projections beyond the original scope
• SLC student support (6 rows) — gated on Add academic_year period type to the source-package contract #131 / PR Add academic_year period type to the source-package contract #136 (academic_year); successor PR
• ISC private-school census (1 row) — successor PR (true publisher relabel from uk-data's "obr" tag)

#132 closes when the two remaining families land.

Lane: ledger-source-ingestor (.github/ledger-agents.yml) — judges ledger-source-fidelity + ledger-boundary. Paths: packages/** plus the db/data/** writes the fetch-artifact harness flow prescribes (note for maintainers: the ingestor lane's allowed_paths lists neither db/data/** — where the canonical fetch-artifact flow writes — nor ledger/source_package.py, which every package PR must touch for the SOURCE_PACKAGE_ALIASES drift guard; both are candidates for the lane canon).

What

Four source packages, 3,676 facts total, at publisher granularity (single year of age; the fixture's 10-year bands are a populace-side mapping concern):

package	artifact	rows	facts
ons-mye-2023-uk-countries	mye23tablesuk.xlsx (MYE2 persons)	UK, Wales, Scotland, Northern Ireland	368
ons-mye-2023-england-regions	mye23tablesew.xlsx (MYE2 persons, 2023 LA-boundaries edition)	publisher-emitted E12000001–9 region rows	828
ons-mye-2024-uk	mye24tablesuk.xlsx (MYE2 persons; this edition carries the E12 rows directly) — sha256 f78d50e8d77fe5ed9d04667d6875aefcf62b56380af7aceb2e90dce067981526	all 13 geographies in one artifact	1,196
ons-uk-population-projections-2024	uk.zip → uk_ppp_machine_readable.xlsx (2024-based NPP principal) — sha256 dea79703b9caddd560f1958b5a8ca12aa817b2396d244e6a2c775b1af7f48ab1	sex × single age, per-year record sets 2024–2029, source_projection	1,284

Cross-artifact coherence check (from adversarial review): the NPP 2024 sex×age total equals the mye24 UK all-ages estimate to the person (69,281,437).

Shape: one record set per single year of age (groupby_dimension: age; the all-ages sets are age-totals), rows carry per-row geography overrides, and age enters fact identity via row filters mirrored verbatim by explicit constraints (the consumer-contract comparator matches unit/label exactly). The YAMLs are machine-generated and uniform (10.9k + 21.4k lines); if a compact templated form is preferred, that belongs with the contract follow-up below.

Sources (for review)

• UK countries workbook: dataset page, mid-2023 edition, mye23tablesuk.xlsx — sha256 7ee5724ecf71abc3d811c89174cdddf1bce7f07d3eefc8b69b15ee8371414b16
• England & Wales workbook: dataset page, mid-2023 (2023 LA boundaries) edition, mye23tablesew.xlsx — sha256 96dcbb73d01c2965a0e8173bdf736f04a16d91dc189c4a52220919792e5e2c94

Both artifacts were fetched twice by independent code paths (manual download, then uv run ledger fetch-artifact) with byte-identical checksums; provenance_class: census (census-anchored official estimates, mirroring the census_pep precedent).

Source decision this PR asks reviewers to ratify

Issue #132 originally framed this family as projections (SNPP/NPP), inheriting policyengine-uk-data's claimed lineage for its committed demographics.csv. Kickoff findings changed that:

1. The CSV's provenance is unrecoverable: last updated 2025-06-10, which predates the England 2022-based SNPP publication (2025-06-24); no vintage is recorded anywhere. No projection package can be shown to reproduce the production fixture either.
2. The production surface calibrates period 2023 only, and official actuals for mid-2023 exist — the ONS mid-year estimates used here (observation-class, census-anchored).
3. Deltas vs the production fixture, after resolving that uk-data's _age_80_89 band is actually 80+ (a systematic −18% otherwise): median 0.76%, p90 2.14%, max 2.94% — the expected estimates-vs-projections gap, enumerated per-target in the populace consumption gate (Consume Ledger UK facts: uk_target_references, mapping module, and a compile-parity gate microcosm#622). Top deltas: south_east_age_10_19 +2.94%, east_age_10_19 +2.86%, west_midlands_age_10_19 +2.46%.

Projection packages (2022-based NPP uk.zip, England 2022-based SNPP) remain queued for future-year calibrations, where actuals cannot exist — porting both sources is consistent with the exhaustive-vintage policy; populace picks the active source per calibration year.

Gates

• validate-package: clean for both packages, zero warnings.
• build-suite --year 2023: green end to end — source cells, source records, facts, DB build, agent_acceptance.json valid with zero errors (only the standard non-fatal no_concept_alignments warning; Axiom alignment is a later step).
• Facts spot-verified against the workbooks by an independent openpyxl read (UK all-ages 68,265,209; Wales age-0 27,701; Scotland 90+ 45,547; London age-35 151,827; North East all-ages 2,711,380 — all exact).
• Lane test files (test_source_files.py, test_ledger_source_cells.py, test_ledger_facts_only.py) pass.

Draft because: the consumer-contract export requires storage.r2 (raw_r2_uri) in the manifests. The committed manifests honestly record checksums and retrieval metadata only; the suites were proven green under a locally injected, disclosed, and reverted R2 block. After source review, a credentialed maintainer runs uv run ledger publish-raw --root db/data and the last gate closes with no package changes.

Contract findings for maintainers

1. Digit-text column headers are unguardable: _render_value int-coerces digit-string expectations while _resolve_guard_cell compares raw — MYE2's text-typed 0..89 headers can't be expressed as expected_column_header. The age span is pinned by the guarded non-digit endpoints (All ages, 90+) plus the sha-pinned artifact. Follow-up contract-lane issue to be filed.
2. ledger-agents.yml's ingestor lane omits db/data/** although the canonical fetch-artifact flow writes there.

🤖 Generated with Claude Code

[juaristi22]

Source-review outcomes and CI triage (commit ab98578).

Source decisions (reviewed):

• Wave-1 region×age ships the mid-year estimates (MYE). The _age_80_89 fixture band is proven to be 80+: the nine bands in policyengine-uk-data's demographics.csv sum to the all-ages totals at ratios 0.999–1.003, so the bands are exhaustive.
• National projections are now in scope and in this PR: the demographics.csv lineage problem only ever concerned the regional family. ons-uk-population-projections-2024 packages the 2024-based NPP principal (uk_ppp from the z1 uk.zip, sha dea79703…48ab1) — one record set per year 2024–2029, rows = sex × single year of age, 1,284 source_projection facts, spot-verified byte-exact. Later projection years land via the vintage backfill lane; regional projections stay deferred until a calibration year needs them.

CI triage: the red check was not the R2/storage gate — it was the SOURCE_PACKAGE_ALIASES drift guard (unregistered packages/* fail loudly rather than being silently dropped from bundles, per #78). All four packages are now registered, and the year-2023 bundle expectations are updated to the post-publication truth measured by a full local bundle build: fact_count 39,219 → 42,895, source_package_count 60 → 64, period_count 16 → 19, ons by-source 1,246 → 4,922, source tables 55 → 59, geography_count 1,053 → 1,065.

Remaining red, by design: the bundle test's consumer export requires raw_r2_uri, and the committed manifests honestly carry no storage.r2 until the artifacts are uploaded. One maintainer command closes it with no further code changes:

uv run ledger publish-raw --root db/data

(Counts were verified under locally injected and reverted storage.r2 blocks, disclosed as in the PR description; committed manifests remain upload-truthful. Four artifacts to publish: mye23tablesuk.xlsx, mye23tablesew.xlsx, mye24tablesuk.xlsx, uk.zip.)

🤖 Generated with Claude Code

[juaristi22]

Re the failing test_build_bundle_writes_merged_consumer_contract: that's the one designed red this PR carries (PR body, "Draft because…", and the CI-triage comment above) — not a regression, and nothing to fix in code.

Causal chain: the bundle now includes this PR's four packages (the alias registration doing its job), their consumer-contract export requires raw_r2_uri, and the committed manifests deliberately carry no storage.r2 blocks because the artifacts are not actually uploaded yet — manifests stay upload-truthful. The test's expectations are already set to the post-publication numbers (42,895 facts / 63 packages), measured under a disclosed-and-reverted local sandbox, so the test converges with zero further commits once this runs on the branch:

uv run ledger publish-raw --root db/data

(4 artifacts here; #139 has the same designed red with 9.)

Current blocker on running it: María's Cloudflare membership role doesn't include R2 — user API tokens are intersected with membership roles, so her Workers R2 Storage:Edit token still 403s (code 10042), and current wrangler's OAuth flow carries no R2 scope at all, so the wrangler login route can't reach remote R2 either. Fastest unblock is a role bump for María (her existing token then works for this and every future wave); alternatives: an account-owned R2 token, or running the two publish-raw passes yourself.

🤖 Generated with Claude Code

[juaristi22]

Adversarial review complete — all findings addressed in 4b456ec; the packages themselves came through clean.

Fidelity core: all 3,676 pins independently re-derived from the committed artifact bytes with zero mismatches — row headers, guard cells, column letters, header strings, cell types, filters↔constraints shape. Best cross-check in the review: the NPP 2024 sex×age total equals the mye24 UK all-ages estimate to the person (69,281,437). All shas, manifests (no premature R2 blocks), aliases, and boundary checks clean.

Fixed from findings:

• Major: my earlier constants pass updated the headline bundle expectations but missed four deeper assertion blocks (by_period, by_geography length, country:K02000001, by_entity) — the test was self-contradictory and could never pass even after publish-raw. All four now carry the post-publication truth, proven by running the bundle test itself under the disclosed-and-reverted R2 sandbox: 1 passed.
• NPP's twelve 105 - 109 rows normalized to {female,male}_105_109 tokens (guards keep the publisher's raw label); geography-name cosmetics (Yorkshire and the Humber; East of England) — guards pin the raw workbook strings and are untouched.
• PR title/body now cover all four packages with all four artifact shas; the maintainer note names both out-of-lane surfaces every package PR touches (db/data/** and the SOURCE_PACKAGE_ALIASES map in ledger/source_package.py) as lane-canon candidates.

🤖 Generated with Claude Code

[vahid-ahmadi]

Read the registry diff and the bundle-constant updates; the four ONS population packages are pure data plus four SOURCE_PACKAGE_ALIASES lines, so there's no logic here to get wrong. Nothing blocking from me.

Two questions rather than findings:

1. Are the ~87k lines of source_package.yaml harness-generated or hand-authored? packages/ons/mye_2024_uk alone is 29,836 lines. If they're generated from the committed artifact bytes by validate-package, then review is genuinely "the gates passed" and that's fine — but it'd help to say so in the description, because the alternative reading (hand-transcribed) implies a very different review burden and a very different regeneration story when ONS reissues a table.

2. NPP 2024 is a projection, MYE 2023/2024 are estimates. They'll sit side by side in the bundle under calendar_year periods with no type-level distinction. Is the estimate-vs-projection boundary carried by provenance_class, or is it left to the consumer profile to know that ons/npp_2024_uk facts past 2024 are modelled? Worth being explicit somewhere, since a target profile that resolves "latest not after base period" would happily pick up a projection without noticing.

The description's task list shows 1 of 3 checked — is the remainder in-scope for this PR or tracked into #132's later waves?

[juaristi22]

Answering the review's three questions:

1. The ~87k YAML lines are machine-generated, not hand-transcribed. Row/measure blocks were emitted by generator scripts running over the parsed artifact bytes (the same parse the harness performs), then validated by validate-package/build-suite and independently re-derived pin-by-pin in the adversarial review (all 3,676 pins, zero mismatches — including the NPP total equalling the mye24 UK estimate to the person). So review is indeed "the gates passed" plus the fidelity re-derivation. Honest caveat: the generators were ad-hoc session scripts, not committed — the durable regeneration story today is fetch-artifact + the package spec + the gates. Committing package generators under scripts/ so an ONS reissue is a re-run rather than a re-authoring is a fair lane-canon candidate; happy to file that.

2. Estimate-vs-projection is carried by assertion, not provenance_class — and it's now resolution-enforced. Every NPP forward-year fact in this PR carries assertion: source_projection (the MYE facts are observation), and that axis flows into every consumer row. You were right that the resolver ignored it: under latest_not_after_build_base_period a mixed selector would happily resolve a projection. #154 closes exactly that gap — profiles now declare an assertion_policy (observed_only default: projections invisible, projection-only families fail loudly; prefer_observed; allow_source_projection for forecast families like the OBR EFO lines), an explicit assertion selector bypasses the policy as maximal intent, and any projection resolution emits a resolved_from_projection warning with the fact's assertion exposed on the resolved row. provenance_class stays orthogonal (how the publisher measured): NPP is model_output + source_projection; a balance-sheet estimate is model_output + observation.

3. The unchecked remainder is deliberately in sibling PRs, per the one-family cadence. SLC was gated on the academic_year vocabulary and unblocked when #136 merged — those packages are next in flight. The ISC private-school fact landed in #141 batch 4 (as the true-publisher isc relabel). #132 closes when the SLC family lands.

🤖 Generated with Claude Code

[juaristi22]

R2 publish is done for this branch. Sequence since the last push:

• Rebased onto main at d05b694 (picks up the merged Add academic_year period type to the source-package contract #136 academic_year support and the Pin Cloudflare account for R2 publishing and fix publish-raw failures #148/Fill missing source_id/package_id so publish-raw covers every manifest #149 publish-raw fixes) — clean replay, no conflicts.
• Ran the first credentialed chronicle publish-raw --root db/data against the account pinned in wrangler.toml. The four wave-1 ONS artifacts (mye 2023 countries + England regions, mye 2024 UK, npp 2024) uploaded to ledger-raw under content-addressed keys; the already-published bootstrapped artifacts re-put to byte-identical keys as expected.
• Committed the manifests with their real storage: blocks (7a57882) — they were deliberately kept storage-less until an actual upload happened, so the repo stays upload-truthful.
• Full test suite now passes with no designed-red remainder: 615 passed, 1 skipped, including the bundle consumer-contract test that requires a live raw_r2_uri per fact.
• Spot-verified end-to-end: downloaded mye24tablesuk.xlsx back from R2 and its sha256 matches the manifest (f78d50e8…).

Same publish step will follow on #139/#141/#155 once their bases settle.

[vahid-ahmadi]

Verified this against the artifacts rather than just reading the YAML: re-hashed both committed workbooks and uk.zip against the manifests (match), then opened them with openpyxl and checked the pinned cells.

Everything I checked is exact:

• MYE column span — header row 8 is All ages at D, 0 at E … 89 at CP, 90+ at CQ, and CR is empty. The 92 record sets map 1:1 onto D..CQ with the guarded non-digit endpoints where you said they are.
• MYE geography rows — per-row overrides are right, including E12000001–9 correctly carrying geography_level: region inside a record set whose default is country.
• NPP — Population sheet, Females rows 2–108 / Males 109–215, C..H = 2024..2029, B215 = '110 and over' and B107/B214 = '105 - 109' guarded on both sexes. Ages are 0–104 contiguous plus 105_109 and 110_plus, no gaps.
• NPP year truncation to 2029 is documented in the header note with the backfill lane called out, which is the right call.

The digit-text header finding and the db/data/** lane-path gap are both real; agree they're contract-lane follow-ups, not blockers here.

England (E92000001) is missing from all three packages

mye23tablesuk.xlsx row 12 and mye24tablesuk.xlsx row 12 both publish E92000001 / ENGLAND / Country directly — 57,690,323 for mid-2023 and 58,620,101 for mid-2024. Neither MYE package ports it, and mye_2023_england_regions only carries the nine E12 rows, so England-level population isn't in the bundle at any level.

Skipping K03000001 (Great Britain) and K04000001 (England and Wales) is obviously right — they're roll-ups. But England is a constituent country and it's the one nation a UK build is most likely to ask for, and it sits in the same row block you're already reading. It also makes the mye_2023_uk_countries label — "UK and constituent countries" — inaccurate, since it's UK + three of four.

Is the omission deliberate (recoverable by summing the nine regions, so not worth the rows) or an oversight? If deliberate I'd still add it: Chronicle's job is publisher truth, and a consumer summing nine regions to get England is doing exactly the arithmetic this repo exists to avoid. Four extra rows across the two UK-countries packages.

provenance_class: census for the mid-year estimates

You justify it as census-anchored, mirroring census_pep, which is a fair precedent. But #154's own doc draws the line as "a national-balance-sheet estimate is model_output + observation" — and a mid-year estimate is the same shape: a cohort-component roll-forward of the last census with births, deaths and migration, asserted by ONS as an estimate. Under the new two-axes framing that reads as model_output + observation, with census reserved for enumerated counts.

Not blocking, and I'd rather not churn 92 record sets over it — but now that #154 makes the axes explicit it's worth settling the convention once, because it's about to be copied into every UK package in wave 2.

Coverage asymmetry worth knowing before the profile work

MYE is persons-only (MYE2 - Persons); NPP is sex × age. So observed UK population by sex isn't available at any period, while projected population by sex is. If the Microcosm UK dataset wants sex-split observed targets, the MYE2 - Females / MYE2 - Males sheets are sitting in the same workbooks — worth a note in #132 either way so it isn't discovered downstream.

Merge order with #139

Both PRs rewrite the same absolute constants in test_chronicle_bundle.py (fact_count 39219 → 42895 here, → 40131 there, plus geography_count / period_count / by_source). They'll conflict textually, and the second one through needs a genuine re-measure, not a hand-merge of the two numbers. Both are MERGEABLE / CLEAN right now. Let's merge this one first since #139's OBR lines are the ones that actually need #154's policy, then you re-run the bundle for #139.

---

Nice work — the cross-artifact coherence check (NPP 2024 sex×age total equalling the mid-2024 UK all-ages estimate to the person) is the kind of thing that makes this reviewable. England is the one I'd like resolved before merge; the rest can follow.

[vahid-ahmadi]

Four ONS population packages with a four-line registry change and no parser work — the harness earning its keep. The MYE/NPP split (2023 countries + England regions, 2024 UK, 2024-based projections) is a sensible slicing.

1. This will collide with the other three open UK waves on the golden-count test.

#137, #139, #141 and #155 all branch from the same base and all rewrite the absolute totals in tests/test_chronicle_bundle.py to different values:

PR	fact_count	source_package_count	period_count
#137	42895	64	19
#139	40131	69	61
#155	39418	66	33

Each is right on its own branch and stale the moment a sibling merges. So this is four rebases with hand-recomputed constants, and the tempting way to resolve the conflict — take one side's number — produces a green test that has stopped checking anything.

No change needed here, but it's worth filing: per-package or per-source delta assertions would compose across parallel work, where one whole-bundle snapshot can't. Given four UK waves are in flight simultaneously this is costing real time right now.

2. The projection years are exactly the case #154 is about.

This PR adds calendar_year:2027/2028/2029 at 214 facts each, from ons/npp_2024_uk — national population projections. #154 (open) introduces the assertion: source_projection axis and makes observed_only the default resolution policy, calling out "NPP projection years" by name as the motivating example.

If #154 lands first, these facts should carry assertion: source_projection or every consumer profile will silently treat 2024-based projections as observations. If this lands first, they'll need a follow-up pass. Either way the two PRs should probably be sequenced deliberately rather than merged in whatever order review finishes — the NPP package is the one that most needs the new field.

Worth checking whether the 2026 jump (21 → 235) is also projection-side.

3. Minor

• person doubles (3688 → 7364) and by_source["ons"] goes 1246 → 4922 — a ~3700-fact increase from population tables looks right for MYE age×sex×geography detail, but since the test is the only thing pinning it, a sentence in the PR description about the expected magnitude would help the next reviewer.
• geography_count moves 1053 → 1065, i.e. 12 new geographies for UK countries + English regions. That's 4 countries + 9 regions = 13, so one is presumably already present (K02000001?). Cheap to confirm.