Skip to content

UK source packages, wave 3: back the uk_local_geography profile (constituency + local authority) #134

Description

@juaristi22

Lanes: packages under ledger-source-ingestor; any profile extension under ledger-target-profile-author
Depends on: conventions from #132 / #133 (GSS geography IDs, curated-extract lane, Stat-Xplore api_response pattern)

Context

policyengine_ledger/target_profiles/uk_local_geography.json declares 25 targets across five families (8 ons_population age bands, 5 dwp_universal_credit, 5 ons_housing, 4 hmrc income, 3 ons_income) at constituency + local_authority levels — with zero backing source packages. Of Ledger's 70 committed packages, all 4 UK ones are national firm/VAT. Meanwhile policyengine-uk-data feeds its local calibration from per-area CSVs under datasets/local_areas/{constituencies,local_authorities}/targets/ (built by targets/sources/local_age.py, local_income.py, local_uc.py, local_la_extras.py — these emit no registry Target()s; they are loss-matrix inputs).

Sub-national precedent exists: irs_soi/historic_table_2_state_*, census/pep_2024_state_age_sex, census/acs_s0101_state_2024 (state level), and per-CD Census API extractions.

Packages

backing package(s) source mechanics
ons_population age bands ons/census21_age_by_pcon24, ons/census21_age_by_lad Census 2021 / mid-year estimates by PCON24 and LAD; uk-data's local_age.py reads two per-area CSVs
hmrc employment / self-employment income hmrc/spi_by_pcon24, hmrc/spi_by_lad SPI geography tables (collated 3.12–3.15a) — amounts and taxpayer counts
earnings (profile hmrc count/amount complements) nomis/ashe_earnings_by_pcon24 NOMIS ASHE workbook uk-data commits (nomis_earning_jobs_data.xlsx)
dwp_universal_credit households (+ by children) dwp/uc_households_by_pcon24, dwp/uc_households_by_lad Stat-Xplore extracts — archive the API response as the artifact per the existing api_response pattern
ons_income equivalised income BHC/AHC + housing costs ons/small_area_income_msoa ONS small-area income estimates (MSOA), plus the aggregation lookup the profile's levels need — only source-published aggregation levels enter Ledger; any MSOA→PCON re-aggregation PolicyEngine performs is downstream build work, not a fact
ons_housing tenure + private rent ons/tenure_by_lad, ons/private_rents_by_lad uk-data local_la_extras.py sources (PRMS rents, tenure)
LA council tax stock voa/council_tax_stock_by_lad wave-2 VOA publication, LA cut

Geography contract: GSS codes as geography_id (E14… PCON24, E06/E07/E08/E09… + devolved LAD codes), explicit geography_vintage (pcon_2024, lad_2023-style), geography_level: constituency / local_authority matching the profile's levels.

Row-count mechanics (verify on the first package)

650 PCON24 × band and 360 LAD × band tables must not become hand-enumerated per-row YAML (the VAT worked example enumerates 9 rows manually). Multiple committed extraction methods parse full tables ("delimited text full-row parse with selected source-row cells", "zip archive CSV member full-row parse", Census API unpivot) — confirm the row-spec form that pins geometry for full-table parses on the first constituency package, and document it in docs/agent-source-package-harness.md if the answer isn't already written down.

Profile work

uk_local_geography already declares selectors and bindings — the deliverable is that its selectors resolve against these new facts. Any selector/binding corrections discovered while backing it are a separate small PR in the profile lane (no values, sum-only).

Acceptance

  • Every declared uk_local_geography target resolves from a consumer artifact at both declared geography levels, every fact carrying full lineage (manifest sha256 + R2 + cell guards).
  • Coverage parity with uk-data's local CSV inputs: each per-area family uk-data feeds its constituency/LA loss matrices either has a Ledger package or an enumerated, signed exclusion in this issue — no silent drops.
  • Deterministic checks + judge lanes green.

Out of scope

  • Populace's local solve (already Ledger-profile-aware via populace.build.uk.local_targets, populace#158). uk-data's local calibration remains as is until populace consumption switches over.

Sizing

~8–9 packages ≈ 3 PRs (ONS batch, HMRC/NOMIS batch, DWP/VOA batch) + possibly 1 profile-correction PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions