One consistent UK target surface: enforced cross-grain reconciliation, declared uprating holds, nation household controls and the 942a1fa re-pin (#1123 PR A)

[juaristi22]

Part of #1123 (PR A: items 1, 2, 6 and 7; private rent is PR B). Addresses #104.

Why

The K=25 audit found that most local misses surviving the dense solve were on the target side. Local rows were never uprated, 509 national rows were held at an older period with no declared index, nothing refused an undeclared hold, and reconciliation across grains was opt-in. This PR makes the UK target surface one consistent, enforced surface.

What changes

• Cross-grain reconciliation is enforced. Every family bound at more than one grain declares how its grains agree (uk/cross_grain_declarations.json: bridges, partitions, relations) or the compile refuses. The lattice gains a nation grain, rows that span several legs, and same-geography partitions. The national register is reconciled across UK, nation and region in both loaders, and the joint local pass refuses a national factor away from one.
• Uprating doctrine. A target bound from a fact older than the calibration year carries a declared index or a declared hold (uk/uprating_holds.json, 15 entries, approved_by null). The compile refuses an undeclared, expired or stale hold, nationally and locally.
• Chronicle re-pin to 942a1fa (UK local-surface facts for the 2025 calibration: household counts by nation and area, communal-establishment residents by age, mid-2025 population estimates, UC households by area for calendar 2025, and BRMA geography and rent inputs (PolicyEngine/microcosm#1123, PolicyEngine/microcosm#1090) chronicle#313 via Add UK 2025 calibration publisher inputs chronicle#314). National population, local-authority ages, land, SLC repayments, ISC, DfI and public sector employment move to their 2025 editions. Spine inputs move only by publisher revisions (revised mid-2024 population, DfI rounding).
• Households: nations win. Each nation's official estimate controls its census cells: LFS 2025 for England's nine regions, NRS for Scotland, Welsh Government rolled by population for Wales, Census 2021 rolled by LPS dwelling stock for Northern Ireland. The UK composition takes the nations' sum (29,082,020; the LFS UK total is a diagnostic, 0.27% lower).
• HMRC. Income targets are on a taxpayer basis at both grains; area cells take HMRC's all-ranges growth and reconcile to the national SPI bands within 2%. Higher and additional rate taxpayers bind for each English region and nation (Income Tax liabilities Table 2.2, calendar 2025, on the engine's tax_band), and higher rate taxpayers bind by constituency and local authority in England, Wales and Northern Ireland (919 cells).
• UC by area binds the calendar-2025 mean of DWP's monthly series, and the child bands follow each constituency's caseload.
• Tenure. England's census shares drift by ONS SPREE's 2022 to 2024 share change; tenure cells follow their authority's households.
• Small cells. Council-tax band cells under 50 dwellings defer by rule with an expiry (16 cells; uk/local_small_cell_rule.json, unsigned). A deferred cell still counts in its leg's control.
• Make zero-valued calibration targets intentionally nonzero-weighted #104. The loss-scale options carry a census of zero-valued targets. The weighting rule is unchanged (UK sparse selection: keep household mass, composition, nation shares and per-area ESS at a tractable size — test the target-weight rule and the L2 penalty first, dataset size only if they fall short #1124 owns it).
• Tooling. A rolled_forward_by_ratio value operation; both UK target kernels hash every module that decides a target value; tools/uk_target_surface_receipt.py writes and diffs receipts of the compiled surface; the national compile scans only each reference's own sources (the national parity receipts drop from about 2h25m to under 20 minutes on the 927k-fact feed, with identical candidate facts).

Value moves (receipt against the pre-change surface)

• National: 1,267 rows. 37 added (12 nation household cells, 24 marginal-rate cells, dwp.hb.amount_pension_age), 168 moved (149 population rows, the 10 household-composition rows, land, SLC repayments, ISC, public sector employment, DfI), 2 removed (top UC payment-distribution bands the 2025 series does not publish).
• Local: 21,788 rows. 919 added, 16 deferred, 18,374 moved (every age, census-household, tenure and UC cell; the four existing HMRC area targets; private rent through tenure).
• Reconciliation factors: HMRC area cells to the national bands 0.996 to 1.019 (gate 2%; employment counts at 1.92%); nation household controls over constituency census cells 0.973 (North East) to 1.038 (London), Scotland 0.989, Wales 0.994, Northern Ireland 0.998; higher rate area cells to their 2025 rows 1.19 to 1.40; UC households by area to the GB caseload 0.921 (0.942 before this PR).

Verification

• tools/uk_target_surface_receipt.py --communal runs clean end to end on the pinned feed.
• Engine-free suites: microcosm-build UK and shared 7,105 of 7,106, microcosm-calibrate 361 of 361, microcosm-graph 727 of 729; tools/ci_test_plan.py verify ok. The failures are local-environment only and also fail at the base commit (pe-uk installed in the engine-free venv; a node package the Orrery parser tests need).
• The Make zero-valued calibration targets intentionally nonzero-weighted #104 edit to microcosm.calibrate.solve moves the loader golden vector, the US seed digests, the US coverage evidence and the calibrate graph parity pins; all are re-pinned as main does for solver edits.
• Not run: engine tests (the tax_band measure is tested against fakes only here), and the dense re-solve, so the effect on fit is not yet measured.

Departures from the plan

• No closure gate in gates.json: closure fails closed at compile instead.
• No council-tax band partition, and the preflight tool is not updated.
• The A15 census-household uprating step is kept; it now targets the nations' sum and the nation bridge then rescales.

Open decisions (María)

Before merge:

• Signatures on the 15 holds and the small-cell floor of 50.
• England households: LFS regional controls (as built) or the England total with projection shares.
• Scope: Scottish higher rate area cells and all additional rate area cells stay unbound. Scotland's 2023-24 area cells predate its advanced rate; additional rate area cells are rounded to the thousand and suppressed in 85 constituencies.
• Accept the departures above, or build the gate and remove A15.

Can follow as issues:

• Clip self-employment losses in the amount rows to match HMRC.
• Salary-sacrifice relief growth.
• The Northern Ireland household method.

After merge

• Re-freeze the scoring register in uk-candidate-eval (the reconciled national values change the registry version).
• PR B (rent) rebases on this branch.

🤖 Generated with Claude Code

[juaristi22]

Independent adversarial review (Claude Code) at 67fc1dd04

Static review only. I read the code and the committed registers at this head and at the merge base aca40a692, and I checked the UC source package in Chronicle at 942a1fa. I ran no tests, no compile and no receipt tool, so every factor below is control ÷ the sum of the committed cells before reconciliation, the same ratio the code takes. Engine CI was still queued when I looked.

Verdict. The core machinery holds up. Cross-grain coverage is complete for the committed contract: 15 multi-grain ids, 28 overlap pairs, none undeclared. The holds cover every lagged national row, and the #104 edit to solve.py changes no numbers. But four things should change before this leaves draft:

• the UC bridge reconciles two different concepts;
• the 2% fan-out gate never sees the local-authority grain, which needs 2.1%;
• the new national band bridges have no tolerance;
• the signatures the body asks for aren't enforced by any code.

Must fix

1. UC by area reconciles all-claims cells onto a paid-claims control.

   • National control: dwp.uc.households binds payment_indicator: "Yes" (uk_population_targets.json:14716).
   • Area cells: the selector (:28243) reads universal_credit_households with no payment dimension. The Chronicle package at 942a1fa (dwp/uc_households_by_constituency_january_2025_may_2026) has only the date and PCON24 fields, so it counts all UC households, nil-award claims included. docs/uk-uc-paid-target-contract.md is why the national rows moved to paid claims in the first place.
   • Size: the areas sum to about 6.73m against 6.20m for GB, hence the 0.921. Aligning the windows widened the gap (0.942 → 0.921), which points at concept, not timing.
   • The selector note says the bridge "reconciles like with like"; it doesn't.
   • Effects:
     • The uniform factor assumes the same nil-award share in every constituency.
     • The child bands keep their all-claims shares, so after the bridge they disagree with the national paid rows: +2.8% with no children, −1.5% with one, −3.5% with two and −5.1% with three or more. (The national no-children figure is the total less the child rows.) The solve gets conflicting targets across grains.
     • The coverage check can't see this, because the signatures differ.
   • Fix: extract the area series with Payment Indicator = Yes (UC_Households carries the field).

2. The 2% fan-out gate is never applied at local-authority grain, which needs more than 2%.

   • Factors: at constituency grain the committed values reproduce the body exactly: 1.0096, 1.0192, 1.0082 and 0.9958. At local-authority grain they are 1.0118, 1.0211 (employment counts), 1.0160 and 1.0023. Torbay and Scilly have no SPI LA cell.
   • Why nothing refuses 1.0211:
     • hmrc.employment_income.count at constituency and LA forms an exact-signature group with no control-grain row, so constituency parents LA under the single-winner rule.
     • The sort key at cross_grain.py:256-266, (precedence[lower], -precedence[winning], id), runs constituency_over_la before the bridge's country_over_la.
     • So by the time the bridge pair runs, LA already equals the reconciled constituency total. Its factor is about 1, and _assert_uk_fanout_sum_factors (uk_runtime/ledger_targets.py:2579) only reads bridge groups.
   • The declaration says "the constituency and local-authority cells reconcile to it per grain… a factor beyond 2% is refused". test_uk_fanout_sum_bridges.py only builds a constituency receipt.
   • Fix: either check the bridge against the pre-reconciliation LA total, or gate the signature group too.
   • A related point: the gate's rationale is "the tables agree within 0.5% at source", yet employment counts land at 1.92%. The likely cause is that the area cells take all-ranges growth while the bands take band-specific growth. If so, the gate mostly measures that index choice, and with 0.08 points of headroom the next ITL re-pin may trip it.

3. The 24 national band bridges move cells by up to 5.7% with no bound.

   • No bound: _apply_band_bridges (national_reconciliation.py:~229) takes factor = control / raw_total and refuses only a non-positive total.
   • Wrong reason: all 24 declarations give the same reason, "about 0.5 percent short".
   • Factors:
     • income tax £12,570–15,000: 1.0573
     • taxpayers £12,570–15,000: 1.0274
     • total income £12,570–15,000: 1.0228
     • income tax £15,000–20,000: 1.0142
     • the rest: 1.002–1.008
   • Knock-on: the £12,570 income-tax bridge includes the West Midlands cell deferred in target_fit_reviewed_exclusions.json:5. That deferral was measured against a 55.1m target, which becomes about 58.3m. The body's "Reconciliation factors" section leaves the band bridges out.
   • Fix: add a max_factor_shift as the fan-out bridges have, state the real gaps per bridge, and re-measure the deferral (UK landlords: bind the SPI net property amounts, leave property_wealth to the engine, add receipts and finance costs #1145 holds the same deferral as stale).

4. Nothing enforces the signatures the body asks for.

   • Unsigned: all 15 holds and the small-cell rule are approved_by: null, status: "proposed".
   • What the loader checks: load_uk_uprating_holds (uprating_holds.py:62-84) checks only kind and reason. expires_on is optional (:169); every entry has one today, so that part is latent.
   • Contrast: the measure exclusions refuse an entry without approved_by, approved_on or expires_on (measure_simulation.py:719).
   • Small cells: the 16 cells are labelled signed_deferred in the membership while the rule is unsigned.
   • Fix: make proposed or a null approver fail closed, so the open decision can't be merged around.

Should fix

5. "Value moves" doesn't match the committed registers, and no receipt is committed.

   • Active national rows go from 1,231 to 1,267: 36 added, not 37, and none removed.
     • dwp.hb.amount_pension_age is already active at the base, from UK: track the defects uk-data is fixing that microcosm still has #1095. The comment at test_country_spec.py:1163 says 12 + 24.
     • The two UC bands listed as removed are the ones UK: track the defects uk-data is fixing that microcosm still has #1095 already excluded, so they aren't new removals.
   • Moved: 160 rows move at register level, against the body's 168.
   • Missing from the count: the moves this PR's own reconciliation makes (the band bridges on the regional SPI rows and the CGT regional rows). So the "before" receipt seems to postdate 7a7ddbdf0.
   • Can't be re-run: tools/uk_target_surface_receipt.py needs local .codex-work inputs.
   • Please commit the before receipt taken at aca40a692 and the after receipt, or at least their diff, with the band-bridge and both fan-out grains' legs.

6. The doctrine has escape paths.

   • Relations: declared relations (31) only silence the static overlap check (cross_grain_declarations.py:373-395). Nothing tests "subset" (parts ≤ whole) numerically, so an infeasible pair would compile.
   • Coverage path: the coverage check runs only on the joint path (uk_runtime/ledger_targets.py:2464), not in the national-only loader.
   • is_uprating_hold: it returns False for any row carrying uprating_factor (uprating_holds.py:110). tenure_drift.py:119 stamps factor 1 on every non-English tenure cell, so the Welsh, Scottish and NI tenure cells (a few hundred) are held at census shares without a declaration.
   • Missing periods: a missing or unparseable period also returns "not held" instead of refusing.

7. The 919 higher-rate area cells.

   • One factor per region: each region's factor (1.19 London to 1.40 NI LAs) keeps 2023-24 shares within the region. Across regions the growth is steeper where the base share is lower: about 1.37 for NI at roughly a 5% share, against 1.19 for London at about 12%. Within-region spreads are wider than that, so a uniform factor within a region probably overstates affluent areas and understates poor ones.
   • Rounding: every one of the 919 values is a multiple of 1,000, 192 are at or below 5,000, and the minimum is 2,000. That's the rounding the PR cites for leaving additional-rate cells unbound. NI constituencies sum to 89k against 87k for NI LAs.
   • Adjudications not amended: local_binding_adjudications.json full_frs_tei_band_unavailable still says hmrc_income_by_area binds only the employment and self-employment amount and count cells. population_universe_private_households still says mid-2024.

Minor and questions

8. Land: the production-2023 parity receipt gains ledger_absent for the three ons.land.* rows, which resolved at 2023 on the base.
9. rolled_forward_by_ratio:
   • The base sums every eligible fact at the latest period with no count or duplicate check (ledger_targets.py:~2359).
   • The operands aren't required to be the same series apart from the period; only the (0.5, 2] bound guards them.
   • The current NI and Wales values are right.
10. SPREE drift: as far as I can tell it only moves social renting against everything else; the private-rent-to-owned ratio barely moves in any authority. Social-stock factors such as Rushmoor's (about 0.76) look like reclassifications, and nothing bounds them. That matters for PR B, which builds on tenure/private_rent.
11. Question: Scottish tax_band. The engine applies Scottish thresholds to income including savings and dividends, while HMRC's Scottish rates apply to non-savings, non-dividend income. This affects only Scotland's national higher and additional rows. It's tested against fakes only, as the body says.
12. Question: Scotland and Wales. The feed at 942a1fa appears to carry NRS 2025 households by council area and Welsh Government mid-2024 households by LA. The census-share hold keeps 2022 and 2021 shares instead. Is that deliberate, given rules 1 and 2 of the vintage audit's source preference?
13. Stale text.
    • docs/uk-local-cross-grain-reconciliation-802.md:115 and :165-168 (A15 and the composition partition).
    • uk_local_target_census.json:848 and :871 (the "all cells bind" counts, which now include deferred cells).
    • d10b8b96b's rationale: those UC rows were already calendar averages at the base.
    • "eleven" DfI values: 17 changed.
    • The Make zero-valued calibration targets intentionally nonzero-weighted #104 changelog says zero targets "keep a positive loss weight", but the code only reports it.

Interaction with #1145 and #1167

• Feed pins: Set UK road-fuel and energy levels in the LCFS stage on a calendar-2024 basis (#1113) #1167 pins the feed to 1ee7dfe, which is an ancestor of this PR's 942a1fa. Whichever merges second keeps 942a1fa. If Set UK road-fuel and energy levels in the LCFS stage on a calendar-2024 basis (#1113) #1167 lands first, this PR must carry its new vendored resources (desnz_road_fuel_by_vehicle.json and the Consumer Trends rows) through the re-vendoring.
• Conflicts: 20+ files with Set UK road-fuel and energy levels in the LCFS stage on a calendar-2024 basis (#1113) #1167 and 9 with UK landlords: bind the SPI net property amounts, leave property_wealth to the engine, add receipts and finance costs #1145.
• If UK landlords: bind the SPI net property amounts, leave property_wealth to the engine, add receipts and finance costs #1145 lands first: its 13 property rows go active (about 1,280 references), and the parity receipts, membership, counts and uk_spine.json need regenerating. No hold or declaration is needed: they carry an uprating_index and sit at one grain. But UK landlords: bind the SPI net property amounts, leave property_wealth to the engine, add receipts and finance costs #1145's fit inherits this PR's taxpayer filter and band bridges, so it needs re-measuring.

Checked and fine:

• The nations sum to 29,082,020, and A15 doesn't double-scale.
• The nation household factors match the body.
• Deferred small cells count in their leg's control and don't inflate neighbours.
• Spine inputs move only by publisher revisions: the revised mid-2024 population and the DfI rounding. ons_household_expenditure_facts is metadata only.
• The US and loader digest moves come from solve.py being an attested kernel, with no US value change.
• The source-indexed national compile selects the same candidates by construction.