policyengine-uk 2.102.4+ needs policyengine-core 3.32.9+, which the certified US default refuses

[MaxGhenis]

Problem

From policyengine-uk 2.102.4 (2026-09-30), policyengine-uk requires policyengine-core>=3.32.9; 2.102.3 and earlier accept >=3.30.1. microcosm's workspace resolves one policyengine-core for both country extras, so uv lock --upgrade-package policyengine-uk==2.104.2 moves core 3.32.5 → 3.32.12 for the US environment too.

The certified US default that latest.json points to, populace-us-2024-spm-20260915, declares compatible_core_packages: policyengine-core ==3.32.5. With core 3.32.12 installed, microcosm.data.loader._enforce_package_compatibility on that release manifest raises:

_CertifiedPackageCompatibilityError: Certified release 'populace-us-2024-spm-20260915' is incompatible with installed policyengine-core 3.32.12; release certification requires ==3.32.5.

That is the failure #936 fixed for policyengine-us. Until it is resolved, microcosm cannot lock any policyengine-uk release from 2.102.4 on without breaking US loading.

The certified UK default, populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z, already fails the same check on main. It pins policyengine-uk ==2.89.2 and policyengine-core ==3.27.1, while main locks 2.100.0 and 3.32.5.

Options

• A. Resolve each country's core separately with [tool.uv] conflicts between the US-side and UK-side extras. A first trial on main 3598c38 (nine pairwise sets, microcosm-build/data/frame us × uk, with policyengine-uk 2.104.2) is incomplete. These parts work:

  • uv lock forks core into 3.32.5 and 3.32.12.
  • uv sync --all-packages --locked --extra us keeps core 3.32.5, policyengine-us 2.2.1 and spm-calculator 1.0.0.
  • --extra uk gets core 3.32.12.

  These do not:

  • --package microcosm-data --extra us, --package microcosm-frame --extra policyengine and --all-packages --extra policyengine still draw core 3.32.12 with policyengine-us 2.2.1.
  • --extra policyengine --extra uk installs both engines together.

  A complete version needs three more things:

  • conflict sets that cover microcosm-frame's policyengine extra;
  • a core pin on the US-side extras;
  • a lock-level test that every US install path resolves the certified trio and that every US × UK combination is refused.

  CI already installs one extra per job.

• B. Hold policyengine-uk at 2.100.0 until a certified US release moves core.

• C. Re-certify the US default under the newer core.

What the next policyengine-uk bump touches

This was rehearsed against policyengine-uk 2.104.2 plus PolicyEngine/policyengine-uk#2021, with the engine-uk suite run under each.

• Uprating pins. tools/pin_uk_uprating_engine_values.py moves only the version stamp; the values are unchanged.
• Concept mapping. Review the mapping, set engine_version, then run tools/refresh_concept_coverage.py --engine policyengine-uk.
  • 28 inputs were added since 2.100.0: alcohol and tobacco quantities, car attributes, lifetime_isa_balance, mortgage_debt and consumer_debt.
  • None matches an existing concept. fact:household.mortgage_principal is the repayment flow, already bound to mortgage_capital_repayment.
  • brma leaves the input list once Default the BRMA from the household's region policyengine-uk#2021 is in.
• UK spine graph. tools/graph_uk_spine_fixture.py rewrites only uk_spine.json, and only the numerical_dependencies version strings change.
• Lock hash. APPROVED_UV_LOCK_SHA256 in us_runtime/worker_identity.py follows the new lock hash.

These four are the only engine-uk failures at 2.104.2. Adding PolicyEngine/policyengine-uk#2021 does not change that list.

The eFRS parity reference (optional, and deliberately fenced)

uk/efrs_parity_reference.json still records policyengine-uk 2.89.0. Regenerating it with any later engine changes the surface: 2.89.1 adds bus_fare_spending, and 2.89.2 adds employment_sector and sic_industry_division. With 2.89.2 or later it does three things:

1. It adds bus_fare_spending, employment_sector and sic_industry_division, taking the reference from 145 to 148 populated layers. This already happens at the lock's 2.100.0.
2. It makes build_uk_release_input_coverage_manifest.py require a --candidate-h5 evidence refresh.
3. With Default the BRMA from the household's region policyengine-uk#2021, it adds brma to formula_owned_persisted_overrides_included, taking that list from 13 to 14.

The refresh is purely additive. All three new columns carry signal in the certified candidate, which gives 148 required columns and 0 exclusions, and the spine produces all three. The engine-free tests still pin the frozen candidate at 2.89.0 and 145 columns (test_frozen_candidate_retains_its_original_engine_provenance and three siblings), so the re-pin should be a deliberate step, not a side effect of a bump. experiments/791-household-composition-receipts.md says it belongs with #749.

PolicyEngine/policyengine-uk#2021 needs nothing here

brma computed from the pinned enhanced FRS for 2024, 2025, 2026, 2027, 2030 and 2032 is byte-identical with and without PolicyEngine/policyengine-uk#2021. The same holds on microcosm's certified candidate for 2023, 2024, 2025, 2026, 2028 and 2030; see the comment below. microcosm's frs_brma stage sets brma for every household, so an explicit input always beats the new region default. The only brma change is the reference bookkeeping above.

[MaxGhenis]

Also checked on microcosm's certified UK candidate populace_uk_2023.h5 (revision populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z): brma for 2023, 2024, 2025, 2026, 2028 and 2030 is byte-identical under policyengine-uk 2.104.2 with and without PolicyEngine/policyengine-uk#2021, for all 535,080 households (same sha256 every year). The new formula only fills households with no stored BRMA, and microcosm stores one for every household.

[MaxGhenis]

Corrected the description after an independent review:

• The option A trial is incomplete. microcosm-data[us] and microcosm-frame[policyengine] still drew core 3.32.12, and policyengine + uk could be installed together.
• The certified UK default already fails the loader's check on main (==2.89.2 / ==3.27.1).
• The measured years are now listed exactly.
• The 2.89.1 and 2.89.2 layer additions are now split out.

[juaristi22]

Two ways to fix this. Which do you prefer, @MaxGhenis?

Why it's urgent. #1095 tracks the uk-data fixes microcosm still needs. Since #1100 merged, most of its open items wait only on a policyengine-uk bump, and policyengine-uk has now released all of them:

• Replace generic child and adult flags with each programme's legal definitions policyengine-uk#1896 (2.107.0): is_hbai_dependent_child and is_claimant_or_partner;
• Let a dataset supply each benefit unit's own Pension Credit capital policyengine-uk#2018 (2.105.0): Pension Credit reported capital;
• Date the mixed-age couple Pension Credit exclusion and model the SI 2019/37 saving policyengine-uk#1940 (2.109.4): the mixed-age couple Pension Credit saving;
• Give the legacy severe disability premium its statutory conditions (stacked on #1896) policyengine-uk#1946 (2.112.0): the severe disability flag;
• Give sharers, boarders and lodgers their own rent, tenure and LHA category policyengine-uk#2006 (2.113.0): shared rent, and the rent boarders and lodgers pay.

The same bump also brings engine-only fixes:

• pension-age Housing Benefit new claims (Let pension-age families make new Housing Benefit claims policyengine-uk#1901);
• interest, dividends and rent leaving UC unearned income (Stop counting interest, dividends and rent as Universal Credit unearned income policyengine-uk#1950);
• published LHA rates (Use published LHA determinations and apply Sch 3B paras 3 and 3A policyengine-uk#2022);
• is_renting and Child Benefit opt-outs (Fix Child Benefit opt-outs, rental flags and bus fare modelling policyengine-uk#2140).

Every one of these needs policyengine-core 3.32.9 or later, and 3.32.12 or later from policyengine-uk 2.109.8. The latest releases are policyengine-uk 2.121.0 and core 3.32.17. So microcosm UK can't take any of them until this is fixed.

Option 1: build and certify a new US default under a newer core.

• Both countries keep one core, and the lock stays as simple as it is today.
• Re-claiming the same H5 bytes with a wider core range, as populace-us-2024-spm-20260915 did for policyengine-us, probably won't hold here. Core 3.32.8, 3.32.16 and 3.32.17 change how uprating and auto-carry-over work, so projected-year outputs can move. That makes this a new build and certification.
• The coupling comes back the next time policyengine-uk raises its core floor.

Option 2: give each country its own core pin (completing option A above).

• Add [tool.uv] conflicts between each US-side and each UK-side engine extra, and add policyengine-core==3.32.5 on the US-side extras. Every US install path then resolves the certified trio, while the UK fork moves on its own.

• The 2 October trial had two gaps:

  • its pairs used microcosm-frame[us], which installs no engine, instead of microcosm-frame[policyengine], which installs policyengine-us;
  • the US extras had no core pin, so some US install paths still drew 3.32.12.

• CI already installs one country extra per job, and docs/agent-guide.md already sets up one environment per country, so nothing needs both engines in one environment.

• The PR would add a CI check that:

  • every US install path resolves core 3.32.5, policyengine-us 2.2.1 and spm-calculator 1.0.0;
  • every US × UK combination is refused.

  It would also re-pin APPROVED_UV_LOCK_SHA256.

• On its own it changes no engine version. The UK bump would follow as a separate PR, and the US would stay on core 3.32.5 until its next certification.

Which would you prefer?