Skip to content

National generalization of the ACS SLD packages (all states, ~6.8k districts) #138

Description

@MaxGhenis

Follow-up to the Utah-pilot SLD packages (branch census-acs-sld-2024, populace#625). Verified mechanics for going national, so the generalization is mechanical once the pilot shape passes judge review:

  • Fetch: the API rejects in=state:* for SLD hierarchies but accepts comma-listed FIPS (in=state:01,02,...) — one canonical national witness per (table, chamber) works, same single-file shape as the CD packages. 50 states + DC; PR excluded per the populace state spine.
  • Scale (measured from the pilot's compact record-set shape — rows = districts with geography overrides, measures = columns with constraints): ~1,950 SLDU + ~4,833 SLDL districts → roughly 2.3MB YAML for the biggest package (S0101 lower) and ~3-4MB for the biggest witness; totals ~12MB YAML + ~12MB witnessed JSON across the six packages. CD-package-proportionate, no harness change needed.
  • Suppression handling to add: nationally, B19013 medians (and possibly sparse B19001 cells) will carry ACS suppression markers for thin districts. The pilot generator hard-fails on suppression; the national generator needs the skip-with-record path — suppressed cells excluded from the package with an explicit exclusions list in the package (never silently absent).
  • Special cases already structural in the pilot: NE unicameral and DC wards (no lower chamber), multi-member districts as Census tabulates them; NH floterial representation should be verified against the witnessed rows when the national fetch lands.
  • Ordering note: witness row order is the API's (sorted by state then district in practice); the generator asserts sortedness per state and derives row numbers from the parsed stream, so ordering drift is a loud failure.

Blocked behind: the pilot PR's judge review (shape adjudication) and the pending publish-raw R2 run for the pilot manifests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions