[PULL REQUEST] Align employment estimates with SANDAG categories

[GregorSchroeder]

Describe this pull request. What changes are being made?

SANDAG's Economics Team has finalized the employment categories used in the Series 16 Forecast. The categories used a combination of ownership as well as 2-digit NAICS (with the exception of the 721/722 split). They also do not include self-employment as a category, so that is removed here.

Validation and testing

Ran employment module in debug mode for [run_id]=224 for years 2022-2025. Manually checked output CSVs and spot-checked that regional controls were being hit.

What issues does this pull request address?

closes #281
closes #200

Additional context

After first reviews are in and methodology is settled, will update validation scripts and Wikipedia edits to reflect changes.

Once the initial methodology here is approved changes will be made in the production database to the jobs tables adding [ownership_title] and setting existing records to "Total Covered".

A follow up pull request will be the methodology to produce regional employment control totals from the BLS QCEW for 2010-2021, where suppression exists. See https://github.com/SANDAG/BLS/issues/58.

[Copilot]

Pull request overview

Updates the employment module to use SANDAG Series 16 employment categories (ownership + NAICS-derived industry groups) instead of NAICS-only groupings, and removes the self-employment pathway to match the finalized category definition.

Changes:

• Introduces ownership_title throughout the jobs pipeline (SQL inputs/outputs, crosswalks, and Python aggregation/control logic).
• Reworks regional QCEW controls and LODES ingestion/mapping to produce SANDAG employment categories (including GOV rollups and 721/722 handling constraints).
• Removes ACS self-employment inputs/controls and associated blockgroup/tract-to-MGRA crosswalk logic.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
sql/employment/xref_se_to_mgra.sql	Removes the self-employment geography-to-MGRA crosswalk SQL (self-employment no longer produced).
sql/employment/xref_block_to_mgra.sql	Updates block-to-MGRA allocation logic to operate within SANDAG employment categories (ownership + industry).
sql/employment/get_region_self_emp.sql	Removes regional self-employment control query (category removed).
sql/employment/get_region_qcew.sql	Replaces NAICS-only QCEW controls with SANDAG category controls (ownership + industry) using annual vs quarterly sources.
sql/employment/get_naics72_split.sql	Comment/doc formatting updates for NAICS 72 split helper.
sql/employment/get_military_employment.sql	Adds ownership_title to military employment output.
sql/employment/get_lodes_data.sql	Produces LODES block data keyed by ownership_title and SANDAG categories, including GOV rollups.
sql/employment/get_B24080.sql	Removes ACS B24080 self-employment extract (self-employment removed).
sql/create_objects.sql	Adds ownership_title to inputs.controls_jobs and outputs.jobs schema + uniqueness constraints.
README.md	Updates schema diagrams to include ownership_title in jobs-related tables.
python/tests.py	Updates expected distinct key counts and row-count validation to support tuple key columns.
python/employment.py	Refactors employment pipeline to control/allocate by (ownership_title, industry_code) and removes self-employment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (6)

sql/employment/get_region_qcew.sql:45

• THROW 5000 is not a valid user-defined error number in SQL Server (must be >= 50000), and the script no longer returns the sentinel msg value ('QCEW data does not exist') required by utils.read_sql_query_fallback(). As written, missing QCEW years (including future years) will not trigger lookback and can fail later in Python with less actionable errors.

DECLARE @year INTEGER = :year;

-- Data suppression limits this query to 2022-2025 only
IF @year < 2022 OR @year > 2025
    THROW 50000, 'Data suppression prevents calculation outside 2022-2025', 1;

-- Drop temporary table holding final result set

python/employment.py:320

• control_value is retrieved with .iloc[0] without checking whether a matching control row exists (or whether multiple rows exist). If control_totals is missing a category for a year, this will raise an IndexError that’s hard to debug. Consider validating the lookup and raising a clearer error.

                # Get control value and apply integerize_1d
                control_value = (
                    jobs_inputs["control_totals"]
                    .loc[
                        (
                            jobs_inputs["control_totals"]["ownership_title"]
                            == ownership_title
                        )
                        & (
                            jobs_inputs["control_totals"]["industry_code"]
                            == industry_code
                        ),
                        "value",
                    ]
                    .iloc[0]
                )

sql/employment/xref_block_to_mgra.sql:17

• Docstring note has a grammar error ("this query return") and the quoted return value should match the actual sentinel string used elsewhere.

    2) Data prior to year 2017 is not present in the EDD view and must be
    queried directly from the source database table. Note there is no 2016
    data available nor is there ownership data for 2014. In both instances, 
    this query returns "EDD point-level data does not exist".
    3) This must be run on the GIS server.

sql/employment/xref_block_to_mgra.sql:67

• The employment-category shell is derived from a hard-coded EDD year (WHERE [year] = 2024). This makes the script brittle in environments where 2024 data is not present or where the “full set of categories” year changes over time.

        INNER JOIN [EMPCORE].[ca_edd].[ownership]
            ON [vi_ca_edd_employment].[ownership_id] = [ownership].[ownership_id]
        -- Filter year 2024 provides all 23 distinct employment categories
        WHERE [year] = 2024
    ) AS [tt]

sql/employment/get_lodes_data.sql:5

• Typo in comment: niacs_code should be naics_code.

    The mapping below used for [CNS01] to [CNS20] to [niacs_code] (2-digit NAICS) in WAC 
    section of the document linked below. The mapping for [SEG] and [TYPE] are included in 

sql/employment/xref_block_to_mgra.sql:196

• This comment says records with NULL/placeholder industry codes are kept for the total EDD-jobs fallback, but the query inserts into #edd via fn_get_sandag_employment and then filters fn_get_sandag_employment.industry_code IS NOT NULL, so these rows are actually dropped. The comment should be updated to match the current behavior (or the logic changed to truly keep them for pct_edd).

                -- NULL records are mapped to NULL despite falling into this ELSE condition
                -- Keep these records for total EDD jobs xref even if industry code is NULL
                ELSE LEFT([code], 2)

[Copilot]

Pull request overview

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.