improving lobbying data pipeline reliability - #2222
Open
nesanders wants to merge 3 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
nesanders
changed the title
…ata scale
Three related Firestore document/field size-limit bugs surfaced when the
weekly incremental scraper was run for the first time against the full
production-scale dataset (300K+ filings):
1. The live weekly cursor (scrapers/lobbying) stored the entire processed-URL
history and summary cache as two fields on one document. That document
exceeded Firestore's 1MB limit partway through a run, silently failing
(and thus skipping) every registrant processed afterward. Moved to
subcollections — one small doc per URL — mirroring the pattern the
backfill cursor already used, with point lookups instead of an in-memory
set/dict.
2. compute_stats() streamed the full lobbyingFilings/lobbyingRegistrants
collections (300K+ docs) in one unbounded query, which timed out
server-side; the client library's automatic stream-retry then crashed on
an internal AttributeError instead of recovering. Replaced with
cursor-paginated batches (50K docs/request) and a manual retry that
re-issues a fresh query rather than resuming a broken stream.
3. Once (2) was fixed, compute_stats() reached a third limit: the
billSummaries_{court} JSON blob itself exceeded Firestore's 1MB
field-size limit for the current session (1,057KB for court 194's ~5,600
bills), with courts 192/193 close behind. Restructured to one small doc
per bill in a bills subcollection instead of one JSON blob per court —
same fix pattern as (1), applied to writer.py, seedLobbyingStats.ts, and
the frontend fetcher in components/db/lobbying.ts.
All three fixes validated end-to-end against dev Firestore at current scale
(373K filings, 25.6K registrants, 11 courts including the previously-failing
194th).
Also includes scripts/firebase-admin/checkLobbyingFreshness.ts, a read-only
diagnostic for checking scraper cursor state and data recency.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
run_backfill() marked a year "complete" after one pass and skipped it forever on every future run. That's wrong for the current (still-accruing) year: a run partway through the year would mark it complete based on whatever existed at that moment, silently missing every disclosure filed afterward — no future backfill run would ever see it again. This is exactly what happened to 2026 in production: marked complete in July with 0 disclosures captured. run_backfill already has a fully correct, granular completeness check — _is_backfill_processed, a per-URL subcollection lookup. The year-level flag only ever bought a coarse fast-path (skip re-listing a year's registrants entirely) and it's what caused the bug. Removed it: every run now always re-lists every requested year (one cheap HTTP request per year) and relies solely on the per-URL cursor for correctness, so no year can ever be skipped wholesale again. Added tests/test_scrape.py with a small in-memory Firestore fake (real enough to simulate write-then-read-back across calls, unlike a plain mock) covering both cursor systems: - Regression test reproducing the exact bug scenario (empty pass, then real data appears for the same year) — fails against the old code with 4/4 backfill tests red, passes with the fix, confirmed by checking out the pre-fix scrape.py and rerunning the suite against it. - Backfill always re-lists every year, per-URL dedup still works, dry-run never touches Firestore, completedYears is never written anywhere. - Weekly-mode cursor sanity checks (prior-year caching, current-year always live, parent doc stays small — all state in subcollections). Also validated live against dev: --mode backfill --year 2005 --limit 2 run twice confirms the year is re-listed both times while already-processed disclosures are correctly not reprocessed (0 new on both runs, as expected since 2005 was already backfilled in July). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nesanders
force-pushed
the
fix/lobbying-weekly-scraper-reliability
branch
from
September 9, 2026 00:30
2d12bb5 to
462c917
Compare
nesanders
marked this pull request as ready for review
September 9, 2026 00:32
nesanders
requested review from
Mephistic,
alexjball,
kiminkim724,
mertbagt,
mvictor55,
sashamaryl and
timblais
as code owners
September 9, 2026 00:32
…ction
The billSummaries_{court} restructure (previous commit) moved per-bill
counts from a single document field to a `bills` subcollection, but the
existing `match /lobbyingMeta/{id}` rule only covers documents directly in
that collection — Firestore rules aren't recursive, so it never covered the
new subcollection. The bills index page was failing with "Missing or
insufficient permissions" as a result; caught on a preview deployment,
since all earlier validation ran through the Admin SDK, which bypasses
security rules entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three more reliability fixes for the lobbying data pipeline, found while running the weekly scraper for the first time against the full production-scale dataset (see #2221 for the first, already-mergeable fix). Both are variants of the same root cause: a single Firestore document/field has a hard size ceiling, and this pipeline had a couple more places that stored unboundedly-growing state in one doc.
compute_stats()crashed on the full-collection stream and the summary blob it wrotelobbyingFilings/lobbyingRegistrants(300K+ docs) via one unbounded query, which timed out server-side, and the installed Firestore client's retry-on-timeout path hit an internal bug instead of recovering. Fixed with cursor-paginated batches and a manual retry that re-issues a fresh query.completedYearspermanently skipped a year based on a partial scanFix bills index page failing with "Missing or insufficient permissions."
billSummaries_{court}restructure in this PR moves per-bill counts from a single document field into a bills subcollection, but Firestore security rules aren't recursive — the existing match/lobbyingMeta/{id}rule only covers documents directly in that collection, not the new nested subcollection. Added an explicit match/bills/{billId}rule underlobbyingMeta/{id}(mirroring the existing transcriptions/{tid}/utterances/{uid} nested-rule pattern already in the file), deployed to dev, and confirmed the bills page loads correctly.Testing
tests/test_scrape.py) against a small in-memory Firestore fake, including a regression test for thecompletedYearsbug — confirmed it fails against the pre-fix code and passes with the fix.pytest tests/).compute_stats()completing end-to-end including the previously-failing session, and backfill correctly re-listing a year while still deduping already-processed disclosures.Full details and validation steps are in the individual commit messages.