perf: precompute collection_contributor_dependency filtered variants (IN-1196)

[gaspergrom]

Summary

• Extends the collection-page precompute work to cover the contributor-dependency widget's real filtered traffic — production telemetry showed 86% of requests carry a date range, and 98% of those match one of the 8 fixed date-picker presets (not the "no filters at all" case the original v1 precompute covered).
• Adds a 16-variant precomputed matrix (8 date presets × 2 includeCollaborations states) for collection_contributor_dependency, replacing the old v1 single-variant (unfiltered-only) precomputed path.
• Splits collection_contributor_dependency_copy and collection_contributors_leaderboard_copy each into 8 presetKey-scoped copy pipes (past90days, past180days, past365days, previousQuarter, previousYear, previous5years, previous10years, allTime), mirroring the existing collection_insights_aggregate_copy pattern.
• Rewrites collection_contributor_dependency_copy_ds and collection_contributors_leaderboard_copy_ds as append + snapshotId datasources so a daily run doesn't clobber the prior day's rows mid-query.
• Updates the serving pipes (collection_contributor_dependency.pipe, collection_contributors_leaderboard.pipe) to route between the precomputed path (requires presetKey, no repos/platform/activity_type, limit=100) and the live GROUP BY path at template-render time — ClickHouse evaluates all UNION ALL branches regardless of WHERE guards, so routing has to happen at render time, not query time.
• Deletes the superseded v1 collection_contributor_dependency_copy.pipe / collection_contributor_dependency_copy_ds.datasource.

Why

At collection scale (e.g. cncf has 149,677 distinct contributors), the live GROUP BY af.memberId path for the contributor-dependency widget was triggering Tinybird production rate-limit/concurrency alerts under load. The original precompute only covered fully-unfiltered requests, but real traffic is overwhelmingly filtered by one of the date picker's 8 fixed presets — so the fast path almost never activated. This closes that gap.

Changes

• services/libs/tinybird/pipes/collection_contributor_dependency.pipe — adds precomputed-path routing node, updated DESCRIPTION.
• services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe — same routing pattern (was already partially split in an earlier phase; this makes the presetKey contract consistent with contributor-dependency).
• services/libs/tinybird/pipes/collection_insights_aggregate.pipe — minor consistency update.
• 16 new *_copy_<presetKey>.pipe files (8 for contributor-dependency, 8 for leaderboard).
• collection_insights_aggregate_copy.pipe — new copy pipe for that datasource.
• 3 rewritten/new *_copy_ds.datasource files (append + snapshotId schema).
• Deletes collection_contributor_dependency_copy.pipe (v1, superseded).

Deploy order

Must deploy before the companion insights PR (#2041) — that PR sends presetKey values that need to already resolve against a live pipe here.

Test plan

• Validated pipe/datasource SQL and schema changes via Tinybird MCP against production data
• Deployed through staging then production via tb CLI
• Verified all 16 contributor-dependency variants and 16 leaderboard variants present with matching row counts for a real collection (cncf)
• Root-caused and fixed a past365days zero-rows gap post-deploy (one pipe instance's daily cron hadn't fired yet after the redeploy reset its run history) — manually triggered, verified correct data end-to-end

JIRA: IN-1196

[cursor]

PR Summary

Medium Risk
Large analytics pipeline change with snapshot-based reads and strict presetKey routing; collection_insights_aggregate has no live fallback if copy data is stale or missing.

Overview
Adds daily precomputed paths for collection Contributors-tab Tinybird APIs so typical traffic (fixed 8 date presets × 2 collaboration states) avoids live GROUP BY over full activity at collection scale.

Leaderboard & bus-factor: Eight collection_contributors_leaderboard_copy_<presetKey> jobs append into collection_contributors_leaderboard_copy_ds (snapshotId + 2-day TTL). Eight matching collection_contributor_dependency_copy_<presetKey> jobs derive the 51% bus-factor rows from that leaderboard table (no raw activity scan). collection_contributor_dependency.pipe and collection_contributors_leaderboard.pipe now branch at template render time on presetKey (and filter params / limit=100 for dependency) to point lookups on the copy tables, with the previous live SQL kept as fallback.

Insights aggregate: New collection_insights_aggregate_copy.pipe + datasource; collection_insights_aggregate.pipe only reads the copy table (no live fallback). Copy logic aggregates distinct segment members before collection fan-out to avoid multiplying activity scan cost.

Supersedes the v1 single-variant contributor-dependency precompute with the full 16-variant matrix aligned to the frontend picker contract.

^{Reviewed by Cursor Bugbot for commit e08c160. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e08c160. Configure here.}

[cursor]

Precomputed path blocked by dates

High Severity

The precomputed-path guard requires startDate and endDate to be absent, but the pipe docs say those params are ignored when presetKey is present, and the sibling collection_contributors_leaderboard.pipe does not check them. Once the companion frontend sends presetKey with the matching date range (the common traffic shape), this widget stays on the expensive live path and the optimization never activates.

 

^{Reviewed by Cursor Bugbot for commit e08c160. Configure here.}

[cursor]

Empty collections dropped from aggregate

Medium Severity

The final copy node INNER JOINs project counts to activity rows, so collections with projects but no qualifying contributors never get a row in collection_insights_aggregate_copy_ds. The old live pipe always returned one row via scalar subqueries (with uniqueContributorCount 0). The serving pipe has no live fallback, so those collections now return empty instead of project count and health score.

 

^{Reviewed by Cursor Bugbot for commit e08c160. Configure here.}

[cursor]

allTime copy schedule race

High Severity

collection_contributor_dependency_copy_allTime and collection_contributors_leaderboard_copy_allTime both run at 15 2 * * *. The dependency pipe reads upstream rows with snapshotId = max(snapshotId). Once earlier leaderboard presets have written today's snapshot, a still-running allTime leaderboard job makes that lookup return zero rows, so the dependency variant is missing until a later successful run—the same class of gap already seen for past365days in the test plan.

Additional Locations (1)

• services/libs/tinybird/pipes/collection_contributors_leaderboard_copy_allTime.pipe#L136-L137

 

^{Reviewed by Cursor Bugbot for commit e08c160. Configure here.}

[Copilot]

Pull request overview

Precomputes collection contributor metrics for fixed date presets to reduce expensive live Tinybird aggregations.

Changes:

• Adds preset-specific leaderboard and dependency materializations.
• Routes eligible requests to precomputed snapshots.
• Adds aggregate collection insights precomputation and supporting datasources.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
collection_insights_aggregate.pipe	Serves precomputed aggregates.
collection_insights_aggregate_copy.pipe	Builds daily collection aggregates.
collection_contributors_leaderboard.pipe	Routes live/precomputed leaderboards.
collection_contributors_leaderboard_copy_past90days.pipe	Precomputes 90-day leaderboard.
collection_contributors_leaderboard_copy_past180days.pipe	Precomputes 180-day leaderboard.
collection_contributors_leaderboard_copy_past365days.pipe	Precomputes 365-day leaderboard.
collection_contributors_leaderboard_copy_previousQuarter.pipe	Precomputes previous-quarter leaderboard.
collection_contributors_leaderboard_copy_previousYear.pipe	Precomputes previous-year leaderboard.
collection_contributors_leaderboard_copy_previous5years.pipe	Precomputes five-year leaderboard.
collection_contributors_leaderboard_copy_previous10years.pipe	Precomputes ten-year leaderboard.
collection_contributors_leaderboard_copy_allTime.pipe	Precomputes all-time leaderboard.
collection_contributor_dependency.pipe	Routes live/precomputed dependency results.
collection_contributor_dependency_copy_past90days.pipe	Derives 90-day dependency results.
collection_contributor_dependency_copy_past180days.pipe	Derives 180-day dependency results.
collection_contributor_dependency_copy_past365days.pipe	Derives 365-day dependency results.
collection_contributor_dependency_copy_previousQuarter.pipe	Derives previous-quarter dependency results.
collection_contributor_dependency_copy_previousYear.pipe	Derives previous-year dependency results.
collection_contributor_dependency_copy_previous5years.pipe	Derives five-year dependency results.
collection_contributor_dependency_copy_previous10years.pipe	Derives ten-year dependency results.
collection_contributor_dependency_copy_allTime.pipe	Derives all-time dependency results.
collection_insights_aggregate_copy_ds.datasource	Defines aggregate storage.
collection_contributors_leaderboard_copy_ds.datasource	Defines snapshot leaderboard storage.
collection_contributor_dependency_copy_ds.datasource	Defines snapshot dependency storage.

Comments suppressed due to low confidence (1)

services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe:160

• This data branch has the same partial-snapshot race as the count branch: global max(snapshotId) advances when any preset writer completes, making unfinished presets temporarily return an empty leaderboard. Resolve the latest snapshot for the requested preset instead.

                snapshotId = (SELECT max(snapshotId) FROM collection_contributors_leaderboard_copy_ds)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gaspergrom]

Closing without merging — abandoning this optimization work per decision to revert all Tinybird performance-optimization changes (Phases 1-3) and start fresh.