feat: bucketed lambda merger for activityRelations_enriched_deduplicated datasources

[epipav]

No description provided.

[cursor]

PR Summary

High Risk
This changes the core unfiltered activityRelations serving layer and every dependent CDP, PR, and monitoring pipe; deploy needs bucket bootstraps and correct cutover or queries can miss data or use wrong snapshot semantics.

Overview
Replaces the monolithic activityRelations_enriched_deduplicated_ds pipeline with three parallel buckets (activityRelations_enriched_deduplicated_bucket_0-2_ds) keyed by cityHash64(segmentId) % 3, exposed via activityRelations_enriched_deduplicated_bucket_union.

The old daily append merger (activityRelations_snapshot_merger_copy) is removed in favor of staggered daily replace-mode per-bucket mergers (activityRelations_snapshot_merger_copy_0..2) that merge MV deltas with carry-forward rows; failed runs keep the previous bucket data. Bucket datasources partition by toYear(timestamp) and drop bucket TTL; MV delta retention extends to 3 days (activityRelations_enrich_snapshot_MV_ds).

Bootstrap is split into per-bucket on-demand initial snapshot pipes (activityRelations_enrich_initial_snapshot_0..2). Downstream pipes (CDP activities, daily counts, PR initial snapshot, segment metrics, monitoring) read the union and no longer filter max(snapshotId)—docs warn against global snapshot max on the union and against bare count() over it (monitoring sums per-bucket counts instead).

^{Reviewed by Cursor Bugbot for commit a3bd27d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[Copilot]

Pull request overview

Introduces a three-bucket, replace-mode serving layer for unfiltered activity relations.

Changes:

• Adds bucket datasources, bootstrap pipes, daily mergers, and a union pipe.
• Migrates downstream queries from the legacy snapshot datasource.
• Extends delta retention and updates architecture documentation.

Review note: The PR title is missing the required trailing JIRA key (CM-XXX).

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
services/libs/tinybird/README.md	Documents bucketed lambda output.
services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe	Reads from the bucket union.
services/libs/tinybird/pipes/monitoring_entities.pipe	Counts rows per bucket.
services/libs/tinybird/pipes/cdp_segment_metrics_total_sink.pipe	Migrates total metrics source.
services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe	Migrates segment metrics source.
services/libs/tinybird/pipes/cdp_organization_segment_aggregates_initial_snapshot.pipe	Migrates organization aggregates.
services/libs/tinybird/pipes/cdp_member_segment_aggregates_initial_snapshot.pipe	Migrates member aggregates.
services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy.pipe	Removes the legacy merger.
services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy_2.pipe	Adds bucket 2 merger.
services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy_1.pipe	Adds bucket 1 merger.
services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy_0.pipe	Adds bucket 0 merger.
services/libs/tinybird/pipes/activityRelations_enriched_deduplicated_bucket_union.pipe	Unions serving buckets.
services/libs/tinybird/pipes/activityRelations_enrich_initial_snapshot_2.pipe	Bootstraps bucket 2.
services/libs/tinybird/pipes/activityRelations_enrich_initial_snapshot_1.pipe	Bootstraps bucket 1.
services/libs/tinybird/pipes/activityRelations_enrich_initial_snapshot_0.pipe	Bootstraps bucket 0.
services/libs/tinybird/pipes/activities_relations_filtered.pipe	Migrates filtered activity queries.
services/libs/tinybird/pipes/activities_daily_counts.pipe	Migrates daily activity counts.
services/libs/tinybird/lambda-architecture.md	Documents the new architecture.
services/libs/tinybird/datasources/activityRelations_enriched_deduplicated_bucket_2_ds.datasource	Defines bucket 2 storage.
services/libs/tinybird/datasources/activityRelations_enriched_deduplicated_bucket_1_ds.datasource	Defines bucket 1 storage.
services/libs/tinybird/datasources/activityRelations_enriched_deduplicated_bucket_0_ds.datasource	Updates bucket 0 storage.
services/libs/tinybird/datasources/activityRelations_enrich_snapshot_MV_ds.datasource	Extends delta retention.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

services/libs/tinybird/pipes/activityRelations_enriched_deduplicated_bucket_union.pipe:18

• This union becomes the source for all migrated consumers in the same release that creates buckets 1 and 2, but those datasources remain empty until the @on-demand initial-snapshot pipes are run. The scheduled mergers can recover only the MV's three-day retention, so deploying these files before completing the full bootstrap serves an incomplete historical dataset (at least the two new buckets) for the duration of the bootstrap. Please use a staged rollout that seeds and validates all buckets before switching consumers/enabling the union.

    FROM activityRelations_enriched_deduplicated_bucket_0_ds
    UNION ALL
    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_1_ds

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 747071b. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

services/libs/tinybird/pipes/activityRelations_enriched_deduplicated_bucket_union.pipe:21

• All live consumers are switched to this union in the same change, but these three targets are populated only by @on-demand initial-snapshot pipes. A one-shot deployment will therefore serve empty or partially bootstrapped results until all three full copies finish; if a scheduled merger runs first, an empty target is seeded from only the three-day MV retention window. Please stage the rollout: deploy and bootstrap/validate all buckets first, then switch consumers and enable the scheduled mergers.

    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_0_ds
    UNION ALL
    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_1_ds
    UNION ALL
    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_2_ds

[gaspergrom]

LGTM

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

services/libs/tinybird/pipes/activityRelations_enriched_deduplicated_bucket_union.pipe:21

• The two newly created buckets are published through this union before anything guarantees that their @on-demand initial snapshots have run. If a scheduled merger runs first, it can populate them only from the MV's three-day retention window and then carry that partial history forward indefinitely, so consumers silently return incomplete historical data. Make bootstrapping a required pre-promotion step or gate publication/scheduled mergers until both buckets are fully initialized.

    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_1_ds
    UNION ALL
    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_2_ds