Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
DESCRIPTION >
- `collection_contributor_dependency_copy_ds` contains precomputed bus-factor (contributor
dependency) results for the DEFAULT, unfiltered view of every collection - i.e. the exact
output `collection_contributor_dependency.pipe` would compute today with no `repos`/`startDate`/
`endDate`/`platform`/`activity_type`/`includeCollaborations` override and `limit=100`.
- Created via `collection_contributor_dependency_copy.pipe` (`TYPE COPY`, daily schedule) to
avoid recomputing the same expensive per-member GROUP BY over the full activity set on every
request - see that pipe's DESCRIPTION for the full rationale (this materialization exists
because the live computation triggered a Tinybird production rate-limit/concurrency alert on
`lfx_insights` from timeouts under load).
- One row per (collectionSlug, memberId) that made it into that collection's top-100 ranked
contributors, already carrying the running-total percentage and the 51% cutoff membership flag
so the serving pipe only needs a cheap point lookup, not a recompute.
- `collectionSlug` is the collection this row belongs to.
- `id` is the member id (contributor).
- `displayName` is the member's display name.
- `githubHandleArray` is the member's GitHub usernames.
- `contributionCount` is the member's raw contribution count within the collection (all-time,
all-repos, code contributions only - the default filter set).
- `contributionPercentage` is `contributionCount` as a percentage of the collection's total
contribution count.
- `roles` is the member's maintainer roles within the collection, if any.
- `contributionPercentageRunningTotal` is the cumulative percentage when contributors are
ordered by `contributionPercentage DESC, id` - matches the live pipe's window function output.
- `rank` is the 1-indexed rank of this contributor within the collection (1 = highest
contributionCount), preserved so the serving pipe can `ORDER BY rank` cheaply instead of
re-sorting by percentage.
- `totalContributorCount` is the collection's total unique contributor count (same value
repeated on every row for the collection) - matches `active_contributors.contributorCount` in
the live pipe's response.
- `isBusFactorCutoff` marks the rows the live pipe's WHERE clause would return (running total
<= 51% OR running total minus this row's own percentage < 51%) - the serving pipe filters on
this instead of recomputing the cutoff logic.
- `updatedAt` is when this row's collection was last (re)computed - lets the serving pipe or an
operator sanity-check freshness independent of the COPY job's own execution log.

SCHEMA >
`collectionSlug` String,
`id` String,
`displayName` String,
`githubHandleArray` Array(String),
`contributionCount` UInt64,
`contributionPercentage` Float64,
`roles` Array(String),
`contributionPercentageRunningTotal` Float64,
`rank` UInt32,
`totalContributorCount` UInt64,
`isBusFactorCutoff` UInt8,
`updatedAt` DateTime64(3)

ENGINE ReplacingMergeTree
ENGINE_SORTING_KEY collectionSlug, rank
ENGINE_VER updatedAt
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
DESCRIPTION >
- `health_score_v2_development_ds` holds the per-repo Development Activity category score (0-25)
for Health Score v2. Populated by `health_score_v2_development.pipe`.
- `repoUrl` is the repository URL — the join key back to `repositories`.
- `developmentActivityScoreV2` — NULL when covered sub-signal weight is <40% of the 25pt max
(spec Layer 1 graceful degradation), otherwise the rescaled 0-25 score.

SCHEMA >
`repoUrl` String,
`developmentActivityScoreV2` Nullable(UInt8)

ENGINE MergeTree
ENGINE_SORTING_KEY repoUrl
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
DESCRIPTION >
- `health_score_v2_impact_ds` holds the per-repo raw Impact Score (0-100 float, pre-rounding)
for Health Score v2. Populated by `health_score_v2_impact.pipe`.
- `repoUrl` is the repository URL — the join key back to `repositories`.
- `impactScoreRaw` — MAX(packages.impact) * 100 over packages published by this repo. NULL when
the repo has no linked packages with a known impact value.

SCHEMA >
`repoUrl` String,
`impactScoreRaw` Nullable(Float64)

ENGINE MergeTree
ENGINE_SORTING_KEY repoUrl
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
DESCRIPTION >
- `health_score_v2_lifecycle_ds` holds the per-repo Lifecycle state for Health Score v2.
Populated by `health_score_v2_lifecycle.pipe`.
- `repoUrl` is the repository URL — the join key back to `repositories`.
- `lifecycleLabelV2` — one of active/stable/declining/abandoned/archived, or NULL when the
repo has no usable activity signal at all (2026-07-24, IN-1196 — see
health_score_v2_lifecycle.pipe for the exact condition).

SCHEMA >
`repoUrl` String,
`lifecycleLabelV2` Nullable(String)

ENGINE MergeTree
ENGINE_SORTING_KEY repoUrl
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
DESCRIPTION >
- `health_score_v2_maintainer_ds` holds the per-repo Maintainer Health category score (0-40)
for Health Score v2. Populated by `health_score_v2_maintainer.pipe`.
- `repoUrl` is the repository URL — the join key back to `repositories`.
- `maintainerHealthScoreV2` — NULL when covered sub-signal weight is <40% of the 40pt max
(spec Layer 1 graceful degradation), otherwise the rescaled 0-40 score.

SCHEMA >
`repoUrl` String,
`maintainerHealthScoreV2` Nullable(UInt8)

ENGINE MergeTree
ENGINE_SORTING_KEY repoUrl
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
DESCRIPTION >
- `health_score_v2_repo_copy_ds` contains the per-repo Akrites Health Score v2 breakdown.
- Populated by `health_score_v2.pipe` copy pipe.
- Health Score v2 is computed entirely from repo-level/CDP data (maintainer activity, security posture,
development activity) — no package data, per the Health Score v2 methodology
(https://linuxfoundation-dxwx.dsp.so/4ezMyw7n-insights-projects-new-scores). Impact Score is the
one exception — package-mediated by design, computed here as MAX(packages.impact) over the repo's
own published packages (via packageRepos), not an average over unrelated packages.
- `repoUrl` is the repository URL — the join key back to `repositories`/`repos`.
- `maintainerHealthScoreV2` (0-40) — bus factor, org diversity, maintainer responsiveness.
- `securitySupplyChainScoreV2` (0-35) — open vulnerabilities, OpenSSF Scorecard, security practices,
dependency health (checked against the repo's own published packages' dependencies, not an average
across unrelated packages), supply chain integrity (hardcoded 0 — provenance/2FA data not yet piped).
- `developmentActivityScoreV2` (0-25) — release cadence, commit activity, issue resolution, PR merge health.
- `healthScoreV2` (0-100) is the sum of the three categories above, clamped to 100.
- `lifecycleLabelV2` — per-repo lifecycle state (active/stable/declining/abandoned/archived), computed
via the spec's decision tree (archived flag > abandoned > declining > stable > active, first match
wins), or NULL when the repo has zero commits and zero issues/PRs in every window checked (2026-07-24,
IN-1196 — no usable activity signal at all). Project-level rollup uses best-state-wins precedence in
project_insights_copy.pipe.
- `impactScore` (0-100) — MAX(packages.impact) * 100 over packages published by this repo. NULL when
the repo has no linked packages.
- No graceful-degradation/signal-coverage redistribution yet — a repo with zero signal in a category
scores 0 there rather than having points redistributed from available categories.

SCHEMA >
`repoUrl` String,
`maintainerHealthScoreV2` Nullable(UInt8),
`securitySupplyChainScoreV2` Nullable(UInt8),
`developmentActivityScoreV2` Nullable(UInt8),
`healthScoreV2` Nullable(UInt8),
`lifecycleLabelV2` Nullable(String),
`impactScore` Nullable(UInt8)

ENGINE MergeTree
ENGINE_SORTING_KEY repoUrl
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
DESCRIPTION >
- `health_score_v2_security_ds` holds the per-repo Security & Supply Chain category score (0-35)
for Health Score v2. Populated by `health_score_v2_security.pipe`.
- `repoUrl` is the repository URL — the join key back to `repositories`.
- `securitySupplyChainScoreV2` — NULL when covered sub-signal weight is <40% of the 35pt max
(spec Layer 1 graceful degradation), otherwise the rescaled 0-35 score.

SCHEMA >
`repoUrl` String,
`securitySupplyChainScoreV2` Nullable(UInt8)

ENGINE MergeTree
ENGINE_SORTING_KEY repoUrl
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@ DESCRIPTION >
- `activeContributorsPrevious365Days` column is the unique count of active contributors in the previous 365 days (365-730 days ago).
- `activeOrganizationsPrevious365Days` column is the unique count of active organizations in the previous 365 days (365-730 days ago).
- `status` column is the current status of the project (e.g., 'active', 'archived').
- `healthScoreV2` column is the Akrites-methodology composite health score (0-100), averaged from `ossPackages_enriched_ds.healthScore` across the project's linked packages. Distinct from the legacy `healthScore` column (community/contributor-based); null when the project has no linked package data.
- `healthLabel` column buckets `healthScoreV2`: 'excellent' (85+), 'healthy' (70-84), 'fair' (50-69), 'concerning' (30-49), 'critical' (<30). Lowercase, matches `ossPackages_enriched_ds.healthLabel` convention.
- `lifecycleLabel` column is the project's aggregated maintenance state across its packages, using best-state-wins precedence (active > stable > declining > abandoned > archived) over `ossPackages_enriched_ds.lifecycleLabel`.
- `impactScore` column is the average Osprey criticality `impact` (0-100) across the project's packages. Null when the project has no linked package data.
Comment on lines +35 to +38
- `impactLabel` column buckets `impactScore`: 'foundational' (85-100), 'major' (60-84), 'moderate' (30-59), 'minor' (0-29). Null when `impactScore` is null.

TAGS "Project insights", "Metrics"

Expand Down Expand Up @@ -65,7 +70,12 @@ SCHEMA >
`starsPrevious365Days` UInt64,
`forksPrevious365Days` UInt64,
`activeContributorsPrevious365Days` UInt64,
`activeOrganizationsPrevious365Days` UInt64
`activeOrganizationsPrevious365Days` UInt64,
`healthScoreV2` Nullable(UInt8),
`healthLabel` Nullable(String),
`lifecycleLabel` Nullable(String),
`impactScore` Nullable(UInt8),
`impactLabel` Nullable(String)

ENGINE MergeTree
ENGINE_SORTING_KEY type, id
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,66 @@ DESCRIPTION >
replacement for that table is `collection_contributors_leaderboard` (a differently-shaped
pipe - see its own DESCRIPTION for why a naive collectionSlug branch on the single-project
leaderboard pipe isn't used there either).
- Combines data from `collection_contributors_leaderboard` and `active_contributors` (both
already collectionSlug-aware) to provide dependency risk assessment at collection scale.
- **Two paths, branched on whether any optional filter is present:**
- **Precomputed path (no `repos`/`startDate`/`endDate`/`platform`/`activity_type` and
`includeCollaborations` absent or `false`)** - this is what the real frontend widget always
requests on first paint, before a user touches a filter control. Reads
`collection_contributor_dependency_copy_ds`, a daily `TYPE COPY` materialization (see
`collection_contributor_dependency_copy.pipe`), instead of recomputing live. This exists
because the live computation - via `collection_contributors_leaderboard`'s
`collection_member_aggregates` node - must fully materialize a `GROUP BY af.memberId` over the
entire filtered activity set before it can ORDER BY/LIMIT (LIMIT only reduces output size, not
scan/aggregation cost), which at collection scale (e.g. `cncf` has 149,677 distinct
contributors) triggered a Tinybird production rate-limit/concurrency alert on `lfx_insights`
from timeouts under load.
- **Live path (any of those filters present)** - unavoidable, since filters are genuine dynamic
user input and the precomputed table only covers the unfiltered default. Unchanged from the
prior implementation: combines `collection_contributors_leaderboard` and `active_contributors`
(both already collectionSlug-aware).
- Precomputed path only supports `limit=100` (the only value the frontend ever sends for this
widget, and the only value the copy pipe materializes) - if `limit` is explicitly set to
anything else, falls through to the live path even with no other filters present.
- Parameters:
- `collectionSlug`: Required collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`.
- `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered_by_collection`)
- `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00')
- `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59')
- `platform`: Optional string filter for source platform (e.g., 'github', 'discord', 'slack')
- `activity_type`: Optional string filter for single activity type (e.g., 'authored-commit')
- `includeCollaborations`: Optional boolean, defaults to false - same toggle as the single-project pipe.
- `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered_by_collection`). Presence routes to the live path.
- `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00'). Presence routes to the live path.
- `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59'). Presence routes to the live path.
- `platform`: Optional string filter for source platform (e.g., 'github', 'discord', 'slack'). Presence routes to the live path.
- `activity_type`: Optional string filter for single activity type (e.g., 'authored-commit'). Presence routes to the live path.
- `includeCollaborations`: Optional boolean, defaults to false - same toggle as the single-project pipe. `true` routes to the live path.
- `limit`: Optional integer for the number of top contributors to rank before finding the 51%
cutoff. Inherited by `collection_contributors_leaderboard` (this pipe has no default of its
own), whose own default is 10 if omitted - callers doing bus-factor analysis should pass
limit=100 explicitly (same convention as the single-project `contributor_dependency.pipe` /
`contributors_leaderboard.pipe` pair) to reliably reach the 51% cutoff.
`contributors_leaderboard.pipe` pair) to reliably reach the 51% cutoff. A value other than 100
routes to the live path.
- Response: `id`, `displayName`, `githubHandleArray`, `contributionCount`, `contributionPercentage`, `roles`, `contributionPercentageRunningTotal`, `totalContributorCount`

TAGS "Insights, Widget", "Collection", "Contributors"

NODE collection_contributor_dependency_precomputed
DESCRIPTION >
Precomputed path: point lookup into the daily materialization, filtered to the rows the live
pipe's WHERE clause would itself return (`isBusFactorCutoff = 1`), ordered by rank.

SQL >
%
SELECT
id,
displayName,
githubHandleArray,
contributionCount,
contributionPercentage,
roles,
contributionPercentageRunningTotal,
totalContributorCount
FROM collection_contributor_dependency_copy_ds
WHERE
collectionSlug
= {{ String(collectionSlug, description="Filter by collection slug", required=True) }}
AND isBusFactorCutoff = 1
ORDER BY rank ASC

NODE collection_contributions_percentage_running_total
SQL >
%
Expand All @@ -50,3 +91,19 @@ SQL >
contributionPercentageRunningTotal <= 51
OR (contributionPercentageRunningTotal - contributionPercentage < 51)
ORDER BY contributionPercentageRunningTotal ASC

NODE collection_contributor_dependency_result
DESCRIPTION >
Routes between the precomputed and live paths. Precomputed path applies only when no optional
filter is present and `limit` (if given) is exactly 100 - the exact shape the frontend always
requests on first paint.

SQL >
%
{% if not defined(repos) and not defined(startDate) and not defined(endDate) and not defined(
platform
) and not defined(activity_type) and not Boolean(includeCollaborations, false) and Int32(
limit, 100
) == 100 %} SELECT * FROM collection_contributor_dependency_precomputed
{% else %} SELECT * FROM collection_contributions_percentage_running_total
{% end %}
Loading
Loading