Organization maintenance and automation tasks for the Netresearch GitHub organization.
File: .github/workflows/star-notifications.yml
Monitors all public repositories in the netresearch organization for new stars, forks, watchers, and dependents, and sends notifications to Matrix.
Schedule: Daily at 06:00 UTC
Manual trigger: Yes (via Actions tab β "Run workflow")
Notifications sent to: Matrix room via Hookshot webhook
Dependencies: requirements/star-notifications.txt, hash-pinned and installed
wheels-only. Regenerate it rather than editing a hash by hand:
uv pip compile --generate-hashes --python-version=3.15 \
--output-file=requirements/star-notifications.txt requirements/star-notifications.in| Secret | Description |
|---|---|
MATRIX_WEBHOOK_URL |
Matrix Hookshot webhook URL |
STAR_NOTIFICATIONS_PAT |
Token for the stargazers, subscribers and forks endpoints, which GitHub restricts to a repository's admins and collaborators since 2026-06-30. Without read access the digest can only report how many stars a repository gained, not who starred it |
- Fetches all public repos in the org
- Gets current stargazers, forks, watchers, and dependents for each repo
- Compares with previously known data (stored as artifact)
- Sends one Matrix digest covering every new star, fork, watcher, or dependent since the previous run
- Updates state for next run
The first run indexes existing data without sending notifications to avoid spam. Dependents are read from every package GitHub lists on a repo's /network/dependents page and from every page of each, since GitHub opens that page on one package only, not necessarily the one in use. A package with more than 100 pages (MAX_DEPENDENTS_PAGES, 3,000 dependents) is not read in part: that repo's dependents read is dropped, its saved dependents stay as they are and nothing is reported for it. When the scraper starts to read more than before (DEPENDENTS_SCOPE in check-stars.py), its first run takes a new silent baseline of every repo's dependents, so dependents that really are new on that day are not announced either. GitHub's "N Repositories" header can be higher than the number of repositories it lists; only listed ones can be reported. A day without new events sends nothing. If the digest cannot be delivered, the run fails without saving the new state, so the next run reports the same events again.
The digest's first line stays visible and counts the new stars, forks and dependents since the previous completed run, naming that start time; the individual events follow in a collapsed <details> block, at most 50 lines with a link to the run's log for the rest. Clients without <details> support show the block expanded.
- β Stars: When someone stars a repository
- π΄ Forks: When someone forks a repository
- π Watchers: When someone starts watching a repository
- π¦ Dependents: When a new repository depends on one of our repositories (includes the dependent's star and fork count)
Automatically adds new issues and pull requests to the Netresearch TYPO3 project board.
Project Board: https://github.com/orgs/netresearch/projects/4
Documentation: docs/project-board-automation.md
| Component | Location |
|---|---|
| GitHub App | Netresearch Project Bot |
| Secrets | Organization Secrets |
| Workflow | .github/workflows/add-to-project.yml (per repository) |
| Secret | Description |
|---|---|
PROJECT_APP_ID |
GitHub App ID |
PROJECT_APP_PRIVATE_KEY |
GitHub App private key (PEM) |
File: .github/workflows/impact-dashboard.yml
Collects community-impact metrics for repositories listed in
config/dashboard-repos.yaml, renders a static
dashboard, and publishes it to the gh-pages branch.
The config combines two mechanisms:
- Patterns auto-include every public repo (archived included) whose name matches
a prefix/suffix or whose primary language matches a value (today:
t3x-*,*-skill, languageGo). includeexplicitly adds individual repos by name and assigns them to a category. This is how the Commerce, Ansible, and Tools categories are populated. Add or remove entries here in a PR β the workflow picks them up on the next run.
Schedule: Daily at 03:00 UTC
Manual trigger: Yes (via Actions tab β "Run workflow")
URL: https://netresearch.github.io/maint/ (once GitHub Pages is enabled on the gh-pages branch).
After a merge: a merge to main does not redeploy the dashboard; the build
runs only on the schedule and on a manual dispatch. Run
gh workflow run impact-dashboard.yml --repo netresearch/maint --ref main or
wait for the next daily run at 03:00 UTC. Check the live page only after that
run and the pages-build-deployment run it triggers on gh-pages have both
finished. Publication uses keep_files: true, so a page the build no longer
produces, such as repo/<name>/ and de/repo/<name>/ of a repository the
dashboard no longer lists, stays on gh-pages until it is deleted from that
branch as well.
Per repo β lifetime and last 30 days where meaningful:
- Metadata: language, license, topics, homepage, created/updated timestamps
- Stars, forks, watchers, network count
- Issues (open / closed, opened in 30d)
- Pull requests (open / merged / closed-unmerged, opened and merged in 30d)
- Releases (count, latest release, total asset downloads)
- Contributors (total, external = not in public org members, top 10)
- Commits (lifetime on default branch, last 30 days)
- Packagist downloads (total / monthly / daily) for PHP repos with a
composer.json - GHCR container pulls (lifetime total + last 30 days, exact numbers) β scraped from the package page (
<h3 title="N">and the 30-day sparkline'sdata-merge-countbars) - Dependents count ("Used by N repositories / N packages") β scraped from
/network/dependents - Traffic (clones, views, top referrers, top paths) β last 14 days only, requires PAT
Aggregate totals and 90 days of daily snapshots feed the time-series charts.
Two metrics rely on HTML scraping of github.com β they work today but may break if GitHub changes the DOM:
- GHCR pulls β
github.com/<org>/<repo>/pkgs/container/<pkg>exposes the exact count in atitleattribute and the 30-day bars asdata-merge-count. Package name is discovered viaGET /orgs/<org>/packages(requiresread:packages) or falls back to assumingpackage == reponame. - Dependents β
github.com/<org>/<repo>/network/dependentsshows"N Repositories"/"N Packages"under toggles filtered bydependent_type. A repo with several packages (one per module path or manifest) gets one such page per package, selected bypackage_id; the largest count among them is reported, a lower bound. Adding them up would count a repo once per package it uses, and every reusable workflow is a package of its own (netresearch/.githublists 65). If any of a repo's package pages cannot be read, the repo keeps the previous run's counts instead of dropping to 0.
Not collected:
- TER downloads (extensions.typo3.org) β no stable public API; Packagist stats are a reasonable proxy for TYPO3 extensions installed via composer.
| Secret | Required | Description |
|---|---|---|
GITHUB_TOKEN |
Automatic | Provided by Actions; covers all core metrics. |
IMPACT_DASHBOARD_PAT |
Optional | Fine-grained PAT on the netresearch org with Administration: Read (for traffic) and optionally Packages: Read (to enumerate GHCR packages exactly instead of falling back to name-based probing). Or a classic PAT with repo + read:packages scopes. Without this secret, traffic is omitted; GHCR still works for repos whose container package name matches the repo name. |
- Add the PAT secret (optional but recommended for traffic):
- Create a PAT at https://github.com/settings/tokens with
reposcope, or a fine-grained token on thenetresearchorg withAdministration: Readon the relevant repos. - Add as
IMPACT_DASHBOARD_PATunder Settings β Secrets β Actions in this repo.
- Create a PAT at https://github.com/settings/tokens with
- Run the workflow once manually (Actions tab β Impact Dashboard β Run workflow). This creates the
gh-pagesbranch and the first snapshot. - Enable GitHub Pages: Settings β Pages β Source = "Deploy from a branch", Branch =
gh-pages, Path =/ (root). - Lifetime traffic: GitHub's traffic API only returns the trailing 14 days. A lifetime total is therefore only as old as the first successful run β the pipeline accumulates it via daily snapshots going forward.
A coarse single-number indicator of participation from outside the organisation, per repository:
estimated_downstream_reach = external_contributors Γ 3
+ total_issues
+ prs_merged
+ forks Γ 2
+ dependents_repos Γ 2
An estimate, not a measurement: the weights are a judgement call and the inputs carry the known gaps documented on the dashboard. Useful for ordering repositories against each other, not for a decision. The dashboard renders the formula next to the number and never shows one without the other β a build gate enforces that.
Formerly blast_radius. That key is still emitted as a deprecated alias for one
release cycle; read estimated_downstream_reach.
scripts/render_site.py turns the collector's JSON into finished HTML before
publishing. The dashboard used to ship an empty shell and build every figure in
the browser, so crawlers and text browsers saw "Loadingβ¦" and nothing was
citable.
What the renderer produces:
| Path | Contents |
|---|---|
index.html, de/index.html |
Dashboard, both languages |
repo/<name>/, de/repo/<name>/ |
One stable URL per repository |
snapshot/<date>/, de/snapshot/<date>/ |
Immutable archive of one run |
data/repositories.csv |
The repository table, for reuse |
data/data-dictionary.json |
What every field means, and its kind |
sitemap.xml, robots.txt, CITATION.cff |
Crawling and citation |
JavaScript is left with sorting, filtering, the charts and the copy-citation button. It never introduces a figure the HTML does not already contain, and the chart series is published as a table beside each chart.
Archived figures and collection timestamps are never regenerated. Legacy action labels, metadata and site navigation can be corrected without changing those figures; where an old daily JSON was overwritten, the preserved HTML is the citable original. New collection runs also receive a content-hash archive ID, so two runs on the same day have different permanent HTML and JSON URLs. Historical daily URLs keep their first snapshot. Copied citations and Dataset metadata point to the exact archived run. The workflow seeds the build directory with the previously published snapshot pages so the renderer skips them: a cited snapshot URL has to keep showing the figures it showed when it was cited.
Content lives in dashboard/i18n/{en,de}.yaml; markup in
dashboard/templates/. Chart.js and the brand fonts are served from this origin
β the page makes no third-party request, and the build gate fails if one appears.
OUTPUT_DIR=build python scripts/render_site.py # render
OUTPUT_DIR=build python scripts/verify_site.py # gate
uv run scripts/render_og.py # social cards, after a headline changescripts/verify_site.py fails the build on placeholder text, a missing
canonical, description, x-default hreflang, og:image, twitter:card or
JSON-LD block, invalid JSON-LD, an untagged contact link, a logo that does not
appear exactly once, a dashboard page with no rendered figure, the reach estimate
shown without its caveat, a missing download or citation file, or any asset
loaded from a third-party origin.
File: .github/workflows/tests.yml
Runs pytest tests/ on every pull request and on every push to main.
Locally:
python3.14 -m venv .venv && . .venv/bin/activate
pip install --only-binary :all: --require-hashes -r requirements/tests.txt
pytest tests/ -vbeautifulsoup4, jinja2 and pyyaml are hard test dependencies. The latter two
exercise the actual renderer and immutable citation output.
Tests use Python 3.14, matching the dashboard runtime. PyYAML 6.0.3 has no
Python 3.15 wheel, which the binary-only installation requires.
beautifulsoup4 is required even for tests that never scrape:
tests/test_check_stars.py imports scripts/check-stars.py at module scope, so
without it pytest fails during collection.
To change a pinned version, regenerate the file rather than editing a hash by hand:
uv pip compile --generate-hashes --python-version=3.14 \
--output-file=requirements/tests.txt requirements/tests.inRenovate keeps both lock files current through its pip-compile manager, which
re-runs the command from the file header. That manager accepts only the
--option=value form and needs the .in file as input, so keep the header in
that shape.
- Create workflow in
.github/workflows/ - Add scripts to
scripts/if needed - Document in this README
- For detailed setup guides, add to
docs/
shared/site-shell/ vendors the common Netresearch header source. The renderer
uses its Python interface and publishes matching CSS, JavaScript and the
authentic symbol under assets/site-shell/. The early inline bootstrap applies
System, Light or Dark before styles load. navigation.js only updates chart
colours when the shared nr-theme-change event fires. Historical snapshot pages
receive the same header without regenerating their main content or citation data.
The browser gate exercises radio-keyboard selection, persistence, legacy and System reset, blocked storage, native NoJS navigation and short viewports.
To publish UI changes using the existing measured snapshot, skip only collection:
gh workflow run impact-dashboard.yml --repo netresearch/maint --ref main -f collect_metrics=falseThis manual mode seeds the data and historical pages from gh-pages, then runs
the same renderer, verification, accessibility gate and publication. It retains
the original collection timestamps and archived figures; it does not retrieve
fresh metrics. Without seeded data/latest.json, rendering fails and nothing
is published. The daily schedule and manual dispatches that omit the input keep
collecting metrics (collect_metrics defaults to true).