ci: build and test only changed packages and dependants#9373
Draft
Mrtenz wants to merge 16 commits into
Draft
Conversation
Instead of rebuilding all 89 packages on every CI run, use `yarn workspaces foreach --since` to build only the packages that changed relative to the target branch, plus their transitive dependants. Falls back to a full `yarn build` on push-to-main events where no base branch ref is available.
`--depth=1` on the base branch fetch doesn't give git enough history to find the common ancestor with the PR branch, causing yarn's `--since` to fail. Replace with `--no-tags` (full history, no tags).
The `action-checkout-and-setup` does a `fetch-depth: 1` shallow clone, so the PR branch history only goes back one commit. Git cannot find the merge base against `origin/$BASE_REF` from such a shallow history, causing yarn's `--since` to fail with "No ancestor could be found". Unshallow the checkout first, so the full history is available for the merge-base computation.
Instead of heuristic depth fetches, call the GitHub Compare API to get the exact merge base SHA, then fetch only that single commit. This avoids fetching large chunks of history while being precise regardless of how old the branch is.
The previous approach fetched the merge base commit but git still couldn't confirm it was an ancestor of HEAD because the PR branch checkout is depth=1. Fix by unshallowing the checkout using `--filter=blob:none`, which fetches full commit and tree history without downloading file content — sufficient for `git diff --name-only`.
Without an explicit refspec, `git fetch --unshallow` deepens all refs that were fetched during checkout. Passing `origin HEAD` limits it to the currently checked-out ref.
Instead of running `yarn workspaces foreach --since` sequentially, generate a temporary tsconfig that lists only changed packages and their transitive dependants as project references. This lets ts-bridge use TypeScript's project-references to parallelise the build.
mktemp creates the file in /tmp by default, causing ts-bridge to resolve relative package paths against /tmp instead of the repo root. Using --tmpdir="$GITHUB_WORKSPACE" and cleaning up with rm -f after the build keeps relative paths correct without leaving a dirty tree.
…dencies Three bugs: - actions/checkout checks out the merge commit (PR + base), so `git diff $MERGE_BASE...HEAD` included all of main's changes. Fix: pass the explicit PR head SHA and diff against that instead. - Packages included as dependants of changed packages couldn't build because their own dependencies had no dist files. Fix: expand the build set to include transitive dependencies as well. - ts-bridge rejects a tsconfig with `files: []` and no references. Fix: skip ts-bridge entirely when no packages need building.
Extract shared workspace logic into scripts/lib/workspaces.mts and add scripts/get-changed-workspaces.mts. A new get-changed-packages CI job fetches the merge base once and computes which packages changed; test-18/20/22, validate-changelog, and build all consume its outputs rather than computing independently. The build job no longer calls the GitHub Compare API itself — it reads the merge base from get-changed-packages and only needs to unshallow its own checkout.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+92
to
+141
| get-changed-packages: | ||
| name: Get changed packages | ||
| runs-on: ubuntu-latest | ||
| needs: prepare | ||
| outputs: | ||
| merge-base: ${{ steps.fetch-merge-base.outputs.merge-base }} | ||
| package-names: ${{ steps.packages.outputs.package-names }} | ||
| steps: | ||
| - name: Checkout and setup environment | ||
| uses: MetaMask/action-checkout-and-setup@v3 | ||
| with: | ||
| is-high-risk-environment: false | ||
| persist-credentials: false | ||
| node-version: 24.x | ||
| - name: Fetch merge base | ||
| id: fetch-merge-base | ||
| if: github.base_ref != '' || github.event.merge_group.base_ref != '' | ||
| run: | | ||
| set -euo pipefail | ||
|
|
||
| PREFIXED_REF_REGEX='refs/heads/(.+)' | ||
| if [[ "$BASE_REF" =~ $PREFIXED_REF_REGEX ]]; then | ||
| BASE_REF="${BASH_REMATCH[1]}" | ||
| fi | ||
|
|
||
| MERGE_BASE=$(gh api "repos/$GITHUB_REPOSITORY/compare/$BASE_REF...$HEAD_SHA" --jq '.merge_base_commit.sha') | ||
| git fetch --unshallow --filter=blob:none --no-tags origin HEAD | ||
|
|
||
| echo "merge-base=$MERGE_BASE" >> "$GITHUB_OUTPUT" | ||
| env: | ||
| BASE_REF: ${{ github.event.pull_request.base.ref || github.event.merge_group.base_ref }} | ||
| HEAD_SHA: ${{ github.event.pull_request.head.sha || github.event.merge_group.head_sha }} | ||
| GH_TOKEN: ${{ github.token }} | ||
| - name: Get changed package names | ||
| id: packages | ||
| run: | | ||
| if [[ -n "$MERGE_BASE" ]]; then | ||
| PACKAGES=$(yarn tsx scripts/get-changed-workspaces.mts "$MERGE_BASE" "$HEAD_SHA") | ||
| # Fall back to all packages if no packages changed (e.g. CI-only PRs), | ||
| # to ensure required status checks are not skipped. | ||
| if [[ "$PACKAGES" == "[]" ]]; then | ||
| PACKAGES=$(yarn workspaces list --no-private --json | jq --slurp --raw-output 'map(.name) | @json') | ||
| fi | ||
| else | ||
| PACKAGES=$(yarn workspaces list --no-private --json | jq --slurp --raw-output 'map(.name) | @json') | ||
| fi | ||
| echo "package-names=$PACKAGES" >> "$GITHUB_OUTPUT" | ||
| env: | ||
| MERGE_BASE: ${{ steps.fetch-merge-base.outputs.merge-base }} | ||
| HEAD_SHA: ${{ github.event.pull_request.head.sha || github.event.merge_group.head_sha }} |
CI-only PRs (e.g. changes only to workflow files) would previously fall
back to running tests and changelog validation for all packages. This was
meant to ensure required status checks were not skipped, but the only
required check is at the workflow level ("all jobs pass"), not at the
individual job level. An empty matrix safely skips those jobs without
blocking merges.
GitHub Actions errors with "Matrix vector does not contain any values" when a matrix input resolves to an empty array. Add an `if` condition to `validate-changelog`, `test-18`, `test-20`, and `test-22` so they are skipped entirely when `package-names` is `[]`.
This job builds the full wallet-cli dependency subtree, so it can't use the package matrix. Instead, skip it entirely when a merge base is available and `@metamask/wallet-cli` is not in the changed packages list.
Mrtenz
changed the title
If any changed file lives outside all package directories, rebuild and test everything. Root-level configs, workflow files, and scripts can all affect every package, so a full run is the safe default. A set of known-safe root files (yarn.lock, README.md, .gitignore, etc.) are excluded from this check since they don't affect package builds or tests.
getAllWorkspaces was using `yarn workspaces list` without `--no-private`, so private packages (e.g. wallet-framework-docs) were included in the test matrix. The prepare job always used `--no-private`, so these were never tested before and have no working test scripts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Explanation
Currently, every CI run rebuilds all 89 packages from scratch via
yarn build(which callsts-bridgeon the roottsconfig.build.json). This is always a cold full build regardless of how many packages actually changed.This PR replaces the unconditional
yarn buildwithyarn workspaces foreach --since="origin/$BASE_REF" --topological --recursive --no-private run build, which:A new "Fetch base branch" step fetches the target branch at depth 1 before running the build, so the
--sinceGit comparison has a ref to diff against. Formerge_groupevents therefs/heads/prefix is stripped from the base ref (reusing the same pattern as.github/actions/check-release).For push-to-main events (where there is no base branch ref), the job falls back to the existing
yarn build.Checklist