Skip to content

Support private GitHub release assets as SDP archives - #169

Merged
nradakovic merged 2 commits into
mainfrom
github-release-asset-archive
Oct 28, 2026
Merged

nradakovic merged 2 commits into
mainfrom
github-release-asset-archive

Conversation

@4og

@4og 4og commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Why

We want to host a QNX SDP as a release asset of a private GitHub repository and consume it via gcc.sdp(url = ...). Private assets can only be downloaded through the REST API:

GET https://api.github.com/repos/<owner>/<repo>/releases/assets/<id>
Authorization: Bearer <token>
Accept: application/octet-stream

GitHub returns the archive (via a 302 to a short-lived signed URL) only if application/octet-stream is the sole Accept value. Otherwise it returns the asset's JSON metadata.

Bazel 8 cannot do this:

  • HttpConnector always adds Accept: text/html, image/gif, image/jpeg, */*, and headers from a credential helper are added to it, not replacing it. GitHub then sends the JSON metadata, and the fetch fails with a checksum mismatch (Checksum was <sha of ~640 byte JSON> but wanted <sha of archive>).
  • A credential helper cannot work around this. Bazel only reads headers and expires from its response, so it cannot remove the default header or change the URL.
  • The browser URL https://github.com/<owner>/<repo>/releases/download/<tag>/<file> does not accept API tokens for private repositories (it returns 404).
  • The downloader was fixed in Bazel 9 (bazelbuild/bazel@80d531eb7ce2). That fix is not in any 8.x release, and our consumers cannot move to Bazel 9 yet.

What

  • New http_archive wrapper (rules/http_archive.bzl) with the same arguments the gcc extension passes today. It decides which repository rule to call: github_release_archive if urls is a single https://api.github.com/repos/<owner>/<repo>/releases/assets/<id> URL, the native http_archive otherwise. The only change in extensions/gcc.bzl is its load statement, so the workaround acts as a plug-in and can be removed in one place once Bazel 9 is required.
  • New github_release_archive repository rule (rules/github_release_archive.bzl) that does the actual fetch.
  • The gcc.sdp API is unchanged. It accepts the same attributes (build_file, patches, patch_*, files, strip_prefix, ...). For these URLs, sha256 and type are mandatory.
  • On a repository cache miss, tools/fetch_github_release_asset.py resolves the signed URL and downloads the asset. Bazel then ingests it through rctx.download(url = "file://...", sha256 = ..., canonical_id = ...), so:
    • the checksum is verified by Bazel;
    • the archive lands in the repository cache, so later fetches (also after bazel clean --expunge) need no token and no network;
    • the signed URL (a bearer credential) never reaches Bazel, so it does not show up in console output or the BEP.
  • The token is read by the script from SCORE_GITHUB_TOKEN / GITHUB_TOKEN / GH_TOKEN or from netrc (api.github.com or github.com). It is only sent to api.github.com and never to the redirect target. It is not part of the repository definition, so rotating it does not trigger a refetch.
  • Docs: docs/extension_api.md, docs/qnx_integration.md.
  • tests/MODULE.bazel.lock was regenerated. Only the extension digest and attribute order changed.

Once all consumers use Bazel 9, this rule can be dropped and http_archive with a credential helper used instead.

Testing

Tested in eclipse-score/baselibs with Bazel 8.6.0 and a local_path_override, using gcc.sdp(url = "https://api.github.com/repos/.../releases/assets/<id>", type = "tar.gz", ...) (~1.7 GB SDP), bazel build --config=score-qnx-x86_64 -- //score/bitmanipulation/...:

  • Cold fetch: build passes and produces a QNX ELF binary.
  • Console log and --build_event_json_file contain no signed URL. The only fetch URL for the SDP is the local file: path.
  • bazel clean --expunge, then a build with no token: served from the repository cache, so it passes.
  • Empty repository cache and no token: fails with GitHub API returned HTTP 404. Check that a token with read access is available.
  • tests/: --config=x86_64-linux build of //:language_and_standards_tests //:feature_verification_tests and //:guardrail_tests pass, so the http_archive path is unchanged. bazel mod show_repo confirms the SDP repository is a github_release_archive and the GCC packages are still native http_archive.
  • //:format.check, //:copyright.check

Bazel 8 cannot download GitHub release assets through the REST API
(https://api.github.com/repos/<owner>/<repo>/releases/assets/<id>), which
is the only way to fetch assets of private repositories. Its downloader
always appends a browser-like Accept header, so GitHub answers with the
JSON asset metadata instead of the archive and the checksum check fails.
A credential helper cannot fix this, because it can only add headers.
The downloader was fixed in Bazel 9 (bazelbuild/bazel@80d531eb7ce2).

Add a github_release_archive repository rule that the gcc extension uses
for such URLs instead of http_archive. The gcc.sdp API is unchanged. On a
repository cache miss, tools/fetch_github_release_asset.py downloads the
asset and the rule hands it to Bazel as a local file, so the checksum
check and the repository cache still apply. The short-lived signed
download URL never reaches Bazel and therefore never shows up in the
console output or the Build Event Protocol.

Regenerate tests/MODULE.bazel.lock for the changed extension.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the GitHub release asset workaround out of the gcc extension. The
new rules/http_archive.bzl provides an http_archive with the same
arguments the extension uses and decides which repository rule to call:
github_release_archive for GitHub REST API release asset URLs, the native
http_archive otherwise. The extension only changes its load statement, so
the workaround can be removed in one place once Bazel 9 is required.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nradakovic
nradakovic merged commit ee97811 into main Oct 28, 2026
13 of 15 checks passed
@nradakovic
nradakovic deleted the github-release-asset-archive branch October 28, 2026 09:09
4og added a commit to eclipse-score/baselibs that referenced this pull request Oct 30, 2026
The QNX SDP 8.0.5 is published as an asset of a private GitHub release
(eclipse-score/qnx_sdp_pkg). Bazel 8 cannot download such assets, so the
SDP is fetched through the http_archive wrapper of score_bazel_cpp_toolchains
(eclipse-score/bazel_cpp_toolchains#169).

Add score-qnx-{x86_64,aarch64}-sdp_8.0.5 configs that inherit the existing
QNX configs and register the 8.0.5 qcc and IFS toolchains last, so they take
precedence. The default configs and local development are unchanged and do
not fetch the private SDP.

Run the QNX coverage workflow with the new SDP on pull_request instead of
pull_request_target and without the GitHub Actions Bazel cache.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants