Skip to content

GH-50991: [CI] Enable sccache just like ccache on Docker-based builds - #50992

Open
pitrou wants to merge 4 commits into
apache:mainfrom
pitrou:gh50991-sccache
Open

GH-50991: [CI] Enable sccache just like ccache on Docker-based builds#50992
pitrou wants to merge 4 commits into
apache:mainfrom
pitrou:gh50991-sccache

Conversation

@pitrou

@pitrou pitrou commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Rationale for this change

Enable sccache in more Docker-based builds on Crossbow CI, so as to make jobs faster and consume less resources.

Are these changes tested?

Yes, by existing Crossbow CI runs.

Example results:

Crossbow build before after
test-conda-python-3.11-dask-latest 21 min 12 min
test-debian-c-glib 25 min 8 min
test-fedora-python 29 min 13 min
test-ubuntu-ruby 24 min 8 min
wheel-manylinux-2-28-cp314-cp314-arm64 20 min 14 min

Are there any user-facing changes?

No.

@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g python

@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheellinuxcp314

@github-actions github-actions Bot added the awaiting review Awaiting review label Aug 25, 2026
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

⚠️ GitHub issue #50991 has been automatically assigned in GitHub to PR creator.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

Revision: ef5090a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1826ba0836

Task Status
example-python-minimal-build-fedora-conda GitHub Actions
example-python-minimal-build-ubuntu-venv GitHub Actions
test-conda-python-3.11 GitHub Actions
test-conda-python-3.11-dask-latest GitHub Actions
test-conda-python-3.11-dask-upstream_devel GitHub Actions
test-conda-python-3.11-hdfs-2.9.2 GitHub Actions
test-conda-python-3.11-hdfs-3.2.1 GitHub Actions
test-conda-python-3.11-hypothesis GitHub Actions
test-conda-python-3.11-pandas-2.2.2-numpy-2.0.2 GitHub Actions
test-conda-python-3.11-spark-master GitHub Actions
test-conda-python-3.12 GitHub Actions
test-conda-python-3.12-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.13 GitHub Actions
test-conda-python-3.13-pandas-latest-numpy-2.1.3 GitHub Actions
test-conda-python-3.13-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.14 GitHub Actions
test-conda-python-3.14-cpython-debug GitHub Actions
test-conda-python-3.14-pandas-nightly-numpy-nightly GitHub Actions
test-conda-python-3.14-pandas-upstream_devel-numpy-nightly GitHub Actions
test-conda-python-emscripten GitHub Actions
test-debian-13-python-3-amd64 GitHub Actions
test-debian-13-python-3-i386 GitHub Actions
test-fedora-42-python-3 GitHub Actions
test-ubuntu-24.04-python-3 GitHub Actions

@github-actions

This comment was marked as outdated.

Sign in to view
@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheelmanylinuxcp314

@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g ruby

@github-actions

This comment was marked as outdated.

Sign in to view
@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

Revision: 3561c77

Submitted crossbow builds: ursacomputing/crossbow @ actions-cada341cec

Task Status
test-debian-ruby GitHub Actions
test-ubuntu-ruby GitHub Actions

@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheelmanylinuxcp314

@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheelmanylinuxcp314

@github-actions

This comment was marked as outdated.

Sign in to view
@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheelmanylinuxcp314

@github-actions

This comment was marked as outdated.

Sign in to view
@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheelmanylinuxcp314

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

Revision: b6c60bc

Submitted crossbow builds: ursacomputing/crossbow @ actions-505a98c871

Task Status
wheel-manylinux-2-28-cp314-cp314-amd64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314-arm64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314t-amd64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314t-arm64 GitHub Actions

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

Revision: 14269a9

Submitted crossbow builds: ursacomputing/crossbow @ actions-896a507684

Task Status
wheel-manylinux-2-28-cp314-cp314-amd64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314-arm64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314t-amd64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314t-arm64 GitHub Actions

@pitrou pitrou mentioned this pull request Aug 25, 2026
Open
@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheellinuxcp314

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

Revision: b398bf0

Submitted crossbow builds: ursacomputing/crossbow @ actions-853d2ba796

Task Status
wheel-manylinux-2-28-cp314-cp314-amd64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314-arm64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314t-amd64 GitHub Actions
wheel-manylinux-2-28-cp314-cp314t-arm64 GitHub Actions
wheel-musllinux-1-2-cp314-cp314-amd64 GitHub Actions
wheel-musllinux-1-2-cp314-cp314-arm64 GitHub Actions
wheel-musllinux-1-2-cp314-cp314t-amd64 GitHub Actions
wheel-musllinux-1-2-cp314-cp314t-arm64 GitHub Actions

@paultiq

paultiq commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

FYI, I have this (wheels building with sccache) working on a branch... but the branch is a bit messy (I used OIDC permissions instead of akey/skey).

In case it's helpful: a few things that might be helpful, see:

  • SCCACHE_BASEDIR is really important, otherwise the random build dir will pollute the sccache key (lots of misses)
  • Added the following in python\CMakeLists.txt. This is longer than it needs to be, since you just need to check for SCCACHE_BUCKET, I guess.
find_program(SCCACHE_FOUND sccache)

if(NOT SCCACHE_FOUND AND DEFINED ENV{SCCACHE_PATH})
  message(STATUS "Did not find sccache, using envvar fallback.")
  set(SCCACHE_FOUND $ENV{SCCACHE_PATH})
endif()

# Only use sccache if a storage backend is configured
if(SCCACHE_FOUND
   AND NOT CMAKE_C_COMPILER_LAUNCHER
   AND NOT CMAKE_CXX_COMPILER_LAUNCHER
   AND (DEFINED ENV{SCCACHE_AZURE_BLOB_CONTAINER}
        OR DEFINED ENV{SCCACHE_BUCKET}
        OR DEFINED ENV{SCCACHE_DIR}
        OR DEFINED ENV{SCCACHE_GCS_BUCKET}
        OR DEFINED ENV{SCCACHE_MEMCACHED}
        OR DEFINED ENV{SCCACHE_REDIS}
       ))
  message(STATUS "Using sccache: ${SCCACHE_FOUND}")
  set(CMAKE_C_COMPILER_LAUNCHER ${SCCACHE_FOUND})
  set(CMAKE_CXX_COMPILER_LAUNCHER ${SCCACHE_FOUND})
endif()

And, other things I did:

  • explicitly disabled ARROW_USE_CCACHE
  • Uploaded the sccache logs, helpful for diagnosing cache misses (nearly impossible without)
  • Split the "build image" from "build wheel": this was just helpful to not pollute the build wheel time with the image build / docker pull.

There's a few things where I used OIDC to be ignored: AWS_SESSION_TOKEN and AWS_ROLE_ARN can be ignored,

@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g ruby

@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g r

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

Revision: 37d53be

Submitted crossbow builds: ursacomputing/crossbow @ actions-4c426b805e

Task Status
test-debian-ruby GitHub Actions
test-ubuntu-ruby GitHub Actions

@github-actions

This comment was marked as outdated.

Sign in to view
@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit -g c-glib

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown

Revision: 37d53be

Submitted crossbow builds: ursacomputing/crossbow @ actions-63fb582c95

Task Status
test-debian-c-glib GitHub Actions
test-ubuntu-c-glib GitHub Actions

@pitrou
Revert "Also install sccache on Debian-based R builds": sccache is al…
f5df088 
…ready installed by base image

This reverts commit 37d53be.
@pitrou
pitrou marked this pull request as ready for review August 25, 2026 19:22
@pitrou

pitrou commented Aug 25, 2026

Copy link
Copy Markdown
Member Author

@kou Would you like to take a look?

Also cc @rok

rok
rok reviewed Aug 25, 2026
View reviewed changes
Comment thread compose.yaml
ulimits: *ulimits
environment:
<<: [*common, *ccache, *cpp]
<<: [*common, *ccache, *sccache, *cpp]

@rok rok Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't manage to check if this tracks, but my AI reviewer wrote:

ubuntu-cpp-bundled-offline sets ARROW_OFFLINE=ON. In ci/scripts/cpp_build.sh, this clears /etc/resolv.conf before compilation. With *sccache added, Crossbow supplies SCCACHE_BUCKET and CMake selects remote sccache instead of local ccache.

On its first compiler invocation, sccache verifies the S3 backend. With networking disabled, server startup fails and compilation exits rather than falling back. I reproduced this using sccache 0.12.0 against an unreachable S3 endpoint.

@github-actions github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 25, 2026
rok
rok reviewed Aug 25, 2026
View reviewed changes
Comment thread compose.yaml
secrets: *vcpkg-build-secrets
environment:
<<: [*common, *ccache]
<<: [*common, *ccache, *sccache]

@rok rok Aug 25, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can only see ccache installed in ci/docker/cpp-jni.dockerfile, but no sccache, so this likely doesn't do much.

@rok

rok commented Aug 25, 2026

Copy link
Copy Markdown
Member

Thank you for your (s)(c)cache crusade @pitrou !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rok rok rok left review comments
@assignUser assignUser Awaiting requested review from assignUser assignUser is a code owner
@jonkeane jonkeane Awaiting requested review from jonkeane jonkeane is a code owner
@kou kou Awaiting requested review from kou kou is a code owner
@raulcd raulcd Awaiting requested review from raulcd raulcd is a code owner

Assignees

No one assigned

Labels

awaiting changes Awaiting changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pitrou @paultiq @rok