GH-50991: [CI] Enable sccache just like ccache on Docker-based builds by pitrou · Pull Request #50992 · apache/arrow

pitrou · 2026-08-25T15:56:35Z

Rationale for this change

Enable sccache in more Docker-based builds on Crossbow CI, so as to make jobs faster and consume less resources.

Are these changes tested?

Yes, by existing Crossbow CI runs.

Example results:

Crossbow build	before	after
test-conda-python-3.11-dask-latest	21 min	12 min
test-debian-c-glib	25 min	8 min
test-fedora-python	29 min	13 min
test-ubuntu-ruby	24 min	8 min
wheel-manylinux-2-28-cp314-cp314-arm64	20 min	14 min

Are there any user-facing changes?

No.

GitHub Issue: [CI][C++] sccache not detected on all builds #50991

pitrou · 2026-08-25T15:56:46Z

@github-actions crossbow submit -g python

pitrou · 2026-08-25T15:57:07Z

@github-actions crossbow submit wheellinuxcp314

github-actions · 2026-08-25T15:57:43Z

⚠️ GitHub issue #50991 has been automatically assigned in GitHub to PR creator.

github-actions · 2026-08-25T16:00:11Z

Revision: ef5090a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1826ba0836

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hdfs-2.9.2
test-conda-python-3.11-hdfs-3.2.1
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-2.2.2-numpy-2.0.2
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-latest-numpy-2.1.3
test-conda-python-3.13-pandas-latest-numpy-latest
test-conda-python-3.14
test-conda-python-3.14-cpython-debug
test-conda-python-3.14-pandas-nightly-numpy-nightly
test-conda-python-3.14-pandas-upstream_devel-numpy-nightly
test-conda-python-emscripten
test-debian-13-python-3-amd64
test-debian-13-python-3-i386
test-fedora-42-python-3
test-ubuntu-24.04-python-3

pitrou · 2026-08-25T16:31:35Z

@github-actions crossbow submit wheelmanylinuxcp314

pitrou · 2026-08-25T16:36:42Z

@github-actions crossbow submit -g ruby

github-actions · 2026-08-25T16:39:02Z

Revision: 3561c77

Submitted crossbow builds: ursacomputing/crossbow @ actions-cada341cec

Task	Status
test-debian-ruby
test-ubuntu-ruby

pitrou · 2026-08-25T16:41:38Z

@github-actions crossbow submit wheelmanylinuxcp314

pitrou · 2026-08-25T16:45:39Z

@github-actions crossbow submit wheelmanylinuxcp314

pitrou · 2026-08-25T16:52:16Z

@github-actions crossbow submit wheelmanylinuxcp314

pitrou · 2026-08-25T16:55:38Z

@github-actions crossbow submit wheelmanylinuxcp314

github-actions · 2026-08-25T16:56:28Z

Revision: b6c60bc

Submitted crossbow builds: ursacomputing/crossbow @ actions-505a98c871

Task	Status
wheel-manylinux-2-28-cp314-cp314-amd64
wheel-manylinux-2-28-cp314-cp314-arm64
wheel-manylinux-2-28-cp314-cp314t-amd64
wheel-manylinux-2-28-cp314-cp314t-arm64

github-actions · 2026-08-25T16:59:15Z

Revision: 14269a9

Submitted crossbow builds: ursacomputing/crossbow @ actions-896a507684

Task	Status
wheel-manylinux-2-28-cp314-cp314-amd64
wheel-manylinux-2-28-cp314-cp314-arm64
wheel-manylinux-2-28-cp314-cp314t-amd64
wheel-manylinux-2-28-cp314-cp314t-arm64

pitrou · 2026-08-25T17:10:38Z

@github-actions crossbow submit wheellinuxcp314

github-actions · 2026-08-25T17:15:27Z

Revision: b398bf0

Submitted crossbow builds: ursacomputing/crossbow @ actions-853d2ba796

Task	Status
wheel-manylinux-2-28-cp314-cp314-amd64
wheel-manylinux-2-28-cp314-cp314-arm64
wheel-manylinux-2-28-cp314-cp314t-amd64
wheel-manylinux-2-28-cp314-cp314t-arm64
wheel-musllinux-1-2-cp314-cp314-amd64
wheel-musllinux-1-2-cp314-cp314-arm64
wheel-musllinux-1-2-cp314-cp314t-amd64
wheel-musllinux-1-2-cp314-cp314t-arm64

paultiq · 2026-08-25T17:25:37Z

FYI, I have this (wheels building with sccache) working on a branch... but the branch is a bit messy (I used OIDC permissions instead of akey/skey).

In case it's helpful: a few things that might be helpful, see:

SCCACHE_BASEDIR is really important, otherwise the random build dir will pollute the sccache key (lots of misses)
Added the following in python\CMakeLists.txt. This is longer than it needs to be, since you just need to check for SCCACHE_BUCKET, I guess.

find_program(SCCACHE_FOUND sccache)

if(NOT SCCACHE_FOUND AND DEFINED ENV{SCCACHE_PATH})
  message(STATUS "Did not find sccache, using envvar fallback.")
  set(SCCACHE_FOUND $ENV{SCCACHE_PATH})
endif()

# Only use sccache if a storage backend is configured
if(SCCACHE_FOUND
   AND NOT CMAKE_C_COMPILER_LAUNCHER
   AND NOT CMAKE_CXX_COMPILER_LAUNCHER
   AND (DEFINED ENV{SCCACHE_AZURE_BLOB_CONTAINER}
        OR DEFINED ENV{SCCACHE_BUCKET}
        OR DEFINED ENV{SCCACHE_DIR}
        OR DEFINED ENV{SCCACHE_GCS_BUCKET}
        OR DEFINED ENV{SCCACHE_MEMCACHED}
        OR DEFINED ENV{SCCACHE_REDIS}
       ))
  message(STATUS "Using sccache: ${SCCACHE_FOUND}")
  set(CMAKE_C_COMPILER_LAUNCHER ${SCCACHE_FOUND})
  set(CMAKE_CXX_COMPILER_LAUNCHER ${SCCACHE_FOUND})
endif()

And, other things I did:

explicitly disabled ARROW_USE_CCACHE
Uploaded the sccache logs, helpful for diagnosing cache misses (nearly impossible without)
Split the "build image" from "build wheel": this was just helpful to not pollute the build wheel time with the image build / docker pull.

There's a few things where I used OIDC to be ignored: AWS_SESSION_TOKEN and AWS_ROLE_ARN can be ignored,

pitrou · 2026-08-25T18:11:46Z

@github-actions crossbow submit -g ruby

pitrou · 2026-08-25T18:14:56Z

@github-actions crossbow submit -g r

github-actions · 2026-08-25T18:15:41Z

Revision: 37d53be

Submitted crossbow builds: ursacomputing/crossbow @ actions-4c426b805e

Task	Status
test-debian-ruby
test-ubuntu-ruby

pitrou · 2026-08-25T19:23:16Z

@kou Would you like to take a look?

Also cc @rok

rok · 2026-08-25T21:54:29Z

    ulimits: *ulimits
    environment:
-      <<: [*common, *ccache, *cpp]
+      <<: [*common, *ccache, *sccache, *cpp]


I didn't manage to check if this tracks, but my AI reviewer wrote:

ubuntu-cpp-bundled-offline sets ARROW_OFFLINE=ON. In ci/scripts/cpp_build.sh, this clears /etc/resolv.conf before compilation. With *sccache added, Crossbow supplies SCCACHE_BUCKET and CMake selects remote sccache instead of local ccache.

On its first compiler invocation, sccache verifies the S3 backend. With networking disabled, server startup fails and compilation exits rather than falling back. I reproduced this using sccache 0.12.0 against an unreachable S3 endpoint.

Hmm, let's try it out then.

Ok, you're right, it seems that sccache breaks under this scenario.

sccache: error: Server startup failed: cache storage failed to read: Unexpected (temporary) at read => send http request Context: url: https://s3.***.amazonaws.com/***/sccache/.sccache_check called: http_util::Client::send service: s3 path: .sccache_check range: 0- Source: error sending request for url (https://s3.***.amazonaws.com/***/sccache/.sccache_check): client error (Connect): dns error: failed to lookup address information: Try again: failed to lookup address information: Try again

Huh, interesting.

rok · 2026-08-25T21:58:00Z

      secrets: *vcpkg-build-secrets
    environment:
-      <<: [*common, *ccache]
+      <<: [*common, *ccache, *sccache]


I can only see ccache installed in ci/docker/cpp-jni.dockerfile, but no sccache, so this likely doesn't do much.

rok · 2026-08-25T21:59:16Z

Thank you for your (s)(c)cache crusade @pitrou !

kou

+1

pitrou · 2026-08-26T08:11:30Z

@github-actions crossbow submit -g cpp

github-actions · 2026-08-26T08:14:30Z

Revision: f5df088

Submitted crossbow builds: ursacomputing/crossbow @ actions-2cbd17ff51

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

…builds

…ready installed by base image This reverts commit 37d53be.

pitrou · 2026-08-26T08:26:57Z

@github-actions crossbow submit test-ubuntu-24.04-cpp-bundled-offline

github-actions · 2026-08-26T08:30:32Z

Revision: 3e9e9ef

Submitted crossbow builds: ursacomputing/crossbow @ actions-6820ae6774

Task	Status
test-ubuntu-24.04-cpp-bundled-offline

github-actions Bot added the awaiting review Awaiting review label Aug 25, 2026

This comment was marked as outdated.

Sign in to view

pitrou force-pushed the gh50991-sccache branch from 3561c77 to 234136c Compare August 25, 2026 16:45

This comment was marked as outdated.

Sign in to view

pitrou force-pushed the gh50991-sccache branch from 234136c to b6c60bc Compare August 25, 2026 16:52

This comment was marked as outdated.

Sign in to view

pitrou force-pushed the gh50991-sccache branch from b6c60bc to 14269a9 Compare August 25, 2026 16:55

pitrou mentioned this pull request Aug 25, 2026

[CI][Python] CMakeLists detects ccache but not sccache #50996

Open

pitrou force-pushed the gh50991-sccache branch from 14269a9 to b398bf0 Compare August 25, 2026 17:10

This comment was marked as outdated.

Sign in to view

pitrou marked this pull request as ready for review August 25, 2026 19:22

pitrou requested review from assignUser, jonkeane, kou and raulcd as code owners August 25, 2026 19:22

rok reviewed Aug 25, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 25, 2026

rok reviewed Aug 25, 2026

View reviewed changes

kou approved these changes Aug 26, 2026

View reviewed changes

github-actions Bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Aug 26, 2026

pitrou added 5 commits August 26, 2026 10:20

apacheGH-50991: [CI] Enable sccache just like ccache on Docker-based …

2dddde5

…builds

Try to enable sccache on manylinux

df7c479

Also install sccache on Debian-based R builds

5a5686a

Revert "Also install sccache on Debian-based R builds": sccache is al…

20df7af

…ready installed by base image This reverts commit 37d53be.

Fix offline build

3e9e9ef

pitrou force-pushed the gh50991-sccache branch from f5df088 to 3e9e9ef Compare August 26, 2026 08:26

github-actions Bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Aug 26, 2026

pitrou merged commit bb76583 into apache:main Aug 26, 2026
56 of 58 checks passed

pitrou removed the awaiting changes Awaiting changes label Aug 26, 2026

pitrou mentioned this pull request Aug 26, 2026

[CI][C++] sccache not detected on all builds #50991

Closed

pitrou deleted the gh50991-sccache branch August 26, 2026 09:21

Uh oh!

Conversation

pitrou commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

This comment was marked as outdated.

pitrou commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

This comment was marked as outdated.

github-actions Bot commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

This comment was marked as outdated.

pitrou commented Aug 25, 2026

Uh oh!

This comment was marked as outdated.

pitrou commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

paultiq commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

pitrou commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

This comment was marked as outdated.

pitrou commented Aug 25, 2026

Uh oh!

rok Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

rok Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

rok Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

rok commented Aug 25, 2026

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Aug 26, 2026

Uh oh!

github-actions Bot commented Aug 26, 2026

Uh oh!

pitrou commented Aug 26, 2026

Uh oh!

github-actions Bot commented Aug 26, 2026

pitrou commented Aug 25, 2026 •

edited

Loading