[BUG]: Program.compile("ptx") fails on machines without a CUDA driver, although NVRTC does not need one

[VolodymyrLinuxovich]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

On a Linux machine with no NVIDIA driver installed (a GitHub Actions ubuntu runner), compiling a Program to PTX with the NVRTC backend raises DynamicLibNotFoundError for libcuda.so.1. Compiling the same source to cubin with an explicit arch works on the same machine, so NVRTC itself is available.

The failure comes from the loadability check that runs before PTX compilation. _can_load_generated_ptx() calls driver_version(), which calls cuDriverGetVersion and needs libcuda. The check only decides whether to emit a RuntimeWarning, but when the driver library is missing it raises instead, and the whole compile fails. The same check is on the cache hit path in Program.compile.

This matters for CI jobs that compile kernels to PTX without a GPU, for example to inspect the generated PTX in tests. My workaround was to call cuda.bindings.nvrtc directly with the flags from ProgramOptions.as_bytes("nvrtc", "ptx"), which works without a driver.

How to Reproduce

On a machine without an NVIDIA driver. I used the ubuntu latest runner on GitHub Actions with Python 3.12.

pip install "cuda-core[cu12]==1.2.1"

from cuda.core import Program, ProgramOptions

src = 'extern "C" __global__ void k(float* x) { x[0] = 2.0f * x[0] + 1.0f; }'

# Works without a driver
Program(src, code_type="c++", options=ProgramOptions(arch="sm_75")).compile("cubin")

# Fails without a driver
Program(src, code_type="c++", options=ProgramOptions(arch="compute_75")).compile("ptx")

A public run on a runner with no libcuda shows both cases, the cubin compile passes and the ptx compile fails. https://github.com/VolodymyrLinuxovich/heat-risk-cuda/actions/runs/37217831804

Trimmed traceback from the failing run

cuda/core/_program.pyx:222: in cuda.core._program.Program.compile
cuda/core/_program.pyx:738: in cuda.core._program._program_compile_uncached
cuda/core/_program.pyx:1192: in cuda.core._program.Program_compile
cuda/core/_program.pyx:815: in cuda.core._program._can_load_generated_ptx
cuda/core/_utils/version.pyx:37: in cuda.core._utils.version.driver_version
cuda/bindings/driver.pyx:23154: in cuda.bindings.driver.cuDriverGetVersion
...
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: "cuda" is an NVIDIA driver library and can only be found via system search. Ensure the NVIDIA display driver is installed.

The _can_load_generated_ptx code on main looks the same as in 1.2.1, so I expect main behaves the same way, but I have only run 1.2.1.

Expected behavior

PTX compilation succeeds without a driver, the same as cubin compilation. When the driver version cannot be determined, the loadability check could skip the warning (or warn that loadability is unknown) instead of failing the compile.

I would be glad to open a PR for this if that approach sounds right to you.

Operating System

Ubuntu (GitHub Actions ubuntu latest runner), no NVIDIA driver

nvidia-smi output

Not available, no driver on that machine.

[efegokdemir]

I’d like to work on this issue. Could a maintainer assign it to me before I start, per the contribution guide?

[VolodymyrLinuxovich]

I’d like to work on this issue. Could a maintainer assign it to me before I start, per the contribution guide?

Thanks for the interest. I offered a fix in the report and would be glad to open the PR. I hit this in my own CI, so I can verify the fix there. Happy to leave it to the maintainers to decide.

[leofang]

This is not a bug. NVRTC happens to work today without driver installed, but it is not a documented behavior that the NVRTC team guarantees. Also, cuda.core could choose to fall back to the driver for JIT-compiling PTX, and in order for us to decide if fallback is needed, we need the driver to be present in the first place.

In general, ALL cuda.core functionalities require a working NVIDIA GPU, necessary CUDA Toolkit libraries, and driver properly installed. The only guarantee we offer on CPU-only machines is that from cuda import core works.

For the specific use case, I recommend all JIT-compilation users to generate CUBIN instead of PTX. It'd offer you better support for CUDA Minor Version Compatibility.

[VolodymyrLinuxovich]

This is not a bug. NVRTC happens to work today without driver installed, but it is not a documented behavior that the NVRTC team guarantees. Also, cuda.core could choose to fall back to the driver for JIT-compiling PTX, and in order for us to decide if fallback is needed, we need the driver to be present in the first place.

In general, ALL cuda.core functionalities require a working NVIDIA GPU, necessary CUDA Toolkit libraries, and driver properly installed. The only guarantee we offer on CPU-only machines is that from cuda import core works.

For the specific use case, I recommend all JIT-compilation users to generate CUBIN instead of PTX. It'd offer you better support for CUDA Minor Version Compatibility.

Thanks for explaining. That makes sense, and I switched my CI to compile cubin instead.