Segfault on NVIDIA, but fine on iGPU

[tombh]

I've got a shader that works fine on my integrated GPU but segfaults on NVIDIA. I use wgpu's unsafe wgpu::Features::EXPERIMENTAL_PASSTHROUGH_SHADERS so that the SPIRV is passed directly to the GPU without transpilation to WGSL.

The shader itself is here https://github.com/tombh/total-viewsheds/blob/main/crates/kernels/vulkan-and-cpu/src/kernel.rs I feel like it could be the use of enum. I've got a gdb trace which suggest the segfault is triggered by libnvidia-glvkspirv.so. Does anybody have any more ideas for debugging this?

(gdb) bt
#0  0x00007fffe0991e32 in ?? () from /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.575.57.08
#1  0x00007fffe099948b in ?? () from /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.575.57.08
#2  0x00007fffe0999686 in ?? () from /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.575.57.08
#3  0x00007fffe0cc684f in ?? () from /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.575.57.08
#4  0x00007fffe09d07ad in ?? () from /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.575.57.08
#5  0x00007fffe09c2fb0 in ?? () from /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.575.57.08
#6  0x00007fffe09c8cd6 in _nv002nvvm () from /lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.575.57.08
#7  0x00007fffe36d0f39 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#8  0x00007fffe36d2a00 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#9  0x00007fffe36d3fef in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#10 0x00007fffe36d7281 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#11 0x00007fffe36d768b in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#12 0x00007fffe36d5c7d in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#13 0x00007fffe36d89c5 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#14 0x00007fffe36ba2a5 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#15 0x00007fffe36ba436 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#16 0x00007fffe35742f5 in ?? () from /lib/x86_64-linux-gnu/libnvidia-eglcore.so.575.57.08
#17 0x000055555648245b in ash::device::Device::create_compute_pipelines (self=0x555557ee4770, pipeline_cache=..., create_infos=..., allocation_callbacks=...)
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/ash-0.38.0+1.3.281/src/device.rs:2185
#18 0x00005555564391e0 in wgpu_hal::vulkan::device::{impl#4}::create_compute_pipeline (self=0x555557ee51c0, desc=0x7fffffff8898) at src/vulkan/device.rs:2304
#19 0x0000555556220ff7 in wgpu_hal::dynamic::device::{impl#0}::create_compute_pipeline<wgpu_hal::vulkan::Device> (self=0x555557ee51c0, desc=0x7fffffff9120)
    at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/wgpu-hal-27.0.2/src/dynamic/device.rs:437
#20 0x0000555556136dc0 in wgpu_core::device::resource::Device::create_compute_pipeline (self=0x7fffffff96b0, desc=...) at src/device/resource.rs:3590
#21 0x0000555556323e3d in wgpu_core::global::Global::device_create_compute_pipeline (self=0x5555571e6f30, device_id=..., desc=0x7fffffff9d40, id_in=...) at src/device/global.rs:1663
#22 0x000055555608eac6 in wgpu::backend::wgpu_core::{impl#12}::create_compute_pipeline (self=0x555557f2a1f0, desc=0x7fffffffb3a0) at src/backend/wgpu_core.rs:1506
#23 0x00005555560cc5b6 in wgpu::api::device::Device::create_compute_pipeline (self=0x7fffffffaae8, desc=0x7fffffffb3a0) at src/api/device.rs:263
#24 0x0000555555dc4fab in total_viewsheds::vulkan::Vulkan::new (constants=..., elevations=..., distances_count=1000000, band_deltas_count=333, total_reserved_rings=1) at crates/total-viewsheds/src/vulkan.rs:175
#25 0x0000555555e2ac36 in total_viewsheds::compute::Compute::new (backend=total_viewsheds::config::Backend::Vulkan, process=..., state_directory=..., maybe_output_directory=..., dem=0x7fffffffc978, rings_per_km=3,
    observer_height=1.64999998) at crates/total-viewsheds/src/compute.rs:84
#26 0x0000555555dce8e8 in total_viewsheds::compute (config=0x7fffffffdcb8) at crates/total-viewsheds/src/main.rs:117
#27 0x0000555555dcca83 in total_viewsheds::main () at crates/total-viewsheds/src/main.rs:41

[nazar-pc]

so that the SPIRV is passed directly to the GPU without transpilation to WGSL

Are you sure it is being transpiled though? Because I don't think it is unless you're running it in the browser.

My understanding is that wgpu will read whatever input you give it and produce whatever output the platform supports, but it will not convert things to and then back from WGSL for no reason. If you give it SPIR-V (which rust-gpu does), then run it on Vulkan-capable GPU, then your SPIR-V will be given to the GPU as is (or close to it).

[tombh]

Ohh, so wgpu just validates, not transpiles! That makes much more sense.

So yes I need some unique features, like i16 support and wgpu doesn't validate that. Hence why I'm using wgpu::Features::EXPERIMENTAL_PASSTHROUGH_SHADERS.

But still, we're left with the curiosity that my .spv runs fine on my iGPU but segfaults on NVIDIA. I've run spirv-val and it doesn't say anything. So maybe I've found a bug in the NVIDIA drivers?

[nazar-pc]

I just ended up rewriting my algorithms to pack everything into u32s. Debugging issues on GPU wasn't fun. Have you tried llvmpipe (usually installed by default on Linux) or SwiftShader? Maybe they'll give you some clues about what is going wrong. llvmpipe was quite picky from my experience and caught some interesting cases that AMD dGPU had no issues with.

[tombh]

It's just that I know from a CUDA implementation that the i16s are giving a significant speed up.

That's a good tip about llvmpipe. I actually use it to run CI tests, so I'll look into it.

[nazar-pc]

BTW you can open an issue about 16-bit types if there isn't one already. Both Apple Silicon Metal and Vulkan dGPUs support them, my understanding is that only Naga doesn't understand it just yet. I could see myself using it as well in some cases, but it wasn't critical for me yet.

[tombh]

So funny thing, I went to see if there was already an issue about this over at the wgpu repo, which there is. But who was the last person to leave a comment there? Me! gfx-rs/wgpu#4424 😅

[tombh]

I have a bit of an update, I didn't actually have the Vulkan Validation Layers installed. They're a separate package on Linux. And now I'm seeing a validation error:

2025-10-08T15:50:50.134532Z ERROR wgpu_hal::vulkan::instance: VALIDATION [VUID-VkShaderModuleCreateInfo-pCode-08740 (0x6e224e9)]
        vkCreateShaderModule(): SPIR-V Capability VulkanMemoryModel was declared, but one of the following requirements is required (VkPhysicalDeviceVulkan12Features::vulkanMemoryModel).
The Vulkan spec states: If pCode is a pointer to SPIR-V code, and pCode declares any of the capabilities listed in the SPIR-V Environment appendix, one of the corresponding requirements must be satisfied (https://docs.vulkan.org/spec/latest/chapters/shaders.html#VUID-VkShaderModuleCreateInfo-pCode-08740)

[LegNeato]

What is the target you are building with?

[tombh]

Just the default, which I think is spirv-unknown-vulkan1.2? When I set it to spirv-unknown-spv1.5 I get a new validation error:

VALIDATION [VUID-VkPhysicalDeviceProperties2-pNext-pNext (0xdd73dbcf)]
        Validation Error: [ VUID-VkPhysicalDeviceProperties2-pNext-pNext ] Object 0: 
        VK_NULL_HANDLE, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0xdd73dbcf | vkGetPhysicalDeviceProperties2KHR: 
        pProperties->pNext chain includes a structure with unknown VkStructureType (1000328001);

[Firestar99]

You can ignore the missing VulkanMemoryModel capability error, that is one of the very few you are allowed to ignore. It formalizes the memory operations and atomics on Vulkan and doesn't leave them as an undefined mess as they were originally, but doesn't actually add any functionality. It's just a guarantee from the driver that it respects that memory model.

[nazar-pc]

BTW I was getting crazy memory usage (crashed system once) and strange segfaults after installing Vulkan validation layer package on Ubuntu 24.04. Removed it and everything is fine again. Looks like that thing is buggy.

[tombh]

I found the problem. Here's the idea... if you have code like this:

#[spirv(compute(threads(8, 8, 4)))]
pub fn main(
    #[spirv(global_invocation_id)] id: glam::UVec3,
    #[spirv(uniform, descriptor_set = 0, binding = 0)] constants: &constants::Constants,
    #[spirv(storage_buffer, descriptor_set = 0, binding = 1)] elevations: &[f32],
    #[spirv(storage_buffer, descriptor_set = 0, binding = 2)] cumulative_surfaces: &mut [f32]
) {
    if constants.is_accumlate_surfaces {
        cumulative_surfaces[0] = 1.234;
    }
}

But don't define the cumulative_surfaces buffer in the pipeline, which in theory is fine because all access to it is gated with constants.is_accumlate_surfaces = false. Then this is fine on Mesa, but causes a segfault on Nvidia. I don't know if there were clues to this in the warnings and errors I got?

[nazar-pc]

This probably depends on API used, I believe wgpu checks this and refuses to run if one of the needed buffers is missing

[tombh]

That's a good point, but the weird thing is that it's an Nvidia driver issue because it works fine on Mesa.

[nazar-pc]

I guess you're in UB territory or something, in which case behavior between implementations is not guaranteed to be anything specific, even Mesa might change in the future versions.

[Firestar99]

I would have expected vulkan validation layers to complain about this before the driver saw the shader, if the pipelineLayout and shader mismatch...

[tombh]

Oh yes you're right. Now I can see that it's actually to do with a struct that mixes both references to buffers and local variables. I moved the buffers off the struct and it's all fine now.

[tombh]

I had a struct that kept references to buffers and owned local variables. Moving the buffers off the struct allowed the Nvidia version to run.

[Firestar99]

If you have a struct that contains both buffer references / slices and normal variables, rust-gpu should emit an illegal OpTypeStruct that makes spirv-val fail, which should fail the compile. Interestingly, structs full of buffer references works by the struct being DCE'ed, it's just mixing them that makes it fail. I don't know what happens when references to local variables are involved. But it not failing spirv-val sounds very suspicious, as if rust-gpu is emitting something very vile that spirv-val is not catching.

Could you provide a minimal example that fails?

[tombh]

Ohh interesting. Okay, great, I'll make a minimal example and post it on rust-gpu...