Skip to content

feat(packages): submitted NVK work to /dev/gpuN queues - #599

Merged
FlareCoding merged 2 commits into
masterfrom
pr/mesa-exec
Nov 7, 2026
Merged

FlareCoding merged 2 commits into
masterfrom
pr/mesa-exec

Conversation

@FlareCoding

Copy link
Copy Markdown
Owner

Summary

On a Stellux GPU, NVK (Mesa's Vulkan driver for NVIDIA GPUs) had no queue to submit work to, so vkCreateDevice failed at NVK's upload queue for any device with queues.

Changes

  • Vulkan queues run on the kernel's GPU queues: patch 0017 gives the Stellux nvkmd (an implementation of NVK's kernel driver interface over /dev/gpuN) an exec ctx in nvkmd_stellux_ctx.c, through which NVK submits a queue's work. Each owns one queue of the device's GPU context, GPU_QUEUE_GRAPHICS for 3D or compute work and GPU_QUEUE_COPY otherwise, and a private timeline whose next point every submission reaches.
  • Batches past the kernel's limits still reach the GPU: a batch past GPU_SUBMIT's limits of 32 points and 4096 command ranges goes in a chain of submissions, each later part waiting on the private point of the one before, so a failed part fails the rest. Pushes NVK marks incomplete go in one submission with the push after them, and a run of them past 4096 ranges fails with VK_ERROR_OUT_OF_DEVICE_MEMORY, on which NVK loses the device.
  • A full queue makes the submit wait instead of failing: when the kernel refuses a GPU_SUBMIT with EBUSY, since the queue holds 64 pending submissions, the exec ctx waits on the CPU for the oldest one's private point and retries once. vkQueueSubmit then blocks the program's thread until the queue has room or the device is lost, and when the oldest submission waits on a point that thread alone would signal, only a loss ends the block.
  • Failed work loses the device: the device check, which asks the kernel whether the device's GPU context is lost, also queries each exec ctx's private timeline, so work that failed on any queue loses its device, since Vulkan reports failed work only as device loss. An EIO from GPU_SUBMIT loses the device with the kernel's reason for the lost context.
  • The shader local memory area outlives its work: when a shader needs more shader local memory than the device's area holds, NVK replaces the area. Patch 0016 keeps the old one on each 3D or compute queue until the GPU has passed that queue's earlier work, since the Stellux nvkmd unmaps a mem's GPU range as soon as NVK drops its last reference.
  • Tests: vktest's new device, memory and timeline stages pass on both architectures against a local double of four GPUs that never lands, covering devices of each queue family, memory of every type allocated and imported through opaque descriptors with the host-visible types mapped, and timeline waits and signals past the kernel's 32 points. The double runs no commands, so no check shows a queue's kind, a full queue, incomplete pushes, a chain past 4096 ranges, failed work, a lost context or a replaced shader local memory area, and the stages needing GPU work follow in the next PR.

@FlareCoding
FlareCoding added this pull request to stack #609 November 7, 2026 18:18
FlareCoding and others added 2 commits November 7, 2026 10:18
NVK on a Stellux GPU had no queue, so no device with queues could be created. Submissions past the
kernel's limits chain through a private timeline whose failed mark reads as device loss, and a
replaced shader local memory area outlives the work reading it, as freeing unmaps it at once.
@FlareCoding
FlareCoding merged commit ecdcd08 into master Nov 7, 2026
10 checks passed
@FlareCoding
FlareCoding deleted the pr/mesa-exec branch November 7, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant