Optimizations for q40, q80, q5k, fix Cosmos VLM prefill - #92
Open
liangliangchang wants to merge 8 commits into
Open
Optimizations for q40, q80, q5k, fix Cosmos VLM prefill#92liangliangchang wants to merge 8 commits into
liangliangchang wants to merge 8 commits into
Conversation
Apply the Q4_K J=128 WMMA split and next-K X prefetch to Q4_0 MMQ.
Co-authored-by: Cursor <cursoragent@cursor.com>
Q5_K unpacks to the same IU8 SRAM as Q4_K, so reuse the J=128 ntx=1 wmma_low / convert / wmma_high schedule and the Q4_K J policy. Co-authored-by: Cursor <cursoragent@cursor.com>
Prefetch Q5_K low- and high-bit data during WMMA while distributing high bits across lanes to keep J128 spill-free. Co-authored-by: Cursor <cursoragent@cursor.com>
The prefetch pipeline was gated on J == 96 || J == 128, but the batched WMMA vec_dot it feeds is gated on J == 128 only, and rows_per_warp() returns 16 (ntx=1) only at J == 128 -- at J == 96 it returns 32 (ntx=2). So J == 96 paired the prefetch with the generic two-minitile kernel. Q8_0 is also the only type caching 2*(I/nwarps) = 32 ints where the others cache 16, on top of two A tiles and 48 accumulators, so that combination spills. mmq_rdna35_tuned_J maps Q8_0 J_occupancy == 64 to 96 to avoid the J == 64 pathology, which made every 64-wide batch land on the broken width. MoE Q8_0 (J > 32 -> 96) had the same exposure. Cosmos-Reason2-8B Q8_0 prefill on gfx1151, llama-bench -p 512 -r 3 against rocm/gfx11 03d2068: ubatch 64: 111.4 -> 178.7 t/s (was -37.7% vs gfx11, now -0.1%) ubatch 96: n/a -> 267.6 t/s (now -0.4% vs gfx11) ubatch 128: 1426.8 -> 1430.9 t/s (+3.0% vs gfx11, unchanged) ubatch 512: 1581.5 -> 1581.9 t/s (+2.8% vs gfx11, unchanged) The J == 128 win is retained in full. test-backend-ops MUL_MAT q8_0: 64/64. Co-authored-by: Cursor <cursoragent@cursor.com>
Q4_0 and Q8_0 only have a batched WMMA vec_dot at J=128. At the other two widths where rows_per_warp() would pick 32 (J=64 and J=96, the only J>=64 multiples of 32) they fall back to the generic two-minitile schedule, which is several times slower on gfx115x. J=72/80/88/104/112/120 are unaffected because they are not multiples of 32 and already run ntx=1. This is a pre-existing cliff, not a regression: gfx11 shows it too. Q4_K is not affected, since its generic path handles ntx=2 well. llama-bench -p 512 -r 3 on gfx1151, versus rocm/gfx11 03d2068: Cosmos-Reason2-8B Q8_0 ubatch 64: 178.9 -> 883.1 t/s (+394%) ubatch 96: 268.8 -> 1151.4 t/s (+328%) Qwen3-14B Q4_0 ubatch 64: 153.6 -> 708.7 t/s (+361%) ubatch 96: 156.8 -> 725.8 t/s (+363%) Unaffected widths are unchanged (Q8_0 ubatch 32: 534.5 -> 533.1, ubatch 128: 1389.7 -> 1416.1). test-backend-ops MUL_MAT q4_0+q8_0: 134/134. Co-authored-by: Cursor <cursoragent@cursor.com>
The widening existed because J=64 was several times slower for q8_0, but that was the generic ntx=2 schedule which rows_per_warp() no longer selects for block quants. With J=64 healthy the widening only pads 64 valid columns into a 96-wide tile. Cosmos-Reason2-8B Q8_0, llama-bench -p 512 -r 3 on gfx1151: ubatch 64 (J 96 -> 64): 883.1 -> 1118.5 t/s (+26.7%) ubatch 96 (J 96, same): 1151.4 -> 1147.2 t/s (-0.4%) ubatch 128 (J 128, same): 1416.1 -> 1398.1 t/s (-1.3%) MoE keeps its own widening, which is untested here and unchanged. test-backend-ops MUL_MAT: 1484/1484. Co-authored-by: Cursor <cursoragent@cursor.com>
liangliangchang
changed the title
liangliangchang
marked this pull request as ready for review
September 1, 2026 19:49
mgehre-amd
reviewed
Sep 3, 2026
| -1, | ||
| ggml_cuda_mmq_load_tiles_q4_0<type, J, fallback>, | ||
| #if defined(RDNA3_5) | ||
| ggml_cuda_mmq_vec_dot_q4_0_q8_1_mma_rdna35<type, J, fallback>, |
Collaborator
There was a problem hiding this comment.
Why is this function called q4_0_q8_1 and the one in the #else branch q8_0_q8_1? Is this a mismatch?
Could you please explain why the optimizations needs a fresh kernel and couldn't be made part of the existing ggml_cuda_mmq_vec_dot_q8_0_q8_1_mma kernel?
Author
There was a problem hiding this comment.
Good point, it doesn't need a fresh kernel. The code is now folded into the existing kernel. The existing q8_0_q8_1_mma is used for all types because weights will be extended to q8_0 before computing with activations. It was named q4_0 because this optimization is only for the q4_0 case. But it is unnecessary to use a separated kernel, and this function is now folded.
The RDNA3.5 J=128 batched-WMMA schedule existed as two near-identical ~75-line copies, ggml_cuda_mmq_vec_dot_q4_0_q8_1_mma_rdna35 and ggml_cuda_mmq_vec_dot_q8_0_q8_1_mma_rdna35. They differed only in how they read the y scales: q4_0 unpacks a half2 (DS4 layout), q8_0 reads a float (D4 layout). ggml_cuda_mmq_vec_dot_q8_0_q8_1_mma already takes that layout as a template parameter and already branches on it, so the schedule now lives there behind J == 128 && ntx == 1, and both types dispatch straight to the generic kernel with their own layout. At J=128 only Q4_K/Q4_0/Q8_0/Q5_K get rows_per_warp=16 (ntx=1), and Q4_K/Q5_K have their own vec_dot, so the new branch covers exactly the two types that had hand-written copies; everything else keeps the generic schedule. This also drops both #if defined(RDNA3_5) blocks from the util-func dispatcher, so Q4_0 and Q8_0 read like every other type. Behaviour-preserving: the generated gfx1151 ISA for mmq-instance-q4_0 and mmq-instance-q8_0 is byte-identical before and after, apart from the __FILE__ string baked in by an assert. test-backend-ops MUL_MAT 1484/1484 and MUL_MAT_ID 869/869 pass; Cosmos-Reason2-8B Q8_0 VLM and Gemma-4-E2B Q4_0 VLM prefill are unchanged (-0.14%, +0.37%). Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Highlights: prefill gains on the shapes the MMQ work targets
Q4_0: Latency and throughput
Q8_0 Latency and throughput
Q5_k Latency and throughput
Model performance