[PyTorch] Fix fake kernel metadata mismatch for empty-input MoE permute ops - #40
Open
pggPL wants to merge 1 commit into
Open
[PyTorch] Fix fake kernel metadata mismatch for empty-input MoE permute ops#40pggPL wants to merge 1 commit into
pggPL wants to merge 1 commit into
Conversation
For empty input, moe_permute (mask and index map) returned row_id_map created with torch.tensor([]) (float32), while the registered fake kernels declare int32 with different shapes. PyTorch >= 2.14 inductor validates fallback output dtype metadata at runtime (pytorch#183731), so torch.compile'd empty-input permute now errors with 'expected dtype torch.int32 but got torch.float32'. Return correctly-typed empty tensors from the eager fast paths, align the index-map fake output shape with the empty-input fast path, and keep permuted_probs dtype consistent between eager and fake. Add an empty-input test for the index map path. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
For empty input,
moe_permute(both mask and index map) returnsrow_id_mapcreated withtorch.tensor([]), i.e. float32 with shape(0,), while the registered fake (meta) kernels declare int32 with shape(0, num_experts * 2 + 1)(mask map) /(num_tokens * topK,)(index map).PyTorch validates fallback output dtype metadata against the fake kernel at runtime since pytorch#183731 (nightlies after 2026-07-22, torch 2.14), so
torch.compile'd empty-input permute now fails with:Reproduced by
test_permutation_mask_map_empty_input[True-*]andtest_permutation_mask_map_alongside_probs_empty_input[True-*]on torch 2.14 nightly. On torch <= 2.13 the mismatch passes silently (no metadata check).Type of change
Changes
permute_mask_map_fwdempty-input fast path returnsrow_id_mapas int32(0, num_experts * 2 + 1)andpermuted_probswithprobs.dtype, matching the fake kernel.permute_index_mapempty-input fast path returnsrow_id_mapas int32(0,)instead of float32._moe_permute_index_map_fakeoutput shape now matches the empty-input fast path (0 rows,num_out_tokensignored), mirroring the existing mask-map fake handling._moe_permute_mask_map_forward_fakeusesprobs.dtypeforpermuted_probsalso whenout_rows == 0, keeping eager and fake consistent.test_permutation_index_map_empty_input(eager + compile) — the index-map path had the same bug but no empty-input test coverage.Verified on torch 2.14.0.dev20260807 nightly: all
empty_inputtests pass (24/24; 9 compile variants fail without the fix), fulltests/pytorch/test_permutation.pygreen (248 passed, 154 skipped).Checklist:
🤖 Generated with Claude Code