Skip to content

[PyTorch] Fix fake kernel metadata mismatch for empty-input MoE permute ops - #40

Open
pggPL wants to merge 1 commit into
mainfrom
moe_permute_empty_input_meta
Open

[PyTorch] Fix fake kernel metadata mismatch for empty-input MoE permute ops#40
pggPL wants to merge 1 commit into
mainfrom
moe_permute_empty_input_meta

Conversation

@pggPL

@pggPL pggPL commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Description

For empty input, moe_permute (both mask and index map) returns row_id_map created with torch.tensor([]), i.e. float32 with shape (0,), while the registered fake (meta) kernels declare int32 with shape (0, num_experts * 2 + 1) (mask map) / (num_tokens * topK,) (index map).

PyTorch validates fallback output dtype metadata against the fake kernel at runtime since pytorch#183731 (nightlies after 2026-07-22, torch 2.14), so torch.compile'd empty-input permute now fails with:

RuntimeError: expected dtype torch.int32 but got torch.float32
Error in op: torch.ops.te_moe.permute_mask_map_fwd.default

Reproduced by test_permutation_mask_map_empty_input[True-*] and test_permutation_mask_map_alongside_probs_empty_input[True-*] on torch 2.14 nightly. On torch <= 2.13 the mismatch passes silently (no metadata check).

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • permute_mask_map_fwd empty-input fast path returns row_id_map as int32 (0, num_experts * 2 + 1) and permuted_probs with probs.dtype, matching the fake kernel.
  • permute_index_map empty-input fast path returns row_id_map as int32 (0,) instead of float32.
  • _moe_permute_index_map_fake output shape now matches the empty-input fast path (0 rows, num_out_tokens ignored), mirroring the existing mask-map fake handling.
  • _moe_permute_mask_map_forward_fake uses probs.dtype for permuted_probs also when out_rows == 0, keeping eager and fake consistent.
  • Added test_permutation_index_map_empty_input (eager + compile) — the index-map path had the same bug but no empty-input test coverage.

Verified on torch 2.14.0.dev20260807 nightly: all empty_input tests pass (24/24; 9 compile variants fail without the fix), full tests/pytorch/test_permutation.py green (248 passed, 154 skipped).

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

🤖 Generated with Claude Code

For empty input, moe_permute (mask and index map) returned row_id_map
created with torch.tensor([]) (float32), while the registered fake
kernels declare int32 with different shapes. PyTorch >= 2.14 inductor
validates fallback output dtype metadata at runtime (pytorch#183731),
so torch.compile'd empty-input permute now errors with
'expected dtype torch.int32 but got torch.float32'.

Return correctly-typed empty tensors from the eager fast paths, align
the index-map fake output shape with the empty-input fast path, and
keep permuted_probs dtype consistent between eager and fake.
Add an empty-input test for the index map path.

Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant