Skip to content

[PyTorch] Allow CP P2P transport group overrides - #3420

Draft
xiaoyao0115 wants to merge 1 commit into
NVIDIA:release_v2.14from
xiaoyao0115:cp-p2p-transport
Draft

[PyTorch] Allow CP P2P transport group overrides#3420
xiaoyao0115 wants to merge 1 commit into
NVIDIA:release_v2.14from
xiaoyao0115:cp-p2p-transport

Conversation

@xiaoyao0115

@xiaoyao0115 xiaoyao0115 commented Aug 24, 2026

Copy link
Copy Markdown

Summary

  • keep the logical CP process group for CP size, rank, and peer selection
  • allow CP attention P2P operations to use a separate transport process group
  • route both batched and direct isend/irecv through that transport group
  • store overrides weakly so registration does not extend process-group lifetimes

Motivation

Dynamic CP CUDA graphs capture multiple logical CP sizes. Initializing one NCCL communicator per logical CP group adds persistent communicator memory. The companion Megatron Core change maps those logical groups to one pre-warmed parent communicator during graph warmup and capture.

The default path is unchanged when no override is registered.

This change was split from #3353 so the transport API can be reviewed independently from CUDA Graph slot-memory sharing.

Validation

  • Black 24.4.2
  • Python AST validation
  • transport-selection and weak-lifetime smoke test
  • upstream Core and all PyTorch checks pass
  • JAX and the aggregate All job fail while compiling unchanged common/activation/gelu.cu: fatbinary cannot open gelu.compute_75.ptx; this PR changes PyTorch files only

Companion change

Signed-off-by: Tailai Ma <tailaim@ferries-work.nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant