support deepseek_v4 packing/CP#140
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for Context Parallel (CP) in DeepSeek-V4 attention by implementing boundary hidden state and rotary position embedding exchanges, supporting contiguous CP partitioning in split_cp_inputs, and adjusting RoPE application for packed sequences. The code reviewer provided valuable feedback to improve robustness, suggesting to handle potential None values for cp_group to avoid AttributeError, use getattr when accessing cp_partition_mode on packed_seq_params, and wrap the import of csa_cp_utils in a try-except block to gracefully handle older Megatron-Core versions.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
3 participants
THD: NVIDIA/Megatron-LM#5011
CP: NVIDIA/Megatron-LM#5087