Skip to content

test(qwen3.5): preserve MoE expert stack ordering#653

Merged
inureyes merged 4 commits into
lablup:mainfrom
ukint-vs:ukint-vs/qwen35-moe-loader-name-fix
Jul 7, 2026
Merged

test(qwen3.5): preserve MoE expert stack ordering#653
inureyes merged 4 commits into
lablup:mainfrom
ukint-vs:ukint-vs/qwen35-moe-loader-name-fix

Conversation

@ukint-vs

@ukint-vs ukint-vs commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • keep this PR as a focused test-only follow-up now that the loader/sanitizer implementation landed through fix(qwen3.5-moe): sanitize toward the gate/up/down expert names SparseMoeBlock actually loads #671
  • remove the redundant ignored output-key-only test from src/models/qwen3_5_tests.rs
  • add a sanitizer unit test that verifies per-expert gate_proj, up_proj, and down_proj tensors are stacked into the current switch_mlp.*_proj loader names while preserving expert order and projection values

Audit result

The original implementation delta is already present on current main via #671, which closed #670. This branch now leaves the main qwen3_5::sanitize_weights implementation unchanged and only contributes additional regression coverage aligned with the current SwitchGLU/SparseMoeBlock naming convention.

Validation

  • cargo fmt --all -- --check
  • cargo test qwen3_5::sanitize_tests -- --nocapture

Refs #670 and #671

@cla-assistant

cla-assistant Bot commented Jul 4, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@cla-assistant

cla-assistant Bot commented Jul 4, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@ukint-vs ukint-vs marked this pull request as ready for review July 4, 2026 09:00
@inureyes inureyes added area:architecture Architecture and code structure changes type:bug Bug fixes, error corrections, or issue resolutions priority:medium Medium priority status:investigation Feasibility spike / under investigation labels Jul 6, 2026
@inureyes inureyes self-assigned this Jul 6, 2026
# Conflicts:
#	src/models/qwen3_5.rs
@inureyes

inureyes commented Jul 7, 2026

Copy link
Copy Markdown
Member

Thanks for the contribution and for tracking down the Qwen3.5 MoE switch projection naming mismatch. I audited the PR against current main and the related fix that landed in #671.

The main loader/sanitizer implementation is now already equivalent on main: src/models/qwen3_5.rs has no remaining delta for this PR after #671, and #670 is closed. The useful part still worth preserving here is the extra test coverage, but it should be adjusted to the current implementation style so it complements the stronger load-through regression test that #671 added.

I will push a small follow-up commit to this PR that drops the now-redundant implementation delta and keeps only a cleaned-up, focused test. After that I will rerun the relevant qwen3_5 sanitizer regression and merge if the branch remains clean.

@inureyes inureyes changed the title fix: preserve Qwen3.5 MoE switch projection names test(qwen3.5): preserve MoE expert stack ordering Jul 7, 2026
@inureyes inureyes merged commit 6605d1e into lablup:main Jul 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:architecture Architecture and code structure changes priority:medium Medium priority status:investigation Feasibility spike / under investigation type:bug Bug fixes, error corrections, or issue resolutions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(qwen3.5-moe): sanitize renames stacked experts to w1/w2/w3 but SparseMoeBlock now loads gate_proj/up_proj/down_proj (load regression since #588)

2 participants