Repository navigation
Conversation
- Add missing expert and attention norm sharding plans to Qwen3VLMoeTextConfig.base_model_tp_plan (gate_up_proj, down_proj, moe_tp_experts, q_norm, k_norm) - Add _tp_plan and _fsdp_plan for lm_head to Qwen3VLMoeForConditionalGeneration - Regenerate modeling_qwen3_vl_moe.py and configuration_qwen3_vl_moe.py from modular_qwen3_vl_moe.py - Add test_tp_plan in test_modeling_qwen3_vl_moe.py Fixes #48308
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…rmers into qwen3_vl_moe_tp_plan
|
run-slow: qwen3_vl_moe |
|
This comment contains models: ["models/qwen3_vl_moe"] |
|
Thanks for investigating the root cause and resolving the checkpoint transposition issue in the loader engine @3outeille! The target-to-source dimension mapping ( One small note: I noticed the model-level unit test |
|
Also, since the Appreciate it! |
|
sure will add co-author ! |
A transpose only moves axes, so a target shard maps to a slice of the source and shard-on-read still applies. #48373 does that translation, which is better than converting in full, so drop the transpose case here. PermuteForRope is not that: it interleaves rows and takes the head size from tensor.shape[0], which on a shard is the local row count, so it permutes with the wrong head size and raises nothing. That one still has to be converted before the shard is taken.
|
Is this PR ready top be merged? |
|
Yes, from my side this is good to merge! The loader dimension mapping handles the transposed checkpoint weights cleanly, the model TP rules are set, and the test suite is green. Just a friendly reminder to please preserve the co-authorship attribution on the squash merge commit when landing: Thanks @3outeille and @qgallouedec! |
|
Nope not yet gimme a bit of time! On it |
|
Hey, any update on this? |
…rs them Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Ah shit, yeah sorry mate |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: qwen3_vl_moe |
CI recapDashboard: View test results in Grafana |
ArthurZucker
left a comment
There was a problem hiding this comment.
main issue with this is how we change core model loading. I want to be a bit more careful, and this mapping will only work for non destructive operations.
As such it's a "patch" and IMO we need to patch in TP not in core model loading.
| _tp_plan = {"lm_head": "colwise_gather_output"} | ||
| _fsdp_plan = {"lm_head": "keep_full_weight"} |
There was a problem hiding this comment.
as I said in general these should be default for conditional generation (otherwise its "kinda" unrelated code for everyone)!
…gface#49206) * GLM-4V MoE: shard the routed and shared experts under tensor parallelism The TP plan covered attention and the dense MLPs only, so tp_plan="auto" left every expert replicated. It now shards the routed experts (packed colwise gate_up, rowwise down) and the shared experts, and gathers lm_head's colwise output. The expert-parallel plan is unchanged. Checked on 2 GPUs: a tiny model's TP=2 hidden states match the unsharded model to 1.7e-6 (fp32). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Load sharded experts through transposing and fuse-and-split conversions Transpose(check_dims=True) compares against this rank's slice of a sharded parameter, and per-expert sharding recognizes ErnieFuseAndSplitTextVisionExperts, indexing within each split stack. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Qwen3-VL-MoE: keep experts in the checkpoint's (num_experts, in, out) layout and shard them under TP Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Cohere2-MoE, EXAONE-MoE, Qwen2-MoE: shard the experts under tensor parallelism Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Aria: return (router_logits, weights, indices) from the router and shard the experts under TP and EP Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Ernie4.5-VL-MoE: shard the text and vision experts under TP and EP The vision TP plan drops the fused qkv/proj entries, which cannot be sharded colwise. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * GraniteMoE family, Jamba, Nemotron-H: add expert-parallel plans The Granite routers return (router_logits, weights, indices) and Jamba gets a JambaTopKRouter, the order ep_router expects. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Test that every MoE parallel plan shards the expert weights Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * remove comments * modular * Move the MoE plan test into TensorParallelTesterMixin and add the mixin to VLMModelTest The test replaces the mixin's EP-plan assertion and also checks TP plans. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Ernie4.5-VL-MoE: run the model tests on VLMModelTest Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Aria: run the model tests on VLMModelTest Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * MiniMax-M3-VL: run the model tests on VLMModelTest Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * using VLMModelTest * put the tesnor parallel mixin in multi modal mixin * Run the MoE plan test on every MoE test class GraniteMoE and GraniteMoeShared move to CausalLMModelTest, the Gemma4 and Inkling audio classes to ALMModelTest, and the remaining MoE classes inherit TensorParallelTesterMixin directly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * GraniteMoE family: record router_logits from the router's first output Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Qwen3-VL-MoE: drop the expert layout and TP plan changes, huggingface#48373 covers them Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Aria, GraniteMoE family, Jamba, Nemotron-H: default expert parallelism to token dispatch Their experts take the standard (hidden_states, top_k_index, top_k_weights), so the new default `ep_dispatch_experts` serves them; the router rule it ignores is dropped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * style * Review: require an EP plan in the MoE plan test, Qwen3-Omni-MoE text TP/EP plans, drop Ernie's test override - The MoE plan test requires every MoE to ship an EP plan again (a TP one stays optional) and reads the expert weights in one pass over the parameters. - Qwen3-Omni-MoE's thinker text model gains `moe_tp_experts` in its TP plan (TP crashed on mixed Tensor/DTensor without it) and the token-dispatch EP plan its talker already has. - Ernie4.5-VL-MoE's `test_mismatching_num_image_tokens` override is covered by `VLMModelTest` now. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Keep this PR to tensor/expert parallelism: the tester migration moves to its own PR The VLM/ALM/CausalLM tester migrations (Aria, Ernie4.5-VL-MoE, Gemma4, Inkling, MiniMax-M3-VL, Qwen3.5-MoE, GraniteMoE, GraniteMoeShared) go back to main's testers here and land separately; those test classes inherit `TensorParallelTesterMixin` directly so the MoE plan test still runs on them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * GraniteMoE, GraniteMoeShared: test on CausalLMModelTest, with the expert-parallel plans it checks for Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * GraniteMoE-SWA: regenerate its config from GraniteMoE's, token-dispatch expert parallelism included Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Every MoE ships a TP and an EP plan that shard its experts - The MoE plan test requires both plans, reading each experts module's weights from its declared layout (`gate_up_proj` when gated, `up_proj` otherwise, and `down_proj`), a tied tensor once. - Expert TP plans for DeepSeek-V4, AFMoE, HunYuan-MoE, Jamba, GraniteMoE / Hybrid / Shared / SWA, LFM2-MoE, Nemotron-H, PhiMoE and Zaya (TP forward and backward match the unsharded model). - GPT-OSS and OpenAI-privacy-filter skip it until their experts are untransposed: stored transposed with biases, an intra-expert TP plan would add the down projection's bias once per rank. - Drops the TP mixin's commented-out encoder-decoder and VLM skips. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Zaya: keep tensor parallelism disabled, as its modular file intends Zaya deletes its inherited `base_model_tp_plan` on purpose (its 2 KV heads cap the useful TP degree), so the expert TP plan added for it goes, and the MoE plan test skips it with that reason. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Review: guard the expert-index wrap on Ernie's converter, drop the GraniteMoE-SWA plan comment - `core_model_loading`: the per-stack expert index only wraps for `ErnieFuseAndSplitTextVisionExperts`, the one converter that splits a stack into several. - GraniteMoE-SWA: remove the four-line plan explanation from its config. - TP mixin: keep the encoder-decoder and VLM skips as comments, and unfold the expert-weight comprehension into a loop. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Review: drop the stale plan-order note, split out Ernie4.5-VL-MoE's expert plans - The `NOTE(3outeille)` comment on GLM-4 MoE / GLM-4V MoE plans no longer holds: `_get_parameter_plan` matches the parameter's exact name before its parent module, so dict order does not matter. - Ernie4.5-VL-MoE's text/vision expert TP and EP plans move to their own PR, with the `core_model_loading` change their EP loading needs; its MoE plan test is skipped here until then. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Review: run the TP tests on VLMs/ALMs, cover Qwen2-Audio and Voxtral, shard Qwen3-VL-MoE's experts - `MultiModalModelTester` exposes its `conditional_generation_class` as `causal_lm_class`, so the distributed TP/EP tests run on an allowlisted VLM or ALM (through a text-only forward) instead of skipping every multimodal tester. - Qwen2-Audio and Voxtral join `TP_DISTRIBUTED_TEST_MODEL_TYPES`: the two ALMs that pass every TP/EP test, on different audio encoders and text backbones. - Qwen3-VL-MoE's text TP plan shards its experts like Qwen3-MoE's (it left them replicated). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * Review: find MoE models by their experts modules in the TP plan test The decorator grep missed composites whose experts live in another file (nemotron_h_omni, kimi_k25), and is_moe_model is True for every built model. The test now looks for experts modules on the built model and skips only when there are none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Aria: record router_logits from the router's first output After the router's outputs were reordered to (router_logits, top_k_weights, top_k_index), the OutputRecorder still read index 2, so router_logits returned the top-k indices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * DeepSeek-V4: give lm_head its TP plan back now that the model has one huggingface#49314 removed it while DeepSeek-V4 was EP-only. With the experts-only TP plan, a tied head reads the row-sharded embedding, so lm_head needs colwise_gather_output again. Also drops the stale "EP only" comment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Review: the TP mixin takes a VLM/ALM tester's conditional_generation_class, and finds experts before testing their plans The multimodal tester no longer aliases causal_lm_class: the mixin falls back to conditional_generation_class itself. test_moe_parallel_plans_shard_experts collects the expert weights with _get_expert_weight_names and skips up front when a model has none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Explanation
Specifying a
tp_planforqwen3-vl-moeis not enough to fix #48308. A fix is as well needed during the loading process.The loading order is:
checkpoint weight -> shard according to TP plan -> WeightConverter -> load local weightThe
tp_planforqwen3_vl_moeisgate_up_proj: packed_colwisewhich means sharding on dimension 1However, the Qwen3-VL-MoE checkpoint stores this weight transposed:
So basically:
Before the fix, the checkpoint would be
[E, H, 2I] -> shard checkpoint dim 1 — wrong dimension -> [E, H/2, 2I] -> transpose -> [E, 2I, H/2]=> incorrectWe could transpose before the the checkpoint but that involves full materialization. Therefore the loader needs to map model dimensions into checkpoint dimensions
The loader therefore needs to map:
this way we get,
[E, H, 2I] -> shard checkpoint dim 2 (mapped) -> [E, H, I] -> transpose -> [E, I, H]=> correctsame thing for
dow_projOutput