Skip to content

handle transpose during model loading with TP (Qwen3-VL-MoE) - #48373

Open
3outeille wants to merge 19 commits into
mainfrom
qwen3_vl_moe_tp_plan
Open

3outeille wants to merge 19 commits into
mainfrom
qwen3_vl_moe_tp_plan

Conversation

@3outeille

@3outeille 3outeille commented Aug 27, 2026 •

Copy link
Copy Markdown
Member

CPU CI GPU run-slow

Explanation

Specifying a tp_plan for qwen3-vl-moe is not enough to fix #48308. A fix is as well needed during the loading process.

The loading order is: checkpoint weight -> shard according to TP plan -> WeightConverter -> load local weight

The tp_plan for qwen3_vl_moe is gate_up_proj: packed_colwisewhich means sharding on dimension 1

However, the Qwen3-VL-MoE checkpoint stores this weight transposed:

 "qwen3_vl_moe": [
            WeightConverter(
                source_patterns="mlp.experts.gate_up_proj",
                target_patterns="mlp.experts.gate_up_proj",
                operations=[Transpose(1, 2, check_dims=True)],
            ),

So basically:

Model:      [E, 2I, H]  -> 2I is dimension 1
Checkpoint: [E, H, 2I]  -> 2I is dimension 2

Before the fix, the checkpoint would be [E, H, 2I] -> shard checkpoint dim 1 — wrong dimension -> [E, H/2, 2I] -> transpose -> [E, 2I, H/2] => incorrect

We could transpose before the the checkpoint but that involves full materialization. Therefore the loader needs to map model dimensions into checkpoint dimensions

The loader therefore needs to map:

model dimension 1 -> checkpoint dimension 2
model dimension 2 -> checkpoint dimension 1

this way we get, [E, H, 2I] -> shard checkpoint dim 2 (mapped) -> [E, H, I] -> transpose -> [E, I, H] => correct

same thing for dow_proj

Output

# torchrun --nproc-per-node 4 repro.py
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from transformers.distributed.configuration_utils import DistributedConfig

model_id = "Qwen/Qwen3-VL-30B-A3B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(
    model_id, dtype=torch.bfloat16, distributed_config=DistributedConfig(tp_size=4)
)
print(torch.cuda.memory_allocated() / 1e9)  # expected ~15 GB/rank

processor = AutoProcessor.from_pretrained(model_id)
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg",
            },
            {"type": "text", "text": "Describe this image in one sentence."},
        ],
    }
]
inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt"
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=30, do_sample=False)
generated_ids = generated_ids[:, inputs.input_ids.shape[1] :]
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])
(env_main) ➜  qwen3_vl_moe_tp_plan git:(qwen3_vl_moe_tp_plan) ✗ torchrun --nproc-per-node 4 repro.py
W0827 12:47:39.173000 4059001 torch/distributed/run.py:851] 
W0827 12:47:39.173000 4059001 torch/distributed/run.py:851] *****************************************
W0827 12:47:39.173000 4059001 torch/distributed/run.py:851] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0827 12:47:39.173000 4059001 torch/distributed/run.py:851] *****************************************
Skipping import of cpp extensions due to incompatible torch version 2.11.0+cu128 for torchao version 0.16.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0+cu128 for torchao version 0.16.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0+cu128 for torchao version 0.16.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
Skipping import of cpp extensions due to incompatible torch version 2.11.0+cu128 for torchao version 0.16.0             Please see https://github.com/pytorch/ao/issues/2919 for more info
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 882/882 [00:04<00:00, 185.92it/s]
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 882/882 [00:04<00:00, 185.29it/s]
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 882/882 [00:04<00:00, 185.23it/s]
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 882/882 [00:04<00:00, 184.67it/s]
git16.363018752
 p16.363018752
16.363018752
us16.363018752
A fluffy Pallas's cat walks through the snow in front of a backdrop of birch trees.
A fluffy Pallas's cat walks through the snow in front of a backdrop of birch trees.
A fluffy Pallas's cat walks through the snow in front of a backdrop of birch trees.
A fluffy Pallas's cat walks through the snow in front of a backdrop of birch trees.

Noxtimo and others added 2 commits August 26, 2026 22:39
- Add missing expert and attention norm sharding plans to Qwen3VLMoeTextConfig.base_model_tp_plan (gate_up_proj, down_proj, moe_tp_experts, q_norm, k_norm)
- Add _tp_plan and _fsdp_plan for lm_head to Qwen3VLMoeForConditionalGeneration
- Regenerate modeling_qwen3_vl_moe.py and configuration_qwen3_vl_moe.py from modular_qwen3_vl_moe.py
- Add test_tp_plan in test_modeling_qwen3_vl_moe.py

Fixes #48308
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@3outeille 3outeille changed the title Fix Native TP expert and lm_head sharding for Qwen3-VL-MoE handle transpose during model loading with TP (Qwen3-VL-MoE) Aug 27, 2026
@3outeille
3outeille marked this pull request as ready for review August 27, 2026 13:14
3outeille and others added 4 commits August 27, 2026 13:19
Merge branch 'qwen3_vl_moe_tp_plan' of github.com:huggingface/transformers into qwen3_vl_moe_tp_plan
@3outeille
3outeille removed the request for review from Rocketknight1 August 27, 2026 14:31
@3outeille

Copy link
Copy Markdown
Member Author

run-slow: qwen3_vl_moe

@github-actions

Copy link
Copy Markdown
Contributor

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/qwen3_vl_moe"]
quantizations: []

@github-actions

Copy link
Copy Markdown
Contributor

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN f4f88ece workflow commit (merge commit)
PR 0826c37b branch commit (from PR)
main 155b8993 base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

@Noxtimo

Noxtimo commented Aug 27, 2026

Copy link
Copy Markdown

Thanks for investigating the root cause and resolving the checkpoint transposition issue in the loader engine @3outeille!

The target-to-source dimension mapping (get_source_dim_mapping) in DtensorShardOperation makes complete sense for avoiding full checkpoint materialization during shard-on-read.

One small note: I noticed the model-level unit test test_tp_plan from #48354 in tests/models/qwen3_vl_moe/test_modeling_qwen3_vl_moe.py was omitted here. Adding that test alongside the engine tests in test_core_model_loading.py would help prevent regressions to Qwen3VLMoe's TP plan in the future.

@Noxtimo

Noxtimo commented Aug 27, 2026

Copy link
Copy Markdown

Also, since the Qwen3-VL-MoE TP & FSDP plan rules and definitions (configuration_qwen3_vl_moe.py, modeling_qwen3_vl_moe.py, modular_qwen3_vl_moe.py) originated from #48354, could you please add co-authorship attribution on the squash merge commit when this lands?

Co-authored-by: Noxtimo <139155109+Noxtimo@users.noreply.github.com>

Appreciate it!

@3outeille

Copy link
Copy Markdown
Member Author

sure will add co-author !

@3outeille
3outeille enabled auto-merge September 1, 2026 19:57
@3outeille
3outeille disabled auto-merge September 1, 2026 19:57
qgallouedec added a commit that referenced this pull request Sep 8, 2026
A transpose only moves axes, so a target shard maps to a slice of the
source and shard-on-read still applies. #48373 does that translation, which
is better than converting in full, so drop the transpose case here.

PermuteForRope is not that: it interleaves rows and takes the head size
from tensor.shape[0], which on a shard is the local row count, so it
permutes with the wrong head size and raises nothing. That one still has to
be converted before the shard is taken.
@qgallouedec

Copy link
Copy Markdown
Member

Is this PR ready top be merged?

@Noxtimo

Noxtimo commented Sep 8, 2026

Copy link
Copy Markdown

Yes, from my side this is good to merge! The loader dimension mapping handles the transposed checkpoint weights cleanly, the model TP rules are set, and the test suite is green. Just a friendly reminder to please preserve the co-authorship attribution on the squash merge commit when landing:

Co-authored-by: Noxtimo <139155109+Noxtimo@users.noreply.github.com>

Thanks @3outeille and @qgallouedec!

@VI-Arthur

Copy link
Copy Markdown

Nope not yet gimme a bit of time! On it

qgallouedec added a commit that referenced this pull request Sep 19, 2026
Fixes rather than the expert-parallel or fast-loading work, and none of the multi-node runs
needed them. After this the tree equals ep-umbrella: main + #48792 + #48227 + #48604 + the
Trainer fix for the parameters FSDP2 does not shard.
@molbap molbap mentioned this pull request Sep 30, 2026
5 tasks
@qgallouedec

Copy link
Copy Markdown
Member

Hey, any update on this?

IlyasMoutawwakil added a commit that referenced this pull request Oct 2, 2026
…rs them

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ArthurZucker

Copy link
Copy Markdown
Collaborator

Ah shit, yeah sorry mate

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: qwen3_vl_moe

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 37553389062:1
Result: success | Jobs: 16 | Tests: 193,277 | Failures: 0 | Duration: 16h 28m

@ArthurZucker ArthurZucker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main issue with this is how we change core model loading. I want to be a bit more careful, and this mapping will only work for non destructive operations.
As such it's a "patch" and IMO we need to patch in TP not in core model loading.

Comment on lines +345 to +346
_tp_plan = {"lm_head": "colwise_gather_output"}
_fsdp_plan = {"lm_head": "keep_full_weight"}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as I said in general these should be default for conditional generation (otherwise its "kinda" unrelated code for everyone)!

Stanley00 pushed a commit to stanley-fork/hf-transformers that referenced this pull request Oct 16, 2026
…gface#49206)

* GLM-4V MoE: shard the routed and shared experts under tensor parallelism

The TP plan covered attention and the dense MLPs only, so tp_plan="auto"
left every expert replicated. It now shards the routed experts (packed
colwise gate_up, rowwise down) and the shared experts, and gathers lm_head's
colwise output. The expert-parallel plan is unchanged. Checked on 2 GPUs:
a tiny model's TP=2 hidden states match the unsharded model to 1.7e-6 (fp32).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Load sharded experts through transposing and fuse-and-split conversions

Transpose(check_dims=True) compares against this rank's slice of a sharded parameter, and per-expert
sharding recognizes ErnieFuseAndSplitTextVisionExperts, indexing within each split stack.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Qwen3-VL-MoE: keep experts in the checkpoint's (num_experts, in, out) layout and shard them under TP

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Cohere2-MoE, EXAONE-MoE, Qwen2-MoE: shard the experts under tensor parallelism

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Aria: return (router_logits, weights, indices) from the router and shard the experts under TP and EP

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Ernie4.5-VL-MoE: shard the text and vision experts under TP and EP

The vision TP plan drops the fused qkv/proj entries, which cannot be sharded colwise.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GraniteMoE family, Jamba, Nemotron-H: add expert-parallel plans

The Granite routers return (router_logits, weights, indices) and Jamba gets a JambaTopKRouter, the order ep_router expects.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Test that every MoE parallel plan shards the expert weights

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* remove comments

* modular

* Move the MoE plan test into TensorParallelTesterMixin and add the mixin to VLMModelTest

The test replaces the mixin's EP-plan assertion and also checks TP plans.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Ernie4.5-VL-MoE: run the model tests on VLMModelTest

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Aria: run the model tests on VLMModelTest

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* MiniMax-M3-VL: run the model tests on VLMModelTest

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* using VLMModelTest

* put the tesnor parallel mixin in multi modal mixin

* Run the MoE plan test on every MoE test class

GraniteMoE and GraniteMoeShared move to CausalLMModelTest, the Gemma4 and Inkling audio classes to
ALMModelTest, and the remaining MoE classes inherit TensorParallelTesterMixin directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GraniteMoE family: record router_logits from the router's first output

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Qwen3-VL-MoE: drop the expert layout and TP plan changes, huggingface#48373 covers them

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Aria, GraniteMoE family, Jamba, Nemotron-H: default expert parallelism to token dispatch

Their experts take the standard (hidden_states, top_k_index, top_k_weights), so the new default
`ep_dispatch_experts` serves them; the router rule it ignores is dropped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style

* Review: require an EP plan in the MoE plan test, Qwen3-Omni-MoE text TP/EP plans, drop Ernie's test override

- The MoE plan test requires every MoE to ship an EP plan again (a TP one stays optional) and reads the expert
  weights in one pass over the parameters.
- Qwen3-Omni-MoE's thinker text model gains `moe_tp_experts` in its TP plan (TP crashed on mixed Tensor/DTensor
  without it) and the token-dispatch EP plan its talker already has.
- Ernie4.5-VL-MoE's `test_mismatching_num_image_tokens` override is covered by `VLMModelTest` now.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Keep this PR to tensor/expert parallelism: the tester migration moves to its own PR

The VLM/ALM/CausalLM tester migrations (Aria, Ernie4.5-VL-MoE, Gemma4, Inkling, MiniMax-M3-VL, Qwen3.5-MoE,
GraniteMoE, GraniteMoeShared) go back to main's testers here and land separately; those test classes inherit
`TensorParallelTesterMixin` directly so the MoE plan test still runs on them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GraniteMoE, GraniteMoeShared: test on CausalLMModelTest, with the expert-parallel plans it checks for

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GraniteMoE-SWA: regenerate its config from GraniteMoE's, token-dispatch expert parallelism included

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Every MoE ships a TP and an EP plan that shard its experts

- The MoE plan test requires both plans, reading each experts module's weights from its declared layout
  (`gate_up_proj` when gated, `up_proj` otherwise, and `down_proj`), a tied tensor once.
- Expert TP plans for DeepSeek-V4, AFMoE, HunYuan-MoE, Jamba, GraniteMoE / Hybrid / Shared / SWA, LFM2-MoE,
  Nemotron-H, PhiMoE and Zaya (TP forward and backward match the unsharded model).
- GPT-OSS and OpenAI-privacy-filter skip it until their experts are untransposed: stored transposed with
  biases, an intra-expert TP plan would add the down projection's bias once per rank.
- Drops the TP mixin's commented-out encoder-decoder and VLM skips.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Zaya: keep tensor parallelism disabled, as its modular file intends

Zaya deletes its inherited `base_model_tp_plan` on purpose (its 2 KV heads cap the useful TP degree), so the
expert TP plan added for it goes, and the MoE plan test skips it with that reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review: guard the expert-index wrap on Ernie's converter, drop the GraniteMoE-SWA plan comment

- `core_model_loading`: the per-stack expert index only wraps for `ErnieFuseAndSplitTextVisionExperts`, the one
  converter that splits a stack into several.
- GraniteMoE-SWA: remove the four-line plan explanation from its config.
- TP mixin: keep the encoder-decoder and VLM skips as comments, and unfold the expert-weight comprehension into a loop.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review: drop the stale plan-order note, split out Ernie4.5-VL-MoE's expert plans

- The `NOTE(3outeille)` comment on GLM-4 MoE / GLM-4V MoE plans no longer holds: `_get_parameter_plan` matches the
  parameter's exact name before its parent module, so dict order does not matter.
- Ernie4.5-VL-MoE's text/vision expert TP and EP plans move to their own PR, with the `core_model_loading` change
  their EP loading needs; its MoE plan test is skipped here until then.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review: run the TP tests on VLMs/ALMs, cover Qwen2-Audio and Voxtral, shard Qwen3-VL-MoE's experts

- `MultiModalModelTester` exposes its `conditional_generation_class` as `causal_lm_class`, so the
  distributed TP/EP tests run on an allowlisted VLM or ALM (through a text-only forward) instead of
  skipping every multimodal tester.
- Qwen2-Audio and Voxtral join `TP_DISTRIBUTED_TEST_MODEL_TYPES`: the two ALMs that pass every TP/EP
  test, on different audio encoders and text backbones.
- Qwen3-VL-MoE's text TP plan shards its experts like Qwen3-MoE's (it left them replicated).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review: find MoE models by their experts modules in the TP plan test

The decorator grep missed composites whose experts live in another file (nemotron_h_omni, kimi_k25), and is_moe_model is True for every built model. The test now looks for experts modules on the built model and skips only when there are none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Aria: record router_logits from the router's first output

After the router's outputs were reordered to (router_logits, top_k_weights, top_k_index), the OutputRecorder still read index 2, so router_logits returned the top-k indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* DeepSeek-V4: give lm_head its TP plan back now that the model has one

huggingface#49314 removed it while DeepSeek-V4 was EP-only. With the experts-only TP plan, a tied head reads the row-sharded embedding, so lm_head needs colwise_gather_output again. Also drops the stale "EP only" comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Review: the TP mixin takes a VLM/ALM tester's conditional_generation_class, and finds experts before testing their plans

The multimodal tester no longer aliases causal_lm_class: the mixin falls back to conditional_generation_class itself. test_moe_parallel_plans_shard_experts collects the expert weights with _get_expert_weight_names and skips up front when a model has none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native TP does not shard qwen3_vl_moe experts (weights fully replicated per rank)

6 participants