Skip to content

remove colwise_gather_output for lm_head for DeepseekV4 - #49314

Merged
3outeille merged 7 commits into
mainfrom
remove-deepseek-v4-tp-plan
Oct 6, 2026
Merged

3outeille merged 7 commits into
mainfrom
remove-deepseek-v4-tp-plan

Conversation

@3outeille

@3outeille 3outeille commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

CPU CI GPU run-slow

Removing colwise_gather_output for lm_head for DeepseekV4 as it doesn't ship a base_tp_plan anyway. If we dont do that, it becomes an issue for #48859 as we run EP (which is decoupled from TP) with tied_embedding=True test and we find ourselves with a default colwise_gather_output for `lm_head for DeepseekV4 that shards the logits (despite not having a base_tp_plan) which makes the ep test fails

@3outeille
3outeille requested a review from vasqu October 5, 2026 10:07
@3outeille

Copy link
Copy Markdown
Member Author

run-slow: deepseek_v4

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

AMD CI

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs on AMD:

models: ["models/deepseek_v4"]

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Nvidia CI

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs on Nvidia:

models: ["models/deepseek_v4"]
quantizations: []

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

CI Results (Nvidia)

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN c2e846c0 workflow commit (merge commit)
PR 3fdf1bfa branch commit (from PR)
main 26c0a7fd base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

CI Results (AMD)

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN c2e846c0 workflow commit (merge commit)
PR 3fdf1bfa branch commit (from PR)
main 26c0a7fd base commit (on main)

⚠️ Model CI failed to report results

The test failure analysis could not be completed. Please check the workflow run for details.

@vasqu vasqu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea makes sense there is a long AI comment on EP as well lol

Imo TP might make sense to readd at some point? It's not like we wouldnt be able to combine EP and TP

Other point I find strange (not on you): We shard the indexer within the ep config, is this not kind of TP => we have blurry lines between EP/TP again?

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: deepseek_v4

@3outeille

3outeille commented Oct 5, 2026 •

Copy link
Copy Markdown
Member Author

Other point I find strange (not on you): We shard the indexer within the ep config, is this not kind of TP => we have blurry lines between EP/TP again?

yeah this should be moved to tp_plan. I removed it from ep_plan here: https://github.com/huggingface/transformers/blob/ep-dispatch-default/src/transformers/models/deepseek_v4/configuration_deepseek_v4.py

@3outeille
3outeille enabled auto-merge October 5, 2026 23:53
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 37392737416:1
Result: success | Jobs: 8 | Tests: 416 | Failures: 0 | Duration: 11m 16s

@3outeille
3outeille added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit 8073e6d Oct 6, 2026
37 checks passed
@3outeille
3outeille deleted the remove-deepseek-v4-tp-plan branch October 6, 2026 00:26
IlyasMoutawwakil added a commit that referenced this pull request Oct 15, 2026
#49314 removed it while DeepSeek-V4 was EP-only. With the experts-only TP plan, a tied head reads the row-sharded embedding, so lm_head needs colwise_gather_output again. Also drops the stale "EP only" comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stanley00 pushed a commit to stanley-fork/hf-transformers that referenced this pull request Oct 16, 2026
…gface#49206)

* GLM-4V MoE: shard the routed and shared experts under tensor parallelism

The TP plan covered attention and the dense MLPs only, so tp_plan="auto"
left every expert replicated. It now shards the routed experts (packed
colwise gate_up, rowwise down) and the shared experts, and gathers lm_head's
colwise output. The expert-parallel plan is unchanged. Checked on 2 GPUs:
a tiny model's TP=2 hidden states match the unsharded model to 1.7e-6 (fp32).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Load sharded experts through transposing and fuse-and-split conversions

Transpose(check_dims=True) compares against this rank's slice of a sharded parameter, and per-expert
sharding recognizes ErnieFuseAndSplitTextVisionExperts, indexing within each split stack.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Qwen3-VL-MoE: keep experts in the checkpoint's (num_experts, in, out) layout and shard them under TP

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Cohere2-MoE, EXAONE-MoE, Qwen2-MoE: shard the experts under tensor parallelism

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Aria: return (router_logits, weights, indices) from the router and shard the experts under TP and EP

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Ernie4.5-VL-MoE: shard the text and vision experts under TP and EP

The vision TP plan drops the fused qkv/proj entries, which cannot be sharded colwise.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GraniteMoE family, Jamba, Nemotron-H: add expert-parallel plans

The Granite routers return (router_logits, weights, indices) and Jamba gets a JambaTopKRouter, the order ep_router expects.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Test that every MoE parallel plan shards the expert weights

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* remove comments

* modular

* Move the MoE plan test into TensorParallelTesterMixin and add the mixin to VLMModelTest

The test replaces the mixin's EP-plan assertion and also checks TP plans.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Ernie4.5-VL-MoE: run the model tests on VLMModelTest

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Aria: run the model tests on VLMModelTest

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* MiniMax-M3-VL: run the model tests on VLMModelTest

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* using VLMModelTest

* put the tesnor parallel mixin in multi modal mixin

* Run the MoE plan test on every MoE test class

GraniteMoE and GraniteMoeShared move to CausalLMModelTest, the Gemma4 and Inkling audio classes to
ALMModelTest, and the remaining MoE classes inherit TensorParallelTesterMixin directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GraniteMoE family: record router_logits from the router's first output

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Qwen3-VL-MoE: drop the expert layout and TP plan changes, huggingface#48373 covers them

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Aria, GraniteMoE family, Jamba, Nemotron-H: default expert parallelism to token dispatch

Their experts take the standard (hidden_states, top_k_index, top_k_weights), so the new default
`ep_dispatch_experts` serves them; the router rule it ignores is dropped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style

* Review: require an EP plan in the MoE plan test, Qwen3-Omni-MoE text TP/EP plans, drop Ernie's test override

- The MoE plan test requires every MoE to ship an EP plan again (a TP one stays optional) and reads the expert
  weights in one pass over the parameters.
- Qwen3-Omni-MoE's thinker text model gains `moe_tp_experts` in its TP plan (TP crashed on mixed Tensor/DTensor
  without it) and the token-dispatch EP plan its talker already has.
- Ernie4.5-VL-MoE's `test_mismatching_num_image_tokens` override is covered by `VLMModelTest` now.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Keep this PR to tensor/expert parallelism: the tester migration moves to its own PR

The VLM/ALM/CausalLM tester migrations (Aria, Ernie4.5-VL-MoE, Gemma4, Inkling, MiniMax-M3-VL, Qwen3.5-MoE,
GraniteMoE, GraniteMoeShared) go back to main's testers here and land separately; those test classes inherit
`TensorParallelTesterMixin` directly so the MoE plan test still runs on them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GraniteMoE, GraniteMoeShared: test on CausalLMModelTest, with the expert-parallel plans it checks for

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* GraniteMoE-SWA: regenerate its config from GraniteMoE's, token-dispatch expert parallelism included

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Every MoE ships a TP and an EP plan that shard its experts

- The MoE plan test requires both plans, reading each experts module's weights from its declared layout
  (`gate_up_proj` when gated, `up_proj` otherwise, and `down_proj`), a tied tensor once.
- Expert TP plans for DeepSeek-V4, AFMoE, HunYuan-MoE, Jamba, GraniteMoE / Hybrid / Shared / SWA, LFM2-MoE,
  Nemotron-H, PhiMoE and Zaya (TP forward and backward match the unsharded model).
- GPT-OSS and OpenAI-privacy-filter skip it until their experts are untransposed: stored transposed with
  biases, an intra-expert TP plan would add the down projection's bias once per rank.
- Drops the TP mixin's commented-out encoder-decoder and VLM skips.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Zaya: keep tensor parallelism disabled, as its modular file intends

Zaya deletes its inherited `base_model_tp_plan` on purpose (its 2 KV heads cap the useful TP degree), so the
expert TP plan added for it goes, and the MoE plan test skips it with that reason.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review: guard the expert-index wrap on Ernie's converter, drop the GraniteMoE-SWA plan comment

- `core_model_loading`: the per-stack expert index only wraps for `ErnieFuseAndSplitTextVisionExperts`, the one
  converter that splits a stack into several.
- GraniteMoE-SWA: remove the four-line plan explanation from its config.
- TP mixin: keep the encoder-decoder and VLM skips as comments, and unfold the expert-weight comprehension into a loop.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review: drop the stale plan-order note, split out Ernie4.5-VL-MoE's expert plans

- The `NOTE(3outeille)` comment on GLM-4 MoE / GLM-4V MoE plans no longer holds: `_get_parameter_plan` matches the
  parameter's exact name before its parent module, so dict order does not matter.
- Ernie4.5-VL-MoE's text/vision expert TP and EP plans move to their own PR, with the `core_model_loading` change
  their EP loading needs; its MoE plan test is skipped here until then.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review: run the TP tests on VLMs/ALMs, cover Qwen2-Audio and Voxtral, shard Qwen3-VL-MoE's experts

- `MultiModalModelTester` exposes its `conditional_generation_class` as `causal_lm_class`, so the
  distributed TP/EP tests run on an allowlisted VLM or ALM (through a text-only forward) instead of
  skipping every multimodal tester.
- Qwen2-Audio and Voxtral join `TP_DISTRIBUTED_TEST_MODEL_TYPES`: the two ALMs that pass every TP/EP
  test, on different audio encoders and text backbones.
- Qwen3-VL-MoE's text TP plan shards its experts like Qwen3-MoE's (it left them replicated).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* Review: find MoE models by their experts modules in the TP plan test

The decorator grep missed composites whose experts live in another file (nemotron_h_omni, kimi_k25), and is_moe_model is True for every built model. The test now looks for experts modules on the built model and skips only when there are none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Aria: record router_logits from the router's first output

After the router's outputs were reordered to (router_logits, top_k_weights, top_k_index), the OutputRecorder still read index 2, so router_logits returned the top-k indices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* DeepSeek-V4: give lm_head its TP plan back now that the model has one

huggingface#49314 removed it while DeepSeek-V4 was EP-only. With the experts-only TP plan, a tied head reads the row-sharded embedding, so lm_head needs colwise_gather_output again. Also drops the stale "EP only" comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Review: the TP mixin takes a VLM/ALM tester's conditional_generation_class, and finds experts before testing their plans

The multimodal tester no longer aliases causal_lm_class: the mixin falls back to conditional_generation_class itself. test_moe_parallel_plans_shard_experts collects the expert weights with _get_expert_weight_names and skips up front when a model has none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants