Repository navigation
[model/training] fix: handle config edge cases - #4104
Conversation
|
/ok to test a6767a7 |
| cfg.hidden_size = 2816 | ||
| cfg.intermediate_size = 2112 # shared expert FFN | ||
| cfg.moe_intermediate_size = 704 # routed expert FFN | ||
| cfg.moe_intermediate_size = 1408 # distinct from provider default to catch config leaks |
There was a problem hiding this comment.
Nit (non-blocking): mock_hf_config uses num_experts=128, top_k_experts=8, moe_intermediate_size=704 — identical to the Gemma4ModelProvider defaults. After the CONFIG_MAPPING exclusion, test_moe_config passes whether or not the enable_moe_block branch reads from HF config, because the provider defaults already match.
Consider using distinct values in the MoE fixture (e.g. num_experts=64, top_k_experts=4) so test_moe_config actually proves the HF values flow through the enable_moe_block branch.
|
Light Code Review - Both fixes are correct and well-scoped. See below for details. Gemma4 CONFIG_MAPPING exclusion - The base CONFIG_MAPPING entries (num_experts, num_moe_experts) and (moe_intermediate_size, moe_ffn_hidden_size) were overwriting Gemma4ModelProvider defaults for dense models (enable_moe_block=False) since hf_config_to_provider_kwargs runs unconditionally. Filtering these three megatron fields out of Gemma4 CONFIG_MAPPING and relying on the explicit enable_moe_block guard in provider_bridge() is the right fix. Good catch on the tense_dense_config to test_dense_config_keeps_default_moe_fields rename -- the old name meant the test never ran. Qwen3ASR thinker_config init ordering - Moving thinker_config construction before super().init() is correct -- Transformers PretrainedConfig.init can call get_text_config() during validation, which needs self.thinker_config to exist. The isinstance guard handles the from_dict path where sub_configs may pass an already-constructed Qwen3ASRThinkerConfig. Minor observation - See inline comment on test_gemma4_bridge.py:73 -- the MoE-enabled test fixture uses values identical to provider defaults, so test_moe_config cannot distinguish read-from-HF-config from fell-through-to-default. Non-blocking. Suggested test cases - No perf tests impacted. |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
a6767a7 to
483b0d8
Compare
|
/ok to test 483b0d8 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com>
Summary
Fixes two model-side config regressions and records the current-main status for the related TokenizerConfig issue:
enable_moe_block=False, using a reusable mapping-gate hook.thinker_configbefore Transformers validation callsget_text_config()during config initialization.TokenizerConfigalready has the MCore tokenizer padding fields on currentmainvia PR [training] fix: support MCore tokenizer padding fields #4075.NVBug Status
mainstatusGemma4Bridge.provider_bridge()still lets baseCONFIG_MAPPINGmap HF MoE fields before theenable_moe_blockguard.TokenizerConfignow hasmake_vocab_size_divisible_by,tensor_model_parallel_size, andrank.Qwen3ASRConfig.__init__()callsPretrainedConfig.__init__()beforethinker_configexists, and Transformers validation callsget_text_config()during that window.thinker_configbefore the base initializer and adding config construction coverage.Validation
ruff format src/megatron/bridge/models/conversion/model_bridge.py src/megatron/bridge/models/gemma/gemma4_bridge.py src/megatron/bridge/models/qwen3_asr/hf_qwen3_asr/configuration_qwen3_asr.py tests/unit_tests/models/gemma/test_gemma4_bridge.py tests/unit_tests/models/qwen3_asr/test_qwen3_asr_config.pyruff check --fix src/megatron/bridge/models/conversion/model_bridge.py src/megatron/bridge/models/gemma/gemma4_bridge.py src/megatron/bridge/models/qwen3_asr/hf_qwen3_asr/configuration_qwen3_asr.py tests/unit_tests/models/gemma/test_gemma4_bridge.py tests/unit_tests/models/qwen3_asr/test_qwen3_asr_config.pypython -m py_compile src/megatron/bridge/models/conversion/model_bridge.py src/megatron/bridge/models/gemma/gemma4_bridge.py src/megatron/bridge/models/qwen3_asr/hf_qwen3_asr/configuration_qwen3_asr.py tests/unit_tests/models/gemma/test_gemma4_bridge.py tests/unit_tests/models/qwen3_asr/test_qwen3_asr_config.pyQwen3ASRConfig(),Qwen3ASRConfig.from_dict(...), andAutoConfig.from_pretrained("Qwen/Qwen3-ASR-1.7B", trust_remote_code=True)now exposethinker_config.pre-commit run --all-filesAttempted targeted pytest with
uv run python -m pytest ...; local execution was blocked before collection by the platform wheel availability fornvidia-resiliency-ext.uv run pre-commit run --all-fileshit the same local resolver blocker, so the installed pre-commit hooks were run directly. No full suite was run.Notes
3rdparty/Megatron-LM/.