Skip to content

NVIDIA Megatron-Bridge 0.4.0

Choose a tag to compare

View all tags
@svcnvidia-nemo-ci svcnvidia-nemo-ci released this 16 Apr 22:46
v0.4.0
0fbfe7d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

Model Collection Support

  • MiniMax M2 / M2.5 support (PR#2602)
  • Kimi 2.5 support, including GB300 MXFP8 recipe and HF config updates (PR#2743)
  • Nemotron 3 Super model support (PR#2912)
  • Sarvam support (PR#1814)
  • Qwen 3.5 VL Bridge with recipes and LoRA bridge / merge support (PR#2530, PR#2654, PR#2736)
  • Qwen 2.5 Omni support (PR#2634)
  • Qwen2-Audio support (PR#2324)
  • Xiaomi MiMo dense MTP model bridge support (PR#2387, by HollowMan6)

Diffusion Collection

Training & Functionality

  • Parquet support for sequence-packing preprocessing, improving handling of larger datasets (PR#2395)
  • Energon integration for sequence packing with WebDataset workflows (PR#2440)
  • Default packed sequences across finetune recipes (PR#2284)
  • More modern finetuning datasets, including OpenMathInstruct V2 and GSM8K (PR#2264)
  • Unified dataset configuration in run_recipe.py (PR#2826)
  • NCCL flight recorder configuration support (PR#2891)
  • Comet ML experiment tracking integration (PR#2910)
  • Refactored SFT and PEFT recipes for VLM workflows (PR#2614)
  • Added the on_checkpoint_save callback event for training workflows (PR#2905)
  • Added MoE LoRA rank normalization for expert layers (PR#3006)
  • Direct export of block-wise FP8 weights and scaling factors (PR#1994)
  • Accelerated first-fit packing with a segment tree for much faster packing on large datasets (PR#2953)

Model Optimization

  • Pruning support and documentation (PR#2244)
  • Post-training quantization support for Nano, Super, and Ultra model families (PR#2303)
  • Distillation quantization support in NeMo 2 (PR#2591)

Performance

  • Nemotron 3 Super perf config, including GB200 improvements and BF16 / NVFP4 functional support via module recompute (PR#3208)

Developer Experience & Compatibility

  • ModelConfig and ModelBuilder refactor integrated into the training loop (PR#2798, PR#2671)
  • Dev branch support and documentation updates (PR#2497)
  • Python 3.12 migration announcement (PR#2773)
  • Transformers 5.0 through 5.3 compatibility (PR#2068, PR#2781)
  • PEFT Bridge offline mode support (PR#2574)
  • LoRA merge on CPU (PR#2194)
  • Self-contained Megatron-to-HF export with auto-config synthesis (PR#2778)
  • Scripts and documentation for Megatron-LM and Megatron Bridge correlation

Examples & Tutorials

Community Contributions

  • @HollowMan6 (Aalto University): Xiaomi MiMo dense MTP bridge support, Qwen 3.5 VL LoRA bridge and merge, and additional export / PEFT fixes (PR#2387, PR#2736, PR#2384, PR#2799)
  • @shaltielshmid: packed-sequence improvements for large datasets and safer model loading defaults (PR#2395, PR#2766)
  • @jaeminh: accelerated first-fit packing with a segment tree (PR#2953)
  • @pavelgein: added the on_checkpoint_save callback event (PR#2905)
  • @ShiftyBlock (UC Berkeley): added auto-config for self-contained Megatron-to-HF export (PR#2778)
  • @erictang000 (Anyscale): added LoRA rank normalization for MoE expert layers (PR#3006)
  • @eternally-z: added direct export support for block-wise FP8 weights and scaling factors (PR#1994)
  • @Hayak3: fixed the unsupported normalization argument for Qwen3-VL (PR#1970)
  • @mohit-sarvam (Sarvam AI): added Sarvam MoE support (PR#1814)

A big thank you to our community contributors for their valuable support!

Changelog Details
  • docs: Update callback code snippets to include all imports needed for example by @ananthsub :: PR: #2283
  • M4 leftover for QWen3-VL with MCore vision encoder by @shifangx :: PR: #2370
  • Update Qwen3 235B B300 Configs to match Qwen3 B200 Configs by @rhmukundan :: PR: #2669
  • [bridge] Fix off-by-one in sliding window size for Gemma2, Gemma3, Mistral, and GPT-OSS by @cuichenx :: PR: #2656
  • fix: Write intermediate results to tmp by @ko3n1g :: PR: #2726
  • Perf recipe dataloader num_workers interface fix by @dingqingy-nv :: PR: #2710
  • Suppress noisy _extra_state warnings during checkpoint loading by @cuichenx :: PR: #2689
  • [model, recipe] Add Qwen 3.5 recipes by @cuichenx :: PR: #2654
  • [ci] chore: add nightly dev commit bump workflow by @ko3n1g :: PR: #2729
  • ci(fix): Unique naming for dev branch by @ko3n1g :: PR: #2747
  • [ci] Refactor Gemma3-VL launch script to run finetune and packed tests separately by @cuichenx :: PR: #2730
  • add qwen2_5_omni by @yuekaizhang :: PR: #2634
  • build: Bump TE 2.13 by @ko3n1g :: PR: #2753
  • [docs, ci] chore: add governance issue forms and triage guide by @yaoyu-33 :: PR: #2716
  • [test] fix: temporarily disable qwen2.5 omni unit tests by @yaoyu-33 :: PR: #2759
  • add nemotron3 super docs by @liding-nv :: PR: #2757
  • ci: Fix stopiteration for Mbridge by @ko3n1g :: PR: #2760
  • GPT-OSS Blackwell MXFP8 recipes by @weijiac0619 :: PR: #2633
  • feat(mimo): phase 2 - model provider, DDP wrapping, process groups by @aroshanghias-nvd :: PR: #2004
  • [build] feat: add OSS NeMo FW dockerfiles by @thomasdhc :: PR: #2722
  • Lm3 70B GB200 FP8_CS SFT cfg update by @malay-nagda :: PR: #2748
  • [docs] chore: use uv run in test file docstring run instructions by @cuichenx :: PR: #2728
  • build: Bump NVRX by @ko3n1g :: PR: #2775
  • NVFP4 memory spike fix compared to M-LM by @sanandaraj5597 :: PR: #2764
  • [doc] feat: Document adapter merge verification in stream_adapter_weights example by @yaoyu-33 :: PR: #2042
  • [doc] chore: Add needs-review to PR state labels guidance by @yaoyu-33 :: PR: #2758
  • [ckpt] fix: broaden exception handling in save_artifacts dynamic module loading by @yaoyu-33 :: PR: #2765
  • [test] fix: use toy configs in qwen2.5 omni unit tests by @yaoyu-33 :: PR: #2761
  • [model] Refactor Qwen3-VL and Ministral3 fine-tuning scripts by @kamran-nvidia :: PR: #2735
  • docs - Update user manual with new MoE features and Megatron FSDP by @onel :: PR: #2529
  • remove encoder_and_decoder usage by @dimapihtar :: PR: #2512
  • Fix attention_mask mismatch in compare.py by @mohsinm-dev :: PR: #2476
  • [model, test] fix: guard hybrid layer count across MCore branches by @yaoyu-33 :: PR: #2776
  • [data] fix: guard eval_interval division to prevent ZeroDivisionError by @yaoyu-33 :: PR: #2732
  • [sync][training] fix: log loss values of exactly 0.0 in training_log() by @mehraakash :: PR: #2740
  • [model] feat: support Qwen 3.5 MTP conversion and training by @cuichenx :: PR: #2711
  • [model] feat: add Rubin model support by @malay-nagda :: PR: #2705
  • docs: Add readme to docker by @thomasdhc :: PR: #2777
  • fix: Allow building trt-llm 1.2.0 by @chtruong814 :: PR: #2790
  • [ckpt] feat: GPU-accelerated checkpoint conversion by @yaoyu-33 :: PR: #2767
  • [data] fix: Remove spurious EOS append in _chat_preprocess by @shanecmoran :: PR: #2699
  • Revert "[ckpt] feat: GPU-accelerated checkpoint conversion (#2767)" by @cuichenx :: PR: #2792
  • fix: disable MoE overlap and delayed wgrad during model load by @shaltielshmid :: PR: #2766
  • [ckpt] fix: transformers 5.x compat shim + dtype mismatch warning cleanup by @yaoyu-33 :: PR: #2794
  • [DFM Migration] Migration DFM -> Bridge by @abhinavg4 :: PR: #2534
  • Onboard NVFP4 and MXFP8 recipes by @dingqingy-nv :: PR: #2600
  • fix: GPTSFTChatDataset.collate_fn use pad_seq_length_to_mult instead of hardcoded 16 by @shanecmoran :: PR: #2643
  • build: Update vllm patch and unpin transformers in FW container by @chtruong814 :: PR: #2800
  • fix: force os._exit in pytest_sessionfinish to avoid PyTorch teardown crash by @ko3n1g :: PR: #2818
  • [peft][ckpt] feat: add HF PEFT adapter export for LoRA/DoRA checkpoints by @yaoyu-33 :: PR: #2574
  • Fix uv.lock by @ko3n1g :: PR: #2821
  • cuda_sw_mxfp8_nsys by @malay-nagda :: PR: #2815
  • chore(beep boop 🤖): Bump uv.lock (main) (2026-03-16) by @svcnvidia-nemo-ci :: PR: #2813
  • [model, recipe, test] refactor: remove specific model provider dataclasses by @yaoyu-33 :: PR: #2599
  • Improving packed sequences SFT for large datasets by @shaltielshmid :: PR: #2395
  • fix: Add get_hf_tokenizer_kwargs to NemotronHBridge by @shanecmoran :: PR: #2664
  • [model] fix: Qwen3.5 VL MoE EP conversion and vision dtype warnings by @cuichenx :: PR: #2804
  • Mup scaling by @mehraakash :: PR: #2666
  • [model,recipe] fix: use dynamic gradient_accumulation_fusion check by @yaoyu-33 :: PR: #2739
  • [recipe] feat: Add MixedPrecisionConfig to model provider pathway. by @cspades :: PR: #2784
  • [ci] fix: Increase unit tests timeout from 15 to 18 minutes by @yaoyu-33 :: PR: #2827
  • [recipe, test] fix: guard muon_nesterov rename and allow dev submodule in commit test by @yaoyu-33 :: PR: #2823
  • [sync][training] fix: backward compat for hybrid_override_pattern in _load_args_from_checkpoint by @mehraakash :: PR: #2742
  • [ci] refactor: tier functional tests into L0/L1/L2 by @yaoyu-33 :: PR: #2715
  • allow multiple comma separated env var values by @malay-nagda :: PR: #2834
  • Announce Python 3.12 migration by @ko3n1g :: PR: #2773
  • profiling_trace_fix- all precisions by @malay-nagda :: PR: #2839
  • [ci] feat: split unit tests into core and diffusion buckets by @ko3n1g :: PR: #2833
  • Bump dependencies to address CVEs by @ko3n1g :: PR: #2832
  • chore: Remove pytest_sessionfinish by @ko3n1g :: PR: #2840
  • ci: Enable claude review by @thomasdhc :: PR: #2845
  • [ckpt] refactor: Consolidate fused expert mappings and fix MTP inference by @yaoyu-33 :: PR: #2685
  • enable nsys option for rubin bash by @malay-nagda :: PR: #2876
  • docs: Improve package installation guidance in FW container README by @ko3n1g :: PR: #2877
  • ci(fix): Dependabot PR with mcore-branch in title by @ko3n1g :: PR: #2878
  • [misc] fix: guard eod_id access in compare_text_generation for HF tokenizers by @yaoyu-33 :: PR: #2854
  • [fix] fix example scripts by @cuichenx :: PR: #2866
  • [fix] Fix recipe names in docs by @cuichenx :: PR: #2864
  • [training,ci] fix: guard get_mup_config_overrides import for mcore main compat by @yaoyu-33 :: PR: #2846
  • Revert MFSDP mixed precision config arguments. by @cspades :: PR: #2848
  • ci: Run L0+L1 tests on dependency bump PRs by @ko3n1g :: PR: #2887
  • [inference] fix: use config.hidden_size directly for Qwen3VL inference wrapper by @yaoyu-33 :: PR: #2856
  • [recipe] fix: pass trust_remote_code=True for Kimi-K2 HF config loading by @yaoyu-33 :: PR: #2881
  • NaN grad gone, re-enable VP by @dingqingy-nv :: PR: #2850
  • [example] fix: use AutoBridge directly in distill_llama32_3b-1b to load HF weights by @yaoyu-33 :: PR: #2859
  • ci: Fix Coverage/Coverage_Fake always being skipped by @ko3n1g :: PR: #2880
  • chore(beep boop 🤖): Bump uv.lock (main) (2026-03-18) by @svcnvidia-nemo-ci :: PR: #2873
  • [training] fix: cap lr_warmup_steps to be strictly less than lr_decay_steps by @yaoyu-33 :: PR: #2857
  • ci: Skip coverage job for mcore triggered tests by @chtruong814 :: PR: #2897
  • Simplify distill example script by @AAnoosheh :: PR: #2865
  • Fix expert number computation in param_mapping for SequentialMLP by @kevalmorabia97 :: PR: #2888
  • Support transformers v5.3.0 by @liding-nv :: PR: #2781
  • fix(ci): Fix dev branch submodule revert in update-dependencies workflow by @ko3n1g :: PR: #2907
  • Remove check that does not allow gradient_accumulation_fusion by @gautham-kollu :: PR: #2851
  • Delete tests/functional_tests/L2_Mcore_Mock_Tests_GPU.sh by @huvunvidia :: PR: #2801
  • [sync][training] fix: free overlap buffers and empty cache before checkpoint save by @mehraakash :: PR: #2741
  • [model] fix: Dequantize FP8 weights during Ministral3 import and fix model init issues by @cuichenx :: PR: #2895
  • [training] fix: use startswith for Blackwell GPU name validation in DeepEP by @yaoyu-33 :: PR: #2731
  • Integrate ModelConfig & ModelBuilder interface into training loop by @maanug-nv :: PR: #2798
  • deepseek: add conversion support for MTP module by @denys-fridman :: PR: #1886
  • ci: dynamic test matrix from launch_scripts/ directory scan by @ko3n1g :: PR: #2904
  • [test] fix: Speed up test_collate_fn_respects_pad_seq_length_to_mult by @cuichenx :: PR: #2923
  • [doc] refactor: split technique guidance into docs, skills, and knowledge by @yaoyu-33 :: PR: #2809
  • feat: Add first-class Comet ML experiment tracking by @cuichenx :: PR: #2910
  • [misc] feat: Add bidirectional MLM <-> MBridge config translation script by @yaoyu-33 :: PR: #2630
  • Fix Megatron->HF export when PP>1 for Qwen3.5 VL MoE models with MTP enabled by @HollowMan6 :: PR: #2799
  • fix: use HF-style attn mask path for qwen3-vl RoPE in BSHD training by @eagle705 :: PR: #2582
  • LoRA bridge & merge for Qwen3.5 by @HollowMan6 :: PR: #2736
  • [sync][training] feat: add NCCL flight recorder configuration support by @cuichenx :: PR: #2891
  • feat(mimo): phase 3 - MIMO data loading scaffolding by @aroshanghias-nvd :: PR: #2007
  • ci: Add full-test-suite label to trigger both L1 and L2 test suites by @ko3n1g :: PR: #2908
  • 2602_perf_table_reorder and patch container data by @malay-nagda :: PR: #2872
  • [model] fix: restore position_embedding_type and gradient_accumulation_fusion for MLA models by @yaoyu-33 :: PR: #2927
  • chore: bump FW-CI-templates to v0.80.2 by @ko3n1g :: PR: #2920
  • docs: Add 0.3.1 docs by @ko3n1g :: PR: #2939
  • Remove 'preferred' field from version 0.3.0 by @ko3n1g :: PR: #2942
  • perf optimizations- GPT-OSS 120B by @malay-nagda :: PR: #2906
  • [perf] feat: add GDN (Gated DeltaNet) FLOPs calculator by @cuichenx :: PR: #2925
  • fix: Fix fsdp config check by @BoxiangW :: PR: #2266
  • [doc] chore: remove duplicate moe-comm-overlap skill by @yaoyu-33 :: PR: #2943
  • [model] feat: add MiniMax-M2 MoE bridge by @yaoyu-33 :: PR: #2602
  • [ckpt] feat: use explicit checkpoint format by @denys-fridman :: PR: #2702
  • [ckpt] fix: minor quantization export issue by @yueshen2016 :: PR: #2700
  • [misc] feat: mlflow improvements by @ryxli :: PR: #2153
  • [training, fix] Support packed sequence packing with pipeline parallelism by @cuichenx :: PR: #2825
  • [examples][model] fix: add Qwen2.5-Omni examples and fix vision encoder inference by @yaoyu-33 :: PR: #2965
  • [examples] docs: add sarvam-30b conversion and inference examples by @yaoyu-33 :: PR: #2963
  • [data] Add openmathinstruct and gsm8k datasets by @cuichenx :: PR: #2264
  • ci: upgrade GitHub Actions for Node.js 24 compatibility by @ko3n1g :: PR: #2947
  • Lm3 70B SFT OOM fix by @malay-nagda :: PR: #2971
  • feat: Add support for Qwen2-Audio by @yuekaizhang :: PR: #2324
  • ci: Ensure generate-test-matrix is ran on merge to main and fix MCore trigger by @chtruong814 :: PR: #2962
  • ci: Allow aws ofi reinstall to be optional by @thomasdhc :: PR: #2975
  • fix: correctly handle rope_scaling dicts for MLA providers by @ZhiyuLi-Nvidia :: PR: #2791
  • chore(beep boop 🤖): Bump uv.lock (main, mcore-main) (2026-03-24) by @svcnvidia-nemo-ci :: PR: #2968
  • [DFM Migration] Wan Mrbdige scripts improvement by @huvunvidia :: PR: #2849
  • [recipe] feat: add yarn + long context recipe by @RayenTian :: PR: #2951
  • [DFM Migration] Remove FLUX pretraining and fine-tuning scripts and Use the run_recipe by @suiyoubi :: PR: #2822
  • fix: VLM packed sequence padding and SP scatter ordering for CP+SP by @cuichenx :: PR: #2932
  • [data] feat: Generic Energon task encoder for HF-encoder VLMs by @cuichenx :: PR: #2782
  • [example] feat: Add Qwen3.5-VL 35B SFT sbatch launch scripts (FSDP + … by @DAISY-gh :: PR: #2902
  • [misc] fix: Change cached_property to property for _causal_lm_architecture by @guyueh1 :: PR: #2070
  • [data] fix: Write full process_example_fn output to JSONL in HFDatasetBuilder by @shanecmoran :: PR: #2612
  • Allow custom checkpoint impl via checkpoint manager interface by @ananthsub :: PR: #1955
  • ci: Update create pull requests version by @thomasdhc :: PR: #2984
  • [model] refactor: Remove individual qwen providers by @yaoyu-33 :: PR: #2556
  • nemotron-3-super model support by @liding-nv :: PR: #2912
  • Use uv run in mlm checkpoint tests by @maanug-nv :: PR: #2994
  • ci: Use AWS image registry when using AWS runners and add link check retries by @chtruong814 :: PR: #3002
  • build: Bump TE to 2.14 by @ko3n1g :: PR: #2980
  • ci: Remove per-unit-test timeout and do not uv sync during test by @chtruong814 :: PR: #3010
  • chore(beep boop 🤖): Bump uv.lock (main, mcore-main) (2026-03-29) by @svcnvidia-nemo-ci :: PR: #3016
  • dsv3, lm3 8b intro for vr200 by @malay-nagda :: PR: #3019
  • vr200_dsv3_version_fix by @malay-nagda :: PR: #3020
  • Muon config update by @janEbert :: PR: #2959
  • [data, perf] feat: Accelerate first_fit packing with segment tree by @jaeminh :: PR: #2953
  • Change pytest fixture to not autouse by @cuichenx :: PR: #2976
  • [model] fix: use dynamic gradient_accumulation_fusion checker in MambaModelProvider by @yaoyu-33 :: PR: #2958
  • Import migrated dataclasses by @maanug-nv :: PR: #2941
  • kimi-k2 optimizer type by @malay-nagda :: PR: #3018
  • nemtron 3 super intro- GB200, B300, B200 by @malay-nagda :: PR: #3025
  • [misc] fix: preserve multi-value cuda-graph-scope in Bridge→MLM translation by @eagle705 :: PR: #3027
  • [doc] feat: add Nemotron Super news and compact model support list in README by @yaoyu-33 :: PR: #3028
  • chore: Move to Py3.12 by @ko3n1g :: PR: #2774
  • Revert "ci: Remove per-unit-test timeout and do not uv sync during te… by @ko3n1g :: PR: #3030
  • [docs] chore: Add docs/tests checklist and refactor PR guidance to CONTRIBUTING.md by @yaoyu-33 :: PR: #2890
  • [ci] feat: auto-remove needs-author label on author reply or push by @yaoyu-33 :: PR: #3033
  • [docs] chore: Add area:diffusion label and test resource constraints by @yaoyu-33 :: PR: #3036
  • [docs] feat: Add public skills and developer guide by @yaoyu-33 :: PR: #2894
  • [doc] fix: update Nemotron Super v3 README links from super-v3 branch to main by @yaoyu-33 :: PR: #3038
  • [doc] fix: update Nemotron Nano v3 and dev branch links to main by @yaoyu-33 :: PR: #3039
  • chore: Bump Python3.12 also for FW container by @ko3n1g :: PR: #3044
  • build: Set NVTE_BUILD_NUM_PHILOX_ROUNDS=3 by @ko3n1g :: PR: #3045
  • build: Fix NVRX build issue by @ko3n1g :: PR: #3046
  • Reapply "ci: Remove per-unit-test timeout and do not uv sync during t… by @ko3n1g :: PR: #3050
  • cp: perf: add KubeflowExecutor support to scripts/performance (3024) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3055
  • cp: ci: Bump build-test workflow (3057) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3058
  • cp: Fix DSV3 gb200 NVFP4 NaN grad norm and improve mappings (3031) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3061
  • cp: [DFM] Add Data Prep guide in README (3052) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3064
  • cp: restore bf16 mapping for OOM (3067) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3073
  • cp: [model] fix: FusedExpertMapping and FusedGatedExpertMapping fail with TP > 1 (3035) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3074
  • cp: [build] fix: pre-install grpcio-tools before nvidia-resiliency-ext 0.6.0 build (3085) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3087
  • cp: [docker] fix: patch wandb-core grpc 1.79.2 → 1.79.3 (CVE) (3082) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3088
  • Cherry Pick: Dump the env for rank0 for slurm based runs. (#2991) by @sudostock :: PR: #3097
  • cp: qwen_fix_env_var_2604 (3083) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3090
  • cp: build: Bump langchain to address CVE (3109) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3113
  • bump submodule by @ko3n1g :: PR: #3054
  • cp: kimi k2.5 support (2743) into r0.4.0 by @yaoyu-33 :: PR: #3093
  • cp: [perf] fix: set NUM_OF_TOKENS_PER_CHUNK_COMBINE_API=128 for all hybrid-ep recipes (3135) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3149
  • cp: Addressing .mp4 files that don't have captions when running openvid1M_preprocess.py (3120) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3146
  • cp: [doc] fix: add missing GLM-4.5V and Qwen3.5 VL entries to VLM README table (3124) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3126
  • cp: [model] fix: MiniMax-M2 dtype mismatch and Sarvam PP=2 uneven pipeline (3041) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3122
  • cp: [model] fix: pass mm_token_type_ids to Qwen2.5-VL get_rope_index (3040) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3121
  • cp: [recipe] Add pretrain configs with mock dataset for Qwen3-VL and Qwen3.5-VL (3100) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3119
  • cp: [training] fix: validate hierarchical_context_parallel_sizes for a2a+p2p cp_comm_type (2665) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3116
  • cp: explicit_dispatcher_perf_scripts (2837) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3103
  • cp: [ci] feat: fix Qwen3 PEFT verify test and add missing launch scripts (3070) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3150
  • cp: [doc] fix: update README quickstart, supported models, and known issues (#3060) into r0.4.0 by @yaoyu-33 :: PR: #3089
  • [ci] fix: Update .main.commit for r0.4.0 mcore bump by @yaoyu-33 :: PR: #3181
  • cp: Switch to new offloading path (3193) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3194
  • cp: Fix distill docs typo (3118) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3183
  • cp: [model] fix: use vocab_size in resiliency examples instead of missing padded_vocab_size (3125) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3186
  • cp: fix: OOM for llama31_405b gb300 fp8 pretrain (3188) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3199
  • cp: Fix NT 3 Super NVFP4 perf config: restore model recipe fields and add te_quant.cfg (3095) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3196
  • build(fix): CVE security patches by @ko3n1g :: PR: #3182
  • cp: perf: Set expandable_segments for qwen3_next_80b H100 fp8_cs to fix OOM (3195) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3215
  • chore(beep boop 🤖): Bump uv.lock (r0.4.0, mcore-core_r0.17.0) (2026-04-07) by @svcnvidia-nemo-ci :: PR: #3191
  • fix(ci): update .main.commit for non-dev mcore branches by @ko3n1g :: PR: #3214
  • cp: fix(perf): detect training completion for normal runs that hang on teardown (3221) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3222
  • cp: Address LLAMA3 70B LoRa OOM (3223) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3231
  • cp: fix(evaluation): lazy-import KubeRayExecutor to avoid hard kubernetes dependency (3217) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3224
  • cp: Add --packager flag for performance scripts. (2893) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3202
  • cp: add container_env to SlurmExecutor for sbatch workloads (3034) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3227
  • cp: fix: quantization calibration race condition, QAT config, and doc (3229) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3237
  • cp: Clarify HF offline behavior with --offline flag (2847) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3240
  • cp: nt super cleanup (#3208) into r0.4.0 by @yaoyu-33 :: PR: #3232
  • recompute=None by @malay-nagda :: PR: #3246
  • perf: Set micro_batch_size=2 for qwen3 235B B300 V2 configs by @rhmukundan :: PR: #3251
  • cp: Fix VLM inference and add functional tests (3178) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3253
  • cp: preserve dtype in Kimi-K2.5 INT4 roundtrip quantization into r0.4.0 by @ko3n1g :: PR: #3257
  • cp: [perf] feat: Qwen3-Next 80B GB200/GB300 parallel mappings (3168) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3205
  • chore(beep boop 🤖): Bump uv.lock (r0.4.0, mcore-core_r0.17.0) (2026-04-09) by @svcnvidia-nemo-ci :: PR: #3241
  • cp: recover qwen3_vl perf script (3270) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3292
  • cp: [data] fix: enable WAN sequence packing by default to fix rope_utils crash (3254) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3272
  • fix(docker): pin peft>=0.18.1 in fw container by @ko3n1g :: PR: #3301
  • cp: [ckpt] fix: Unwrap tuple output in Nemotron VLM forward step (3201) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3302
  • cp: Add Qwen3.5-VL MoE pretrain performance configs (3101) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3304
  • cp: [ckpt] fix: Use device_map=\cuda\ for HF model in compare.py on multi-GPU nodes (3200) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3306
  • fix(docker): remove NeMo checkout step from fw_final Dockerfile by @ko3n1g :: PR: #3308
  • Bump uv.lock to include modelopt 0.43.0rc4 bug fixes by @kevalmorabia97 :: PR: #3303
  • fix(docker): remove nemo-toolkit audio/tts deps from fw_pyproject.toml by @ko3n1g :: PR: #3309
  • cp: [model, ckpt] fix: fix Qwen3 VL MoE fused expert import with ETP (3278) into r0.4.0 by @svcnvidia-nemo-ci :: PR: #3310
  • cp: Update release workflow to use v0.92.0 into r0.4.0 by @ko3n1g :: PR: #3360
  • beep boop 🤖: Bumping megatron.bridge to v0.4.1 by @nemo-automation-bot[bot] :: PR: #3361
  • Update PATCH version to 0.4.0 by @ko3n1g :: PR: #3362

Contributors

onel, sudostock, and 42 other contributors
Assets 2
Loading