[yaoyu-33]

Summary

Add the MCore tokenizer padding fields that Bridge's TokenizerConfig needs when it is passed through megatron.core.tokenizers.utils.build_tokenizer:

• make_vocab_size_divisible_by
• tensor_model_parallel_size
• rank

The defaults are neutral for Bridge checkpoint conversion because Bridge applies model vocab padding separately.

Testing

• uv run --active --no-sync pre-commit run --all-files
• uv run --active --no-sync ruff check src/megatron/bridge/training/tokenizers/config.py tests/unit_tests/training/test_model_load_save.py
• uv run --active --no-sync ruff format --check src/megatron/bridge/training/tokenizers/config.py tests/unit_tests/training/test_model_load_save.py
• uv run --active --no-sync python -m py_compile src/megatron/bridge/training/tokenizers/config.py tests/unit_tests/training/test_model_load_save.py

Targeted pytest was attempted locally, but this workstation Python environment is missing Transformer Engine, which prevents importing the full megatron.bridge package before the selected test starts.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[claude]

Light Code Review

Clean, focused change. The three new fields (make_vocab_size_divisible_by, tensor_model_parallel_size, rank) are required by MCore's build_tokenizer path, and the defaults (1, 1, 0) are neutral — they won't affect existing callers or recipes that don't pass these kwargs.

No bugs, typos, or missing coverage found.

One minor observation: the test at tests/unit_tests/training/test_model_load_save.py:1096 uses Mock(spec=TokenizerConfig), which means hasattr checks will pass for the new dataclass fields. This is correct behavior, but note the mock won't run __post_init__, so the test only verifies the setattr path in load_tokenizer, not the full config initialization. This is fine for the scope of this PR.

Suggested test cases

No perf tests impacted.

[yaoyu-33]

/ok to test edfcc3e