Repository navigation
[training] fix: Stabilize NVRx straggler tests - #4279
Merged
Merged
Conversation
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Yu Yao <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test fa971b0 |
Contributor
|
LGTM The fix is correct and well-scoped:
Suggested test cases No perf tests impacted. |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
Updated the branch to remove the accidental dependency bump artifacts ( The two WAN functional failures currently visible in CI ( |
Contributor
Author
|
/ok to test ba7c55ce9d1e0ab614d3f83136f6603fd38203ba |
@yaoyu-33, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
Contributor
Author
|
/ok to test ba7c55c |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the current daily MCore dev bump blocker from #4270.
Target:
devFailure classification: Bridge broke itself
Root cause: PR #4270 resolves
nvidia-resiliency-extto0.6.0.dev33+15a8515. In that dependency set, importingnvidia_resiliency_ext.attributioncan fail before Bridge binds the module-levelstragglersymbol. The NVRx unit tests then attempted to patchmegatron.bridge.training.nvrx_straggler.stragglerwithout first forcing the mocked NVRx-available state, soLaunch_Unit_Tests_Corefailed withAttributeError: ... does not have the attribute 'straggler'.Fix summary:
stragglerplaceholder when the optional NVRx import is unavailable, preserving the existing runtime behavior where enabled NVRx raisesImportErrorif the dependency cannot be imported.HAVE_NVRX=Trueand the mockedstragglerAPI, matching the mocked dependency path they exercise.Guards added or removed: none.
Validation:
uv run --no-sync python -m pytest tests/unit_tests/training/test_nvrx_straggler.py -q->32 passed, 32 warnings in 0.85son 2026-06-10.uv run --no-sync pre-commit run --all-files-> passed end-of-file, trailing-whitespace, ruff, ruff-format on 2026-06-10.Notes:
uv run python -m pytest tests/unit_tests/training/test_nvrx_straggler.py -qwas blocked before pytest by local dependency setup for optional CUDA packages. The focused CW validation above used the prepared container environment and the same PR chore(beep boop 🤖): Bumpuv.lock(main, mcore-dev) (2026-06-10) #4270 MCore dev source.