Skip to content

[training] fix: Stabilize NVRx straggler tests - #4279

Merged
yaoyu-33 merged 3 commits into
mainfrom
yuya/mcore-dev-autofix-20260610-pr4270
Jun 11, 2026
Merged

yaoyu-33 merged 3 commits into
mainfrom
yuya/mcore-dev-autofix-20260610-pr4270

Conversation

@yaoyu-33

Copy link
Copy Markdown
Contributor

Summary

Fixes the current daily MCore dev bump blocker from #4270.

Target: dev

Failure classification: Bridge broke itself

Root cause: PR #4270 resolves nvidia-resiliency-ext to 0.6.0.dev33+15a8515. In that dependency set, importing nvidia_resiliency_ext.attribution can fail before Bridge binds the module-level straggler symbol. The NVRx unit tests then attempted to patch megatron.bridge.training.nvrx_straggler.straggler without first forcing the mocked NVRx-available state, so Launch_Unit_Tests_Core failed with AttributeError: ... does not have the attribute 'straggler'.

Fix summary:

  • Keep a module-level straggler placeholder when the optional NVRx import is unavailable, preserving the existing runtime behavior where enabled NVRx raises ImportError if the dependency cannot be imported.
  • Update NVRx unit-test fixtures to patch both HAVE_NVRX=True and the mocked straggler API, matching the mocked dependency path they exercise.

Guards added or removed: none.

Validation:

  • CW interactive partition, standard Bridge container: uv run --no-sync python -m pytest tests/unit_tests/training/test_nvrx_straggler.py -q -> 32 passed, 32 warnings in 0.85s on 2026-06-10.
  • CW interactive partition, standard Bridge container: uv run --no-sync pre-commit run --all-files -> passed end-of-file, trailing-whitespace, ruff, ruff-format on 2026-06-10.

Notes:

dimapihtar and others added 2 commits June 10, 2026 06:47
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Yu Yao <yaoyu.094@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 10, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test fa971b0

@claude

claude Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

LGTM

The fix is correct and well-scoped:

  • The module-level straggler: Any | None = None placeholder ensures patch() can find the target attribute even when the NVRx import fails. Runtime safety is preserved because NVRxStragglerDetectionManager.__init__ gates all straggler.* usage behind the HAVE_NVRX check.
  • The test fixture updates correctly co-patch HAVE_NVRX=True alongside the straggler mock, matching the new import semantics.

Suggested test cases

No perf tests impacted.

@yaoyu-33 yaoyu-33 added area:training Training loop, callbacks, and runtime integration bug Something isn't working full-test-suite needs-review PR is ready for code review and waiting on a reviewer labels Jun 10, 2026
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
@yaoyu-33

Copy link
Copy Markdown
Contributor Author

Updated the branch to remove the accidental dependency bump artifacts (.dev.commit, 3rdparty/Megatron-LM, and uv.lock), so the PR diff is now scoped to the NVRx straggler test stabilization.

The two WAN functional failures currently visible in CI (L0_Launch_models_wan and gb200_L0_Launch_models_wan) are known issues and are unrelated to this PR.

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test ba7c55ce9d1e0ab614d3f83136f6603fd38203ba

@copy-pr-bot

copy-pr-bot Bot commented Jun 10, 2026

Copy link
Copy Markdown

/ok to test ba7c55ce9d1e0ab614d3f83136f6603fd38203ba

@yaoyu-33, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

@yaoyu-33

Copy link
Copy Markdown
Contributor Author

/ok to test ba7c55c

@yaoyu-33
yaoyu-33 merged commit b24a2f8 into main Jun 11, 2026
163 of 166 checks passed
@yaoyu-33
yaoyu-33 deleted the yuya/mcore-dev-autofix-20260610-pr4270 branch June 11, 2026 02:29

This branch was previously deployed

2 inactive deployments
public — ba7c55ce Deployed Jun 10, 2026 by copy-pr-bot[bot] via release / finalize / publish-docs #1337
test — ba7c55ce Deployed Jun 10, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #16098
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:training Training loop, callbacks, and runtime integration bug Something isn't working full-test-suite needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants