[zqinglin]

What does this PR do?

While running a multi-teacher on-policy distillation job (reverse-KL distillation of a 4B student), training reliably froze after ~5-8 steps: all GPUs dropped to 0% util and it looked like an sglang/NCCL deadlock.

sglang/NCCL turned out to be a red herring — the NCCL watchdog never fired, and the sglang scheduler was just idle-spinning in its recv_requests() loop waiting for the next request. A SIGABRT + faulthandler dump showed the driver was actually stuck in reward scoring:

ray_trainer.fit
 -> trainer/ppo/reward.py: compute_reward
 -> workers/reward_manager/naive.py: NaiveRewardManager.__call__
 -> utils/reward_score/math_dapo.py: compute_score -> is_correct_minerva -> normalize_final_answer

The student had drifted into generating long, degenerate responses, and one such sample made a single compute_score call extremely slow inside normalize_final_answer. Because NaiveRewardManager scores samples serially in the driver process with no timeout, that one stuck call blocks the entire training loop with no recovery. Swapping compute_score for a no-op made the deterministic hang disappear, confirming reward computation was the blocker.

PrimeRewardManager already guards against this with a per-task timeout; NaiveRewardManager does not. This PR adds an opt-in per-sample timeout to NaiveRewardManager. Default is None (no timeout), so existing behavior is unchanged.

Related: #2236, #4318

Checklist Before Starting

• Search for similar PRs: https://github.com/volcengine/verl/pulls?q=is%3Apr+reward+timeout
• Title formatted as [reward] fix: ...

Test

Added CPU unit tests in tests/workers/reward_manager/test_naive_on_cpu.py:

• _score_timeout raises TimeoutError on a slow block and is a no-op when disabled (None/0).
• NaiveRewardManager with a 10s mock compute_score and compute_score_timeout=1.0 returns in ~1s with reward 0.0 instead of blocking.
• Default (no timeout) behavior is unchanged.

pytest tests/workers/reward_manager/test_naive_on_cpu.py -v   # 5 passed
ruff check / ruff format --check                              # clean

Also verified in verl's actual execution context: running the patched NaiveRewardManager inside a Ray actor with a 30s mock compute_score and compute_score_timeout=2.0 returns in ~2.0s with reward 0.0. This confirms the SIGALRM-based timeout fires in the Ray actor's main thread, which is where the reward manager runs in the training loop.

API and Usage Example

New optional kwarg compute_score_timeout (seconds), settable via reward_model.reward_kwargs:

NaiveRewardManager(tokenizer, num_examine, compute_score=fn, compute_score_timeout=10.0)

When a sample exceeds the timeout it is assigned reward 0.0 (with a warning) and training continues.

Design & Code Changes

• _score_timeout(seconds): a context manager built on SIGALRM. It degrades to a no-op off the main thread (where SIGALRM is unavailable), so it never crashes.
• NaiveRewardManager.__init__ gains compute_score_timeout=None.
• The per-sample compute_score call is wrapped; on TimeoutError the sample gets reward 0.0 + a warning, and the loop continues.

Note: SIGALRM only fires on the main thread, which is where NaiveRewardManager runs today. Happy to switch to a process-pool approach (like PrimeRewardManager) if maintainers prefer a thread-agnostic mechanism.

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks (ruff clean).
• Update docs — not needed; the new kwarg is opt-in and defaults to current behavior.
• Add unit tests (tests/workers/reward_manager/test_naive_on_cpu.py).

[gemini-code-assist]

Code Review

This pull request introduces a per-sample timeout mechanism for the compute_score function in NaiveRewardManager to prevent slow or hanging computations (such as catastrophic regex backtracking or slow symbolic evaluations) from blocking the main training loop. It implements a _score_timeout context manager using SIGALRM and updates NaiveRewardManager to accept a compute_score_timeout parameter, falling back to a reward of 0.0 if a timeout occurs. Comprehensive unit tests are also added to verify this functionality. There are no review comments to address, so no additional feedback is provided.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[CLAassistant]

All committers have signed the CLA.

[zqinglin]

Friendly ping on this one. The three failing checks (npu_unit_tests, the Ascend E2E job, and e2e_fully_async_policy_fsdp2) are unrelated to this change — they also fail on main and on other recently merged PRs, and they exercise the NPU/Ascend and fully-async-policy paths, none of which this PR touches. The reward-manager unit tests and the rest of CI are green.

The change is opt-in: compute_score_timeout defaults to None, so existing behavior is unchanged. Happy to switch the SIGALRM approach to a process pool like PrimeRewardManager uses if maintainers prefer that.

Would appreciate a review when someone has a moment. Thanks!