Repository navigation
feat(ray-data): add scheduler observability - #2293
Merged
Merged
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Signed-off-by: Praateek <praateekm@gmail.com>
praateekmahajan
force-pushed
the
nmcur-214
branch
from
August 4, 2026 18:21
b081b6a to
07f9e69
Compare
praateekmahajan
changed the title
praateekmahajan
commented
Aug 4, 2026
Comment on lines
+136
to
+139
| _install_resource_admission_diagnostics(resource_manager_module, resource_policy_module) | ||
| _install_downstream_capacity_diagnostics(downstream_policy_module) | ||
| _install_scheduling_reasons(executor_state_module) | ||
| _install_actor_autoscaling_diagnostics(autoscaler_module) |
Signed-off-by: Praateek <praateekm@gmail.com>
Signed-off-by: Praateek <praateekm@gmail.com>
weijiac0619
added a commit
that referenced
this pull request
Aug 5, 2026
1. Deadlock prevention (liveness guarantee): when all ops are blocked and no tasks running, Ray Data bypasses backpressure to prevent deadlock -- tasks can dispatch even when scheduling_reason shows block 2. scheduling_reason clarification: no_actor_slot and operator_cannot_accept_input come from operator slot availability (op.can_add_input()), not from any backpressure policy 3. Document execution_resource_fields pattern: each resource group (allocation, usage, remaining_budget) emits _cpu/_gpu/_heap_memory/ _object_store_memory; note GPU pipelines need _gpu/_object_store variants 4. Add missing remaining_budget_heap_memory to ray_data_resource_budget_admission table 5. initial_size defaults to min_size when not specified 6. Add target_max_block_size to tuning knobs (default 128MB); critical for Curator video/embedding pipelines where large per-item payloads drive object store pressure and the pending_output_exceeds_object_store_budget throttle Signed-off-by: Weijia Chen <weijiac@nvidia.com>
praateekmahajan
marked this pull request as ready for review
August 6, 2026 17:16
praateekmahajan
requested review from
abhinavg4,
ayushdg,
oyilmaz-nvidia,
rlratzel and
sarahyurick
as code owners
August 6, 2026 17:16
Contributor
Greptile SummaryThe PR adds an opt-in runtime shim for Ray 2.56.1 that records Ray Data resource-admission, downstream-backpressure, scheduling, and actor-autoscaling decisions in Ray session logs.
Confidence Score: 5/5The PR appears safe to merge with no concrete blocking or independently actionable non-blocking issues identified. The version and opt-in guards constrain the private-API shim appropriately, and the investigated hooks preserve Ray’s scheduling, backpressure, and actor-scaling decisions while adding diagnostic logging. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart LR
Env["Diagnostics environment variable"] --> Executor["RayDataExecutor.execute"]
Executor --> Installer["Install Ray 2.56.1 runtime hooks"]
Installer --> Admission["Resource admission hook"]
Installer --> Backpressure["Downstream backpressure hook"]
Installer --> Scheduling["Scheduling-reason hook"]
Installer --> Autoscaling["Actor autoscaling hook"]
Admission --> Logs["Ray Data session log"]
Backpressure --> Logs
Scheduling --> Autoscaling
Autoscaling --> Logs
Reviews (1): Last reviewed commit: "Merge branch 'main' into nmcur-214" | Re-trigger Greptile |
praateekmahajan
enabled auto-merge (squash)
August 6, 2026 17:34
weijiac0619
approved these changes
Aug 6, 2026
weijiac0619
left a comment
weijiac0619
left a comment
Contributor
There was a problem hiding this comment.
LGTM! Awesome work on this, thanks!
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in Ray 2.56.1 runtime shim that writes Ray Data scheduling decisions to session logs without requiring a patched wheel.
Tests: 26 Ray Data backend tests passed in Docker.