Repository navigation
[Sandbox] KAI Scheduler #372
Description
Application contact emails
ekarabulut@nvidia.com, rdar@nvidia.com, romanb@nvidia.com, omdayan@nvidia.com, hsharon@nvidia.com
Project summary
KAI Scheduler is a robust, efficient, and scalable Kubernetes scheduler that optimizes GPU resource allocation for AI workloads in large-scale clusters.
Project description
KAI Scheduler is a Kubernetes-native scheduler designed to optimize resource allocation for AI/ML workloads running on large-scale GPU clusters. It introduces support for gang scheduling, workload prioritization, resource-aware placement, and dynamic resource management tailored for environments running thousands or tens of thousands of GPU nodes with high workload throughput.
KAI Scheduler supports advanced scheduling policies such as bin-packing to reduce fragmentation, spread scheduling for load balancing and resiliency, and queue-based prioritization and quota management. It enables hierarchical queue structures, allowing organizations to assign quotas, priorities, and fairness constraints per queue (e.g., per team, project, or department).
Through integration with Kubernetes ResourceClaims (Dynamic Resource Allocation), it enables GPU sharing, helping to increase overall cluster utilization. Its ability to preempt and consolidate workloads ensures efficient use of scarce accelerator resources by dynamically reallocating underutilized resources.
Some key features:
- Batch Scheduling: Ensure “all-or-nothing” scheduling for pod groups, preventing suboptimal scheduling of distributed workloads.
- Bin Packing & Spread Scheduling: Optimize node usage either by minimizing fragmentation (bin-packing) or increasing resiliency and load balancing (spread scheduling).
- Workload Priority: Prioritize workloads effectively within queues.
- Hierarchical Queues: Manage workloads with two-level queue hierarchies for flexible organizational control.
- Resource distribution: Customize quotas, over-quota weights, limits, and priorities per queue.
- Fairness Policies: Ensure equitable resource distribution using Dominant Resource Fairness (DRF) and resource reclamation across queues.
- Workload Consolidation: Reallocate running workloads intelligently to reduce fragmentation and increase cluster utilization.
- Elastic Workloads: Dynamically scale workloads within defined minimum and maximum pod counts.
- Dynamic Resource Allocation (DRA): Support vendor-specific hardware resources through Kubernetes ResourceClaims.
- GPU Sharing: Allow multiple workloads to efficiently share single or multiple GPUs, maximizing resource utilization.
- Cloud & On-premise Support: Fully compatible with dynamic cloud infrastructures (including auto-scalers like Karpenter) as well as static on-premise deployments.
Org repo URL (provide if all repos under the org are in scope of the application)
N/A
Project repo URL in scope of application
https://github.com/NVIDIA/KAI-Scheduler
Additional repos in scope of the application
No response
Website URL
https://github.com/NVIDIA/KAI-Scheduler
Roadmap
https://github.com/NVIDIA/KAI-Scheduler?tab=readme-ov-file#roadmap
Roadmap context
Our roadmap translates the feedback and ideas we have received from users and the broader community into a focused set of priorities. Our overarching goal is to support the full spectrum of AI workload scheduling use cases - from basic batch workloads to complex, multi-cluster AI pipelines. Key areas of focus include:
- Scheduling Gates for conditional admission, enabling workloads to wait on specific prerequisites before consuming resources.
- Integration with Kueue to support multi-cluster job dispatching within KAI’s scheduling framework.
- Topology-aware PodGroup placement, ensuring that tightly coupled workloads are scheduled on nodes optimized for locality and performance.
- Refactoring the codebase to enhance vendor neutrality.
- AI-specific enhancements, such as support for minimum and maximum workload runtimes, additional default PriorityClasses, and more.
We are committed to building these features and more in close collaboration with the community, defining requirements together, co-authoring design proposals, and encouraging contributions via open discussion and pull requests. Our aim is to ensure each roadmap item reflects real-world needs and evolves under a vendor-neutral, community-first development model.
Contributing guide
https://github.com/NVIDIA/KAI-Scheduler/blob/main/CONTRIBUTING.md
Code of Conduct (CoC)
https://github.com/NVIDIA/KAI-Scheduler/blob/main/code_of_conduct.md
Adopters
No response
Contributing or sponsoring org
Maintainers file
https://github.com/NVIDIA/KAI-Scheduler/blob/main/MAINTAINERS.md
IP policy
- If the project is accepted, I agree the project will follow the CNCF IP Policy
Will the project require a license exception?
N/A
Trademark and accounts
- If the project is accepted, I agree to donate all project trademarks and accounts to the CNCF
Standard or specification?
N/A
Why CNCF?
Donating the KAI Scheduler to the CNCF would provide a solid foundation for neutral, community-driven evolution of the project, while allowing it to more fully align with the broader cloud-native ecosystem.
- Neutral Governance and Trust: CNCF’s vendor-neutral governance model ensures that the project’s direction is shaped by community needs rather than a single stakeholder. This fosters broader trust, especially among organizations deploying KAI Scheduler in production, and helps avoid concerns about vendor lock-in.
- Ecosystem Alignment and Adoption: KAI Scheduler is designed to integrate cleanly with existing Kubernetes-native components and has initial support for projects like Kubeflow’s training operator. As part of the CNCF, we can accelerate its adoption by aligning more deeply with projects such as Kueue, and Karmada, enabling seamless orchestration of complex, distributed ML workloads.
- Contributor and Community Growth: Joining CNCF offers exposure to a global base of contributors and users, which is essential for sustaining the long-term health of the project. A broader contributor base can help drive innovation, identify edge cases, and ensure responsiveness to the needs of varied environments.
- Collaboration Across Projects: Many CNCF-hosted projects tackle complementary concerns, such as multi-cluster management, topology-aware scheduling, and observability. Being part of the foundation will enable direct collaboration with these efforts, ensuring KAI evolves in harmony with the rest of the ecosystem, while contributing to other project’s work.
Benefit to the landscape
Adding KAI Scheduler to the CNCF sandbox enriches the ecosystem of Kubernetes schedulers by filling several gaps:
- The Podgrouper: Eases the integration with any type of workload - especially important for gang scheduling. It automatically groups related pods - e.g. PytorchJob, MPIJob etc. - ensuring that they are scheduled all together.
- Pre-scheduling simulations: Before taking any actions, the scheduler runs simulations to figure out the most optimal actions to take. This makes sure that workload eviction is only done if other more prioritized workload can be scheduled, thus preventing unnecessary evictions.
- Plugin-based scheduler logic: Allows users to extend the scheduler logic by implementing plugins
- Focus on the entire AI lifecycle: Accommodates all types of workloads in the AI lifecycle in a cluster; inference, training, and interactive workloads.
- Asynchronous Binder: Capable of binding the pods with resources asynchronously. It helps scaling with the scheduler to 10k+ GPU clusters.
- Cloud auto-scale support: Scheduling works seamlessly with different auto-scalers, such as HPA, node auto scaler, and Karpenter - supporting both regular workloads and workloads that leverages gpu sharing.
Cloud native 'fit'
KAI aligns with core CNCF principles:
- Declarative API: all scheduler primitives, like scheduling queues, podgroups and bindRequest are implemented as CRDs. Allowing k8s native integration with other frameworks running on k8s. Status is published natively on these CRDs.
- Pluggability & modularity: scheduler logic and grouping strategies are implemented as plugins and can be extended by users.
- Asynchronous controllers: Binder and PodGrouper use Kubernetes controller patterns for resilience and scalability. KAI Scheduler is built using a micro service paradigm, every service is responsible for a single task.
- Cloud-native integrations: leverages existing CNCF projects (Kubeflow Training Operator, HPA, Cluster autoscaler, Karpenter, CSI, Device Plugins, Dynamic Resource Allocation) without proprietary extensions.
- Support for Kubernetes-native workloads: KAI Scheduler is designed to work with standard Kubernetes objects - Pods, Deployments, StatefulSets, etc. This ensures that users can adopt KAI Scheduler without altering their existing workflows or application definitions.
Cloud native 'integration'
Dependency:
- Kubernetes: KAI Scheduler is implemented as a set of Kubernetes controllers and CRDs. It provides features like gang-scheduling, hierarchical-queues, and plugin-based extensions, while preserving the native declarative and dynamically extensible model of Kubernetes.
Complementary Projects:
- Knative: KAI can schedule GPU-powered knative services, providing bursty inference workloads with rapid scale-to-zero and scale-to-one strategies.
- Kubeflow Training Operator: KAI’s PodGrouper automatically groups TFJob, MPIJobs and PyTorchJob pods, ensuring consistent gang scheduling for distributed training.
- Cluster Autoscaler & Karpenter: By watching autoscaler events and injecting scale-in/scale-out hooks, KAI Scheduler ensures that GPU-driven workloads trigger right-sized node provisioning.
- Prometheus & Grafana: KAI can export metrics for GPU metrics such as scheduler performance, resource division results between queues, number of evictions, number of workloads scheduled in a single scheduling cycle, scheduling cycle latency and scheduler action latency. These make it easy for the users to monitor the cluster state.
Cloud native overlap
- Volcano: Volcano provides batch scheduling, gang scheduling, hierarchical queues, workload preemption, Dominant Resource Fairness (DRF), elastic workloads, bin-packing, and GPU sharing.
- Kueue: Kueue focuses on job dispatching, batch scheduling, gang scheduling, multiple queues, hierarchical queues, cross-queue fairness, guaranteed quotas, and over-quota weights.
While all these projects implement core gang, and fair-share scheduling capabilities, KAI Scheduler was architected from day one for AI at scale and across the entire AI lifecycle. The PodGrouper automatically groups workloads for gang scheduling without manual intervention. Before taking any action on running pods, KAI Scheduler runs a pre-scheduling simulation. This makes sure that higher-priority workloads can be scheduled for sure after an eviction of a lower priority workload, drastically reducing needless evictions. Finally, with asynchronous Binder, KAI maintains high scheduler throughput at 10k+ GPU scale and integrates seamlessly with cloud autoscalers.
Similar projects
Volcano, Kueue
Landscape
No
Business Product or Service to Project separation
KAI Scheduler will remain the upstream open-source engine powering NVIDIA Run:ai platform. All development related to the scheduler occurs in the community repository under Apache 2.0. NVIDIA Run:ai will continue to use the same code as the community, ensuring no divergence and maximum benefit to all adopters.
Project "Domain Technical Review"
No response
CNCF contacts
Kevin Klues, Ricardo Rocha
Additional information
No response
Activity
dims commentedon May 8, 2025
@EkinKarabulut thank you for the submission. a really quick question before i dive into it .. have you had conversations with the k8s sig-scheduling / wg-batch folks already? just to compare / contrast / dig-into-details of how this effort is similar or different from the other things going on or being planned. In the end you need allies to thrive in CNCF, so they seem to be good ones to have on your side, hence the question :)
thanks!
EkinKarabulut commentedon May 8, 2025
@dims Very good question! :) Since we open-sourced the project at KubeCon, we have started engaging with the batch-wg group (including this recording, where we presented and took questions). We have already received insightful feedback from some wg-batch members as well, which we have started incorporating (see here and here).
We definitely see batch-wg, SIG Scheduling and wg-batch as key groups to collaborate with, and that is actually one of our next steps moving forward. We want to align with ongoing efforts and identify synergies
raravena80 commentedon May 8, 2025
Would love to have this project present in the CNCF AI Initiative meeting. https://docs.google.com/document/d/1tS6gL5qjGLaHDtAChssqJhBVm8BIpopUA3Ddzm9LHwg/edit?tab=t.0#heading=h.tprh9feizr4r. (Meeting notes to be renamed)
kannon92 commentedon May 12, 2025
Adding KAI to the agenda for a wg-batch call would be a good idea.
Monokaix commentedon Jun 10, 2025
Given the KAI-Scheduler project's submission for the CNCF Sandbox, the Volcano project team feels it is necessary to raise our community's important concerns.
As long time cloud native open source participants and one of the few early CNCF AI related projects, the Volcano project team has been facilitating and encouraging open source projects to join the CNAI ecosystem and glad to see its tremendous growth recently, however we should be mindful of the importance of open source compliance to have a healthy environment. Since KAI-Scheduler was open-sourced, the Volcano community has consistently received feedback from users and contributors expressing concerns that the project may contain code that was plagiarized or otherwise infringes on the Volcano project. A preliminary analysis conducted by Volcano's maintainers has revealed a number of noteworthy similarities, and we have reason to believe that acts of code infringement may be involved.
Of particular concern, the copyright notices in these similar code segments have been changed to point to NVIDIA. We believe this is in clear conflict with the terms of the Apache-2.0 license that governs the original code.
We have noted that the KAI-Scheduler project mentions Kube-batch in its NOTICE file. However, this action does not fully resolve the potential licensing issues. As the community is aware, the Kube-batch project has been archived for many years. The Volcano project was founded by the same group of maintainers to serve as its successor, with the goals of establishing a more open and mature governance model and seeking broader support from the CNCF. The Volcano community has been actively carrying on and developing the work of Kube-batch, and many projects that previously integrated with Kube-batch have since upgraded to a Volcano integration.
Furthermore, based on user reports and our own analysis, the Volcano community has confirmed the locations of some of the code in question. To ensure the healthy development of the open-source ecosystem and out of respect for existing CNCF projects, the Volcano community strongly suggests that the KAI-Scheduler project conduct a thorough audit and rectification of its codebase to ensure full compliance with its open-source license requirements and to respect the intellectual property of other CNCF projects.
Summary of Key Similarities
Our analysis shows that the similarities between the two projects are not merely on a conceptual level, but run deep into the core architecture, implementation details, and internal API design. This level of similarity cannot be explained away as simply following broad "industry best practices."
The following is a summary of a few key areas of similarity:
Core Scheduling Framework: KAI-Scheduler's overall scheduling framework exhibits a high degree of consistency with Volcano's in its design philosophy, key components, and method of operation.
Session Mechanism: It fully adopts Volcano's design of using a Session as the core unit of a scheduling cycle, relying on a Snapshot to ensure decision consistency.
Plugin Architecture: It reuses Volcano's plugin-based philosophy, including the use of Tiers to organize plugins and the pattern of extending functionality by registering specific function types (e.g., NodeOrderFn) on the Session.
Action-driven Workflow: The scheduling flow is driven by a series of Actions (e.g., allocate, preempt, reclaim) that are identical in name and function to those in Volcano.
Internal API & Core Data Structures: The project shows a high degree of consistency with Volcano in its core abstractions and state definitions, yet its internal implementation contains numerous concepts that are not part of its own declared design.
KAI-Scheduler's core data structures—PodGroupInfo, PodInfo, and its state definition PodStatus—are nearly identical to Volcano's JobInfo, TaskInfo, and TaskStatus, respectively, in their design purpose, key fields, and lifecycle state definitions.
However, a particularly noteworthy phenomenon is that throughout the KAI-Scheduler codebase, especially in numerous code comments and some internal function names, the terms "Job" and "Task" appear frequently.
These two terms are foundational to Volcano's scheduling model but are not concepts included in KAI-Scheduler's official design.
A detailed, code-level comparison is available for review in this document: Code-Level Comparison between Volcano and KAI-Scheduler
Our Proposed Solution
In the spirit of openness and collaboration, we sincerely suggest that the KAI-Scheduler team take the following steps:
A Vision for Future Collaboration
The Volcano community and its maintainers are committed to the principles of openness and neutrality. We would warmly welcome the KAI-Scheduler team to contribute their innovations and improvements on the scheduler framework back to the Volcano project. We believe that through cooperation, we can jointly advance the technological progress of cloud-native batch computing and AI scheduling and foster a thriving community for all.
ronendar commentedon Jun 10, 2025
We're looking into this.
We are looking forward to working with the community and improving the experience of Kubernetes/cloud-native batch computing and scheduling users everywhere.
Thank you for bringing this to our attention.
zvonkok commentedon Jun 16, 2025
/cc @zvonkok
EkinKarabulut commentedon Jul 1, 2025
Hi Volcano team @Monokaix,
First of all, thank you for taking the time to review KAI-Scheduler and for writing up such a detailed comparison. We have huge respect for the Volcano community and agree 100% that healthy collaboration starts with license hygiene and transparent provenance.
What we have done so far
After reading your issue and the attached document we double-checked every file in our repo and reviewed our internal guidance to get more information about the required license headers.
The outcome:
All of the similarities you highlighted except for one file (discussed in the next bullet point) trace back to kube-batch (here is the commit that KAI scheduler used) - the project from which both Volcano and KAI forked in 2019. We compiled a document to address all the points that you pointed out (link here)
One file (our FrameworkHandle adapter, ≈120 LOC) was copied from Volcano after our original fork, which is an interface definition to the kube-scheduler framework.
During our investigation, we identified the specific miscommunication which led to the header change. We have additionally updated our processes to be sure there are more opportunities for this sort of oversight to be caught in our software development lifecycle.
Pull request #291 is now about to be merged. It restores every kube-batch Apache-2.0 header that was errantly removed; adds “© The Volcano Authors” to the FrameworkHandle adapter; and updates NOTICE.md to mention both projects explicitly.
We are happy to consider a full code audit if there are still concerns. However, we believe that our internal investigation has identified and addressed the issues in our process and the errant replacement of headers. The specifics of which are outlined in this doc.
Same roots, different approaches
While there are minor structural similarities (Session/Action/Statement, plugin registry, Job-Task wrappers), the projects have been evolving along very different lines since 2019. Here are some of the main differences we see:
KAI: allocate, consolidate, reclaim, preempt, and staleGangEviction
Volcano: allocate, backfill, enqueue, preempt, reclaim, and shuffle
Better scalability: separating these operations allows us to scale these controllers independently, and also takes the load off the scheduler. This helped us scale the scheduler to 10k+ GPU clusters.
This architecture implements an extendable and open approach, allowing us to easily interface with 3rd party controllers
These are just some of the main differences, and there are more nuanced changes that reflect the distinct paths both projects have taken.
Looking forward
Your invitation to collaborate resonates strongly with us! We are very happy to discuss where joint work on current scheduling challenges and the different approaches both projects have taken could benefit the entire CNAI ecosystem. If the Volcano team is interested, we would love to set up a shared call to explore opportunities.
Again, thank you for holding us to a high standard and for the warm welcome into the CNCF family. We’re excited about building great schedulers together!
The KAI-Scheduler maintainers (Roman, Omer, Omri, Erez, David, Daniel, Ekin, Hagay)
25 remaining items
github-actions commentedon Dec 21, 2025
The TOC vote for KAI Scheduler has passed! 🎉 Next step: Please finalize the Contribution Agreement. This is required before official CNCF on-boarding can begin. Please reach out to CNCF Staff for assistance.
idvoretskyi commentedon Dec 29, 2025
The Contribution agreement has not been signed yet.
github-actions commentedon Feb 8, 2026
🎉 Congratulations! The onboarding issue has been created for KAI Scheduler.
The community vote has been completed successfully, and your project is now ready to begin the CNCF onboarding process.
Next Steps:
Good luck with your project's journey in the CNCF! 🚀