Intelligence beyond any one model.


An open, programmable decision layer for models and compute.
Give your agent harness a stable model API. vLLM Semantic Router selects or combines models through explicit policy, behind an OpenAI- or Anthropic-compatible endpoint, so you can change models and policy without rewriting the harness integration.
- Route each call: a fast local model for one call, a specialist, a longer context or a second opinion for the next.
- Judge with one model: a decision model answers the policy's questions about each request.
- Serve judgments directly: applications can also ask the decision model through the System One API.
Why route?​
Capability, latency, cost and trust vary with the request, the user, the session and the infrastructure at hand. A shared routing policy keeps these choices out of each harness's code.
What you can program​
An entrypoint selects an isolated recipe. Its signals capture intent, difficulty, context, modality, identity, risk and runtime observations. Use them to:
- Select or combine models: choose a local model or specialist, escalate from a small model to a stronger one, or run a bounded multi-model workflow.
- Add route behavior: prompts, retrieval, memory, tool filtering, caching, safety checks and verification.
- Choose an execution path: cloud, data-center or edge backends across heterogeneous hardware.
- Inspect and improve decisions: routing metadata, feedback, replay and evaluation.
Who does what​
| Part | Owns |
|---|---|
| Agent harness | The agent loop, tool execution and task state |
| Semantic Router | Per-call policy and bounded model collaboration |
| Model runtime | The Router's own decision and embedding models |
| Inference platforms | Chat backends, with their placement, batching and capacity |
The Router serves its own listeners by default; an Envoy-based gateway is optional. See the component architecture for the module boundaries.
Start here​
- Run the Quickstart.
- Connect an agent harness.
- Explore use cases.
- Read the System Overview and Routing Pipeline.
- Build virtual models with entrypoints and recipes.
- Compare deployment options.
Project​
vLLM Semantic Router is open source under the Apache 2.0 license. See the contributing guide to propose a change or join the community.