Skip to content

[Feature] Capture Decision 3.0 forwards as runtime graphs - #4999

Merged
mergify[bot] merged 16 commits into
vllm-project:mainfrom
subin9:perf/d3-noncausal-graphs
Oct 17, 2026
Merged

mergify[bot] merged 16 commits into
vllm-project:mainfrom
subin9:perf/d3-noncausal-graphs

Conversation

@subin9

@subin9 subin9 commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

The native engine skipped host masks and graphs for noncausal backbones, so every Decision 3.0 forward ran eagerly and a short text request was bound by launch time. NoncausalMasks builds the masks as noncausal_masks does, from host-known padding with no read-back, so the graph runner now captures d3 text forwards per exact shape like the causal decision models. A capture that follows an eager use of its shape skips the warm-up forwards, since the eager use already compiled and tuned its kernels, and only the capture stream's first capture warms up. That part applies to the graphs of every decoder model, not only d3. It relies on the one-stream change of #4998, which is now on main, and the record docs/records/decision3-graphs.md holds the numbers.

Related #4995

Test Plan

test_noncausal_graphs_equal_eager runs a random noncausal Qwen3.5 on left-padded rows of many shapes through capture and replay and compares every output with the eager backbone. On an MI300X in the router's ROCm image I served d3-flash with vllm-srun from main and from this branch and sent the same System One requests over HTTP: four fixed requests 40 times each, and 120 two-question requests of new lengths three times in the same order.

Test Result

Every answer was identical to main. Warm medians went from 38.7 to 13.7 ms for one question, 39.6 to 18.0 ms for two and 42.2 to 39.8 ms for eight. The new lengths averaged 46.3, 40.9 and 41.2 ms on main and 46.8, 51.7 and 26.9 ms here: the second pass pays the captures, which cost about 65 ms against 41 ms for an eager forward, and the third replays them. Because the warm-up change reaches every decoder model, I also ran Decision-2.0-Kai-0.6B and Decision-2.0-Eos-0.8B on 150 MMLU-Pro prompts three times on the exact profile: every answer of every pass was identical to the stream fix alone, and the capture pass took 0.92 instead of 1.39 s on Kai and 1.10 instead of 1.58 s on Eos. With traffic made only of new lengths a graph can cost more than it saves until shapes repeat. Image and video requests still run eagerly. They are GPU-bound, an image with two questions spent 129 ms of its 144 ms in kernels before #5006, whose caches now let a repeated image skip preprocessing and the tower: after main was merged in, the same repeated request took 74.6 ms, and one, two and eight text questions 13.6, 18.1 and 39.5 ms. d3-flash does not fit on a 16 GB card, so on an RTX 4070 Ti SUPER I ran d3-nano and d3-lite with this stack, #5000, #5001 and #5009 applied, on the exact profile, where only this stack and #5009 change the path. Answers were identical to main for the fixed requests and for 240 new-length requests of two prompt sets at concurrency 1, 4 and 16. One question went from 29.2 to 10.9 ms on d3-nano and from 29.4 to 7.3 ms on d3-lite, two from 30.7 to 17.0 and from 29.6 to 9.5 ms, eight from 49.1 to 45.5 and from 30.5 to 23.6 ms. The three passes of new lengths took 33.2, 33.2 and 33.5 ms on main and 44.0, 42.5 and 27.5 ms here on d3-nano, and 30.2, 30.6 and 31.2 against 41.0, 30.5 and 15.3 ms on d3-lite. Concurrency 16 went from 30.2 to 39.0 rps on d3-nano and from 33.3 to 72.9 rps on d3-lite on the first prompt set (30.6 to 40.2 and 26.0 to 74.9 on the second), while one request at a time dropped on d3-nano's first set, from 29.9 to 27.5 rps, as it paid the captures. Against the stream fix alone, the capture pass of Kai and Eos took 1.41 instead of 1.99 s and 1.53 instead of 2.04 s, with identical answers. The repository's fast-path GPU tests run only on gfx942, so I ran a copy of test_noncausal_graphs_equal_eager on CUDA with the device switched and without the fused kernels: it passes here and fails on main.

Each graph capture opened a new stream, and BLAS keeps a workspace per
stream, so up to 32 pooled streams each held about 75 MiB on MI300X.

On d3-flash, 40 decoder graphs held 2,669 MiB before and 389 MiB after.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
The native engine skipped masks and graphs for noncausal backbones, so
every d3 forward ran eagerly. NoncausalMasks builds their masks in a graph.

On d3-flash a one question text request takes 13.7 ms, down from 38.7 ms,
with identical answers. A capture after an eager use skips its warm-up.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
The record credited the single capture stream alone. Without warm-up
forwards a new stream per capture held no extra memory on d3-flash, 389 MiB
for 40 shapes either way.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
# Conflicts:
#	src/model-runtime/docs/design.md

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
# Conflicts:
#	src/model-runtime/docs/design.md
#	src/model-runtime/vllm_srun/engines/native/fast.py

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
@subin9 subin9 added the wg/router-models-inference-runtime Owned by the Router Models and Inference Runtime Workgroup. label Oct 13, 2026
@github-actions github-actions Bot added pr/blocked Blocked on a named decision, dependency, or required check. and removed wg/router-models-inference-runtime Owned by the Router Models and Inference Runtime Workgroup. labels Oct 13, 2026
@github-actions github-actions Bot added the wg/router-models-inference-runtime Owned by the Router Models and Inference Runtime Workgroup. label Oct 13, 2026
torch.cuda.graph captured on its own default stream, not the cache's
stream, so a second stream and BLAS workspace stayed alive. The test now
records the capture stream and fails without the fix in any test order.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
@github-actions github-actions Bot added pr/needs-rebase Needs rebase or conflict resolution. and removed pr/blocked Blocked on a named decision, dependency, or required check. labels Oct 13, 2026
WUKUNTAI-0211
WUKUNTAI-0211 previously approved these changes Oct 13, 2026

@WUKUNTAI-0211 WUKUNTAI-0211 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks β€” re-ran on GB10 (CUDA, d3-lite): 480 requests answered identically against #4998 alone, one question went from 22.2 to 15.3 ms, and the CUDA forest-graph tests still pass with the shorter warm-up.

wilsonwu
wilsonwu previously approved these changes Oct 13, 2026

@wilsonwu wilsonwu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# Conflicts:
#	src/model-runtime/vllm_srun/engines/native/fast.py

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
@github-actions github-actions Bot added pr/blocked Blocked on a named decision, dependency, or required check. pr/needs-rebase Needs rebase or conflict resolution. and removed pr/needs-rebase Needs rebase or conflict resolution. pr/blocked Blocked on a named decision, dependency, or required check. labels Oct 14, 2026
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
@subin9
subin9 dismissed stale reviews from theohsiung and WUKUNTAI-0211 via 1d5d8a0 October 16, 2026 01:31
@github-actions github-actions Bot added pr/needs-review Ready for reviewer attention. pr/needs-rebase Needs rebase or conflict resolution. and removed pr/needs-rebase Needs rebase or conflict resolution. pr/needs-review Ready for reviewer attention. labels Oct 16, 2026

@theohsiung theohsiung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: the masks reuse noncausal_masks with no read-back, and the warm-up skip only applies to shapes that already ran eagerly. A few non-blocking points:

1. The warm-up skip reaches every graph, not only d3. It lives in the shared Graphs._capture, so it also applies to causal Decision 2.0 graphs and to Vela 2.0 forest graphs on CUDA. Per src/model-runtime/AGENTS.md, an answer-affecting default needs an accuracy record. Could decision3-graphs.md carry the evidence that is now only in the PR body?

  • the Kai / Eos MMLU-Pro passes (identical answers with the warm-ups dropped)
  • an answer check for the 120 new-length d3 requests; the record only shows their latency, and that pass is the one that captures without warm-ups

2. Vela 2.0 forest graphs are fine with the skip, verified on CUDA. Nothing to change in the code. RTX 4090, torch 2.10+cu128, Triton 3.6, FLA 0.5.2, causal-conv1d 1.7.0, PR head aea908aab:

  • tests/test_gpu_forest_graphs.py passes; 11 of its 16 captures skip the warm-up.
  • Released Vela-2.0-0.8B, 120 generated requests (vela2_parity.generate, scale 0.25), graphs-on and graphs-off engines in one process, each request three times (eager / capture / replay): 0 mismatches in 104 requests (byte-identical responses), 188 of 189 forest captures skipping the warm-up. The other 16 hit OOM on a shared card and were not compared.

The skip is safe for Vela. A line in cuda-vela-forest-graphs.md (or in this record) saying so would cover the accuracy-record rule.

3. Nit: the docstring of test_gpu_fast_path.py still says "right-padded batches"; the new test uses left-padded rows.

@wilsonwu wilsonwu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions Bot added pr/blocked Blocked on a named decision, dependency, or required check. and removed pr/needs-rebase Needs rebase or conflict resolution. labels Oct 16, 2026
@mergify

mergify Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Merge Queue Status

  • βœ… Entered queue β€” 2026-10-16 19:06 UTC Β· Rule: default Β· triggered by rule queue approved pull requests
  • βœ… Checks passed Β· in-place
  • βœ… Merged β€” 2026-10-17 00:59 UTC Β· at 9fdcc25e58956810f4f31ad0ed89a856b8d5a7d4 Β· squash

This pull request spent 5 hours 53 minutes 36 seconds in the queue, including 36 minutes 56 seconds running CI.

Required conditions to merge

@github-actions github-actions Bot added pr/needs-rebase Needs rebase or conflict resolution. and removed pr/blocked Blocked on a named decision, dependency, or required check. labels Oct 16, 2026
@mergify
mergify Bot merged commit 9fdcc25 into vllm-project:main Oct 17, 2026
30 of 31 checks passed
@github-actions github-actions Bot removed the pr/needs-rebase Needs rebase or conflict resolution. label Oct 17, 2026
subin9 added a commit to subin9/semantic-router that referenced this pull request Oct 30, 2026
The record now names vllm-project#4999 as the graphs it was measured with and says
the runs came before vllm-project#5316 and vllm-project#5449 changed the serving path.

Related vllm-project#4997
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

wg/router-models-inference-runtime Owned by the Router Models and Inference Runtime Workgroup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants