Repository navigation
[Feature] Capture Decision 3.0 forwards as runtime graphs - #4999
Conversation
Each graph capture opened a new stream, and BLAS keeps a workspace per stream, so up to 32 pooled streams each held about 75 MiB on MI300X. On d3-flash, 40 decoder graphs held 2,669 MiB before and 389 MiB after. Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
The native engine skipped masks and graphs for noncausal backbones, so every d3 forward ran eagerly. NoncausalMasks builds their masks in a graph. On d3-flash a one question text request takes 13.7 ms, down from 38.7 ms, with identical answers. A capture after an eager use skips its warm-up. Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
The record credited the single capture stream alone. Without warm-up forwards a new stream per capture held no extra memory on d3-flash, 389 MiB for 40 shapes either way. Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
# Conflicts: # src/model-runtime/docs/design.md Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
# Conflicts: # src/model-runtime/docs/design.md # src/model-runtime/vllm_srun/engines/native/fast.py Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
torch.cuda.graph captured on its own default stream, not the cache's stream, so a second stream and BLAS workspace stayed alive. The test now records the capture stream and fails without the fix in any test order. Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
WUKUNTAI-0211
left a comment
There was a problem hiding this comment.
Thanks β re-ran on GB10 (CUDA, d3-lite): 480 requests answered identically against #4998 alone, one question went from 22.2 to 15.3 ms, and the CUDA forest-graph tests still pass with the shorter warm-up.
# Conflicts: # src/model-runtime/vllm_srun/engines/native/fast.py Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
theohsiung
left a comment
There was a problem hiding this comment.
Approving: the masks reuse noncausal_masks with no read-back, and the warm-up skip only applies to shapes that already ran eagerly. A few non-blocking points:
1. The warm-up skip reaches every graph, not only d3. It lives in the shared Graphs._capture, so it also applies to causal Decision 2.0 graphs and to Vela 2.0 forest graphs on CUDA. Per src/model-runtime/AGENTS.md, an answer-affecting default needs an accuracy record. Could decision3-graphs.md carry the evidence that is now only in the PR body?
- the Kai / Eos MMLU-Pro passes (identical answers with the warm-ups dropped)
- an answer check for the 120 new-length d3 requests; the record only shows their latency, and that pass is the one that captures without warm-ups
2. Vela 2.0 forest graphs are fine with the skip, verified on CUDA. Nothing to change in the code. RTX 4090, torch 2.10+cu128, Triton 3.6, FLA 0.5.2, causal-conv1d 1.7.0, PR head aea908aab:
tests/test_gpu_forest_graphs.pypasses; 11 of its 16 captures skip the warm-up.- Released Vela-2.0-0.8B, 120 generated requests (
vela2_parity.generate, scale 0.25), graphs-on and graphs-off engines in one process, each request three times (eager / capture / replay): 0 mismatches in 104 requests (byte-identical responses), 188 of 189 forest captures skipping the warm-up. The other 16 hit OOM on a shared card and were not compared.
The skip is safe for Vela. A line in cuda-vela-forest-graphs.md (or in this record) saying so would cover the accuracy-record rule.
3. Nit: the docstring of test_gpu_fast_path.py still says "right-padded batches"; the new test uses left-padded rows.
Merge Queue Status
This pull request spent 5 hours 53 minutes 36 seconds in the queue, including 36 minutes 56 seconds running CI. Required conditions to merge
|
The record now names vllm-project#4999 as the graphs it was measured with and says the runs came before vllm-project#5316 and vllm-project#5449 changed the serving path. Related vllm-project#4997 Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Purpose
The native engine skipped host masks and graphs for noncausal backbones, so every Decision 3.0 forward ran eagerly and a short text request was bound by launch time.
NoncausalMasksbuilds the masks asnoncausal_masksdoes, from host-known padding with no read-back, so the graph runner now captures d3 text forwards per exact shape like the causal decision models. A capture that follows an eager use of its shape skips the warm-up forwards, since the eager use already compiled and tuned its kernels, and only the capture stream's first capture warms up. That part applies to the graphs of every decoder model, not only d3. It relies on the one-stream change of #4998, which is now on main, and the recorddocs/records/decision3-graphs.mdholds the numbers.Related #4995
Test Plan
test_noncausal_graphs_equal_eagerruns a random noncausal Qwen3.5 on left-padded rows of many shapes through capture and replay and compares every output with the eager backbone. On an MI300X in the router's ROCm image I served d3-flash withvllm-srunfrom main and from this branch and sent the same System One requests over HTTP: four fixed requests 40 times each, and 120 two-question requests of new lengths three times in the same order.Test Result
Every answer was identical to main. Warm medians went from 38.7 to 13.7 ms for one question, 39.6 to 18.0 ms for two and 42.2 to 39.8 ms for eight. The new lengths averaged 46.3, 40.9 and 41.2 ms on main and 46.8, 51.7 and 26.9 ms here: the second pass pays the captures, which cost about 65 ms against 41 ms for an eager forward, and the third replays them. Because the warm-up change reaches every decoder model, I also ran Decision-2.0-Kai-0.6B and Decision-2.0-Eos-0.8B on 150 MMLU-Pro prompts three times on the exact profile: every answer of every pass was identical to the stream fix alone, and the capture pass took 0.92 instead of 1.39 s on Kai and 1.10 instead of 1.58 s on Eos. With traffic made only of new lengths a graph can cost more than it saves until shapes repeat. Image and video requests still run eagerly. They are GPU-bound, an image with two questions spent 129 ms of its 144 ms in kernels before #5006, whose caches now let a repeated image skip preprocessing and the tower: after main was merged in, the same repeated request took 74.6 ms, and one, two and eight text questions 13.6, 18.1 and 39.5 ms. d3-flash does not fit on a 16 GB card, so on an RTX 4070 Ti SUPER I ran d3-nano and d3-lite with this stack, #5000, #5001 and #5009 applied, on the exact profile, where only this stack and #5009 change the path. Answers were identical to main for the fixed requests and for 240 new-length requests of two prompt sets at concurrency 1, 4 and 16. One question went from 29.2 to 10.9 ms on d3-nano and from 29.4 to 7.3 ms on d3-lite, two from 30.7 to 17.0 and from 29.6 to 9.5 ms, eight from 49.1 to 45.5 and from 30.5 to 23.6 ms. The three passes of new lengths took 33.2, 33.2 and 33.5 ms on main and 44.0, 42.5 and 27.5 ms here on d3-nano, and 30.2, 30.6 and 31.2 against 41.0, 30.5 and 15.3 ms on d3-lite. Concurrency 16 went from 30.2 to 39.0 rps on d3-nano and from 33.3 to 72.9 rps on d3-lite on the first prompt set (30.6 to 40.2 and 26.0 to 74.9 on the second), while one request at a time dropped on d3-nano's first set, from 29.9 to 27.5 rps, as it paid the captures. Against the stream fix alone, the capture pass of Kai and Eos took 1.41 instead of 1.99 s and 1.53 instead of 2.04 s, with identical answers. The repository's fast-path GPU tests run only on gfx942, so I ran a copy of
test_noncausal_graphs_equal_eageron CUDA with the device switched and without the fused kernels: it passes here and fails on main.