357 seconds.
That's how long the median request waited for its first token when we ran GLM-5.3 on vLLM under sustained load. Eight B200s. 32 concurrent workflows.
Our engine, same model, same box: 3.24 seconds.
I didn't build this. Rob and the team did. They've been cooking on it for a long time, mostly quietly. This week we started telling people. Four reports on the blog, three models, three kinds of GPU.
Quick context. This is not a vLLM fork. It's not a quantized model. We wrote our own inference engine. It runs the full model and gives you the full answer.
The scorecard
| Model | Hardware | Up against | Result |
|---|---|---|---|
| GLM-5.3 (BF16) | 8× B200 | vLLM 0.32 | 4.94× sustained output throughput |
| Qwen3.6-35B-A3B (BF16) | 1× RTX PRO 6000 Blackwell | vLLM 0.33 | Up to 1.95× throughput, up to 5.75× faster median TTFT |
| Qwen3.8-27B (BF16) | 1× RTX PRO 6000 Blackwell | vLLM 0.33 and SGLang | 104/104 metrics vs vLLM, 102/104 vs SGLang |
| Qwen3.8-Flash-Next (BF16) | 8× H100 | SGLang (FP8) | Up to 12.8× p90 interactivity on AgentX |
GLM-5.3 on 8× B200
30-minute window. Target of 32 workflows in flight. Both engines got the same first 116 workflows.
- Output throughput: 370.0 vs 74.9 tok/s (4.94×)
- Workflows admitted: 462 vs 116
- Median TTFT: 3.24 s vs 357.5 s
- Per-request generation: 15.8 vs 7.7 tok/s (2.05×) We cleared work faster, so we kept taking more. vLLM never got past the first batch.
Burst test, same 96 workflows dropped at once: 2.61× throughput, 10× faster median TTFT. We finished 95 of 96. vLLM finished 82.
Qwen3.6-35B on one GPU
One RTX PRO 6000 Blackwell on Google Cloud. Both engines at BF16. Both with 3 MTP draft tokens. Same 32K context, greedy decoding.
We tried a bunch of vLLM settings and landed on 95% GPU memory utilization. If you're going to beat a baseline, beat a tuned one.
| Concurrency | 8 | 16 | 32 | 64 | 128 | 256 |
|---|---|---|---|---|---|---|
| Throughput vs vLLM | 1.63× | 1.57× | 1.62× | 1.65× | 1.87× | 1.95× |
Over 1.5× at every level. From 8 concurrent requests to 256.
Median TTFT hit 5.75× faster under steady traffic and 8.56× under burst. Prompts were a mix of chat, long-doc QA, math, code, and tool use (Dolly, LongBench, MATH, HumanEval, Glaive and a few others).
Qwen3.8-27B vs vLLM and SGLang
Smaller model, less headroom. This is where engines start to look alike. So we brought SGLang in too.
8 workload points × 13 metrics = 104 comparisons per engine.
- vs vLLM 0.33: 104 of 104
- vs SGLang: 102 of 104 Throughput was up to 44% over vLLM and up to 26% over SGLang. Every mean, p50, p90 and p99 end-to-end latency went our way against both.
At 1K input and 64 concurrent requests, median TTFT was 299 ms for us. vLLM took 13.3 seconds.
Somebody will ask, so: we use a proprietary 4-bit draft head here. The BF16 model checks every token it drafts. You get the full model's output.
The agent test
This is the one I'm most interested in.
SemiAnalysis AgentX simulates coding agents. Long multi-turn sessions, shared context, subagents, pauses between turns. Average input in our run was 87K to 106K tokens per request. That's what our customers' traffic looks like. Chat benchmarks don't come close.
Qwen3.8-Flash-Next on 8× H100:
- p90 interactivity, 1 session: 209.0 vs 16.3 tok/s per user (12.8×)
- 12 sessions: 7.0×. 16 sessions: 5.0×
- p90 TTFT, 1 session: 0.78 s vs 10.6 s And we did it at full precision. We ran BF16. The SGLang reference ran FP8. They took the shortcut and we still won.
The SGLang bug
Side story, because it's too good to leave out.
One of our engineers spent a long stretch trying to figure out why SGLang's decode speed beat ours. He tore our engine apart looking for it.
It wasn't us. SGLang was applying simulated acceptance in speculative decoding the wrong way. In his testing that pushed their tokens per second up by around 8%. Their outputs were fine. Their speed numbers weren't.
He was mad enough to start drafting a post about it. Then he found the issue already had a merged fix ([PR link]). And yesterday AgentX republished its SGLang numbers. They came down about 3%.
Our AgentX comparison above used the SGLang reference from before that update. The correction moves their numbers down. Not ours.
SGLang is shipping hard right now, including on DeepSeek V4.1 on B200. Good. We want the competition sharp.
Why we built for throughput
We sell into people who own GPUs. They can't get more of them. What they can get is more work out of the ones they have.
So throughput came first. The latency gains showed up along with it.
And no shortcuts on the model. Full weights. No quantized serving model. No relaxed token acceptance.
Read the full reports
Methodology, configs and every number are in the posts.
- GLM-5.3: 4.94× sustained throughput vs vLLM
- Qwen3.6: up to 95% higher throughput than vLLM
- Qwen3.8-27B: outperforming SGLang and vLLM
- Qwen3.8-Flash-Next: up to 12.8× agent interactivity vs SGLang More models and more hardware are coming.
Think something's off in the methodology? The comments are open. That's what they're for.
Top comments (3)
The TTFT jump (357s → 3.24s) is the eye-catcher, but the number I keep coming back to is admission: vLLM "never got past the first batch" (116 workflows) while you admitted 462. That reads less like raw kernel speed and more like queue-admission behavior once KV cache saturates — vLLM's continuous batching stalls when it can't free blocks fast enough. One thing worth separating in the writeup: the per-request generation gap (15.8 vs 7.7 tok/s) is a lot smaller than the aggregate throughput gap, which suggests most of the win is concurrency/admission rather than single-stream decode speed. Readers can then map it to their own workload — a chat app that's latency-bound cares about different numbers than an agent fleet that's throughput-bound. Were both engines given the same max-num-seqs / KV budget on the GLM-5.3 run, or did your memory layout let you hold more concurrent sequences? And genuinely appreciate that you tuned the baseline to 95% util instead of shipping a strawman comparison.
Real question: how much of the 4.94x on GLM-5.3 comes from the scheduler vs the engine itself? "We cleared work faster, so we kept taking more" suggests vLLM's admission is the bottleneck there, not decode throughput — which makes the burst test (2.61x, same offer) the fairer number. Also on the Qwen3.8-Flash-Next run: BF16 vs FP8 changes KV-cache bandwidth as much as compute, so p90 interactivity isn't apples-to-apples in the other direction either. Would be interested in a run where both engines are forced to the same dtype and the same draft config — the 102/104 vs SGLang result at least is close enough that I'd expect it to tighten a lot.
Your 4.94x decomposes cleanly, and the split is the interesting half.
370.0 tok/s over 15.8 per request is 23.4 requests decoding at a given moment on your engine. On vLLM, 74.9 over 7.7 is 9.7. So of the 4.94x, 2.05x is per-request decode speed and 2.41x (23.4/9.7) is how many requests were actually decoding instead of waiting. 2.05 x 2.41 = 4.94, so that accounts for the whole headline.
Both runs had 32 workflows in flight. The only way one engine holds 9.7 in decode while the other holds 23.4 is that the rest of its slots are sitting in prefill. A 357 s median TTFT is that queue, and it is a scheduling result before it is a kernel result.
Which is the figure I want and you do not publish: per engine, the share of the 30-minute window spent in prefill, or mean prefill tokens batched between decode steps. If vLLM's share is 70% or more, then most of the 4.94x is prefill/decode slot allocation under a 32-workflow load, and someone tuning max-num-seqs and prefill chunking against that same workload would close part of it without touching a kernel. What were those two set to on the vLLM side, and what did you run on yours?