Skip to content

Updated documentation for Grammar Backend - #2545

Merged
zhaochenyang20 merged 9 commits into
sgl-project:mainfrom
shuaills:docs/structured-decoding-updates
Dec 23, 2024
Merged

Updated documentation for Grammar Backend#2545
zhaochenyang20 merged 9 commits into
sgl-project:mainfrom
shuaills:docs/structured-decoding-updates

Conversation

@shuaills

@shuaills shuaills commented Dec 22, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Motivation

Improve documentation clarity around grammar backends.

#2505

Modifications

Updated documentation to clarify Xgrammar's JSON support and regex limitation.

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

@zhaochenyang20 zhaochenyang20 changed the title Related to #2505 Updated documentation to clarify Xgrammar features. Updated documentation for Grammar Backend Dec 22, 2024
zhaochenyang20
zhaochenyang20 requested changes Dec 22, 2024
View reviewed changes
Comment thread docs/backend/openai_api_completions.ipynb
@@ -223,6 +223,13 @@
"## Structured decoding (JSON, Regex)\n",

@zhaochenyang20 zhaochenyang20 Dec 22, 2024

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can define a JSON schema or regular expression to constrain the model's output, which depends on the grammar backend.

SGlang has two backends: outlines (default) and Xgrammar. Xgrammar enhances JSON decoding performance but does not support regular expressions. To use Xgrammar, add --grammar-backend xgrammar when launching the server:

@shuaills shuaills Dec 22, 2024

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and verified the functionality with Xgrammar backend.

@zhaochenyang20

zhaochenyang20 commented Dec 22, 2024

Copy link
Copy Markdown
Collaborator

This is just to update the current document, so a tiny modification is good enough. However, for the constraint decoding tutorial, please truly launch the server to check its correctness. Also, what's the status of the tutorial? @shuaills

@shuaills

shuaills commented Dec 22, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

This is just to update the current document, so a tiny modification is good enough. However, for the constraint decoding tutorial, please truly launch the server to check its correctness. Also, what's the status of the tutorial? @shuaills

I'm working on it. Should it be in Jupyter Notebook format? And where should I put the tutorial?

@zhaochenyang20

zhaochenyang20 commented Dec 22, 2024

Copy link
Copy Markdown
Collaborator

This is just to update the current document, so a tiny modification is good enough. However, for the constraint decoding tutorial, please truly launch the server to check its correctness. Also, what's the status of the tutorial? @shuaills

I'm working on it. Should it be in Jupyter Notebook format? And where should I put the tutorial?

It should be a Notebook. Put it under docs/backend

zhaochenyang20
zhaochenyang20 requested changes Dec 22, 2024
View reviewed changes

@zhaochenyang20 zhaochenyang20 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tiny modification. Then we can merge it.

Comment thread docs/backend/openai_api_completions.ipynb Resolved
@zhaochenyang20
zhaochenyang20 enabled auto-merge (squash) December 22, 2024 23:48
@zhaochenyang20
zhaochenyang20 enabled auto-merge (squash) December 23, 2024 00:15
@zhaochenyang20
zhaochenyang20 merged commit 25e5d58 into sgl-project:main Dec 23, 2024
@shuaills
shuaills deleted the docs/structured-decoding-updates branch December 29, 2024 07:11
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
@shuaills @zhaochenyang20
Doc: Update Grammar Backend (sgl-project#2545)
70a3262 
Co-authored-by: Chayenne <zhaochen20@outlook.com>
kevin-mii added a commit to kevin-mii/sglang that referenced this pull request Oct 14, 2026
@kevin-mii @claude
[Mono] DeepSeek-V4.1 on gfx950: every backbone layer in two monokerne…
f1ca7e1 
…l launches

DeepSeek-V4.1-Flash's monokernels (sgl-project#43497 with K2 and the DSpark capture fix)
moved onto the monokernel plugin (RFC sgl-project#43949), without their env vars, and
extended to all 40 backbone layers (vLLM #61082's layout), with ATOM sgl-project#2545's
hand-off orderings:

- kernels/ops/mono/deepseek_v41_hip/: K1 (the attention seam and front: the
  pending post, the mixes, wqkv_a, the q / kv norms, RoPE, the SWA KV store,
  wq_b) and K2 (the attention core, wo_a, wo_b, the FFN seam, the MoE and both
  all-reduces), on common_hip's guarded waits; the spec lists gfx950 as its
  only ISA. K1 takes the peer addresses too, so a timeout in any launch flags
  every rank. The runner (DSV41MonoRunner) checks every tensor, cache and key
  table it is given; weights.py holds the layout conversions (E8M0 codes,
  exact MXFP8, the RoPE tables).
- Every backbone layer runs whole (LAYER_IN -> LAYER_OUT):
  - K1 folds the previous layer's pending post into the residual. Where none
    is owed (layer 0's broadcast embedding; the Engram layers 1 and 14, whose
    Engram write SGLang's loop runs on the settled residual) it takes the
    residual as it is, collapsed with the previous pre (copy 0 on layer 0).
  - The index sources (2, 8, 14, 20, 24, 28, 32, 36): K1 also writes the normed
    input and the bf16 qr, SGLang's forward_low_ratio_sources runs the layer's
    compressor and indexer on them between the launches, as SGLang runs them
    before the attention, and K2 builds the key table from the top-k they
    wrote (each split task waits on its token's flag).
  - The window-only layers 0 and 1 read no compressed cache.
- deepseek_v41_hip/spec.py: the spec and its SGLang glue. Each span returns
  what the HIP fused mHC boundary returns, with the layer's post pending for the
  next layer. Per-launch TOLERANCE bounds, read by the kernel checks. The load
  hook keeps the FP8 shared expert, which --enforce-shared-experts-fusion
  converts to MXFP4; the other weights are read in place. Any
  --fp8-gemm-backend but aiter is refused at config time (the others reorder
  wqkv_a, wq_b and wo_b in place; K1 and K2 read them row-major). The kernels
  never quantize more than the normal path: K2 keeps the attention output in
  bf16 for wo_a (its FP8 weight widened to bf16 in registers), as SGLang's
  bf16 wo_a reads it. Each rank builds every width, starting at its own share
  (FlyDSL's disk cache hands it the rest).
- From ATOM sgl-project#2545, bit for bit: the FFN seam's loads go out before the wait on
  the ranks' attention partials, a norm polls its rows once, after loading its
  weights, and the seams' Sinkhorn gates (read only by the next launch) run
  after K1's front and after K2's shared expert.
- deepseek_v4.py / deepseek_v4_hip.py / deepseek_v4_fused_mhc.py: the seams
  (the load hook, attach, route once a forward, a span at each start point).
  Under a plan the covered layers take DSpark's captures (the residual each
  target layer's boundary rebuilds), so the layers before DSpark's target
  layers keep their post pending for K1. With --enable-monokernel off nothing
  changes.
- Tests: the spec on CPU (test/registered/unit/mono); K1 (each input form: a
  pending post, a settled residual, layer 0, an index source's outputs), K2
  (an index source's and a window-only layer's bit for bit against the K2 of
  the same keys) and the FFN launch under torchrun at TP2 and TP4, with graph
  replay across widths (test/registered/kernels/ops/mono, helpers in
  sglang.test.kernels.mono); serving at TP4 (test/registered/e2e/mono, also a
  nightly job): DSpark verify and plain decode steps past the largest width and
  back (GSM8K, and the covered steps' tokens scored again by the normal
  prefill path, KL), and a fault armed on one rank mid-request: every rank
  stops, no token is sent, the only faults are the deliberate ones at 0x0. The
  chained stress stays manual.
- test/manual/kernels/mono/validate.py runs the CPU tests, the kernel checks,
  the serving tests and a paired decode A/B at 1, 4, 8 and 16 concurrent
  requests against baselines.json, and prints PASS or FAIL for each step.
  .agents/skills/monokernel gives agents the layout, the seams, the invariants
  and how to repair, tune or add a model.

MI350X TP4 DSpark, simulated acceptance 3.51, per-user decode tok/s at 1 / 4 /
8 / 16 concurrent, both arms in one validate.py run: 453.5 / 328.9 / 283.7 /
219.7 without --enable-monokernel, 579.0 / 399.8 / 303.9 / 221.0 with it
(+28% / +22% / +7% / +1%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kevin-mii added a commit to kevin-mii/sglang that referenced this pull request Oct 15, 2026
@kevin-mii @claude
[Mono] DeepSeek-V4.1 on gfx950: every backbone layer in two monokerne…
52b7a80 
…l launches

DeepSeek-V4.1-Flash's monokernels (sgl-project#43497 with K2 and the DSpark capture fix)
moved onto the monokernel plugin (RFC sgl-project#43949), without their env vars, and
extended to all 40 backbone layers (vLLM #61082's layout), with ATOM sgl-project#2545's
hand-off orderings:

- kernels/ops/mono/deepseek_v41_hip/: K1 (the attention seam and front: the
  pending post, the mixes, wqkv_a, the q / kv norms, RoPE, the SWA KV store,
  wq_b) and K2 (the attention core, wo_a, wo_b, the FFN seam, the MoE and both
  all-reduces), on common_hip's guarded waits; the spec lists gfx950 as its
  only ISA. K1 takes the peer addresses too, so a timeout in any launch flags
  every rank. The runner (DSV41MonoRunner) checks every tensor, cache and key
  table it is given; weights.py holds the layout conversions (E8M0 codes,
  exact MXFP8, the RoPE tables).
- Every backbone layer runs whole (LAYER_IN -> LAYER_OUT):
  - K1 folds the previous layer's pending post into the residual. Where none
    is owed (layer 0's broadcast embedding; the Engram layers 1 and 14, whose
    Engram write SGLang's loop runs on the settled residual) it takes the
    residual as it is, collapsed with the previous pre (copy 0 on layer 0).
  - The index sources (2, 8, 14, 20, 24, 28, 32, 36): K1 also writes the normed
    input and the bf16 qr, SGLang's forward_low_ratio_sources runs the layer's
    compressor and indexer on them between the launches, as SGLang runs them
    before the attention, and K2 builds the key table from the top-k they
    wrote (each split task waits on its token's flag).
  - The window-only layers 0 and 1 read no compressed cache.
- deepseek_v41_hip/spec.py: the spec and its SGLang glue. Each span returns
  what the HIP fused mHC boundary returns, with the layer's post pending for the
  next layer. Per-launch TOLERANCE bounds, read by the kernel checks. The load
  hook keeps the FP8 shared expert, which --enforce-shared-experts-fusion
  converts to MXFP4; the other weights are read in place. Any
  --fp8-gemm-backend but aiter is refused at config time (the others reorder
  wqkv_a, wq_b and wo_b in place; K1 and K2 read them row-major). The kernels
  never quantize more than the normal path: K2 keeps the attention output in
  bf16 for wo_a (its FP8 weight widened to bf16 in registers), as SGLang's
  bf16 wo_a reads it. Each rank builds every width, starting at its own share
  (FlyDSL's disk cache hands it the rest).
- From ATOM sgl-project#2545, bit for bit: the FFN seam's loads go out before the wait on
  the ranks' attention partials, a norm polls its rows once, after loading its
  weights, and the seams' Sinkhorn gates (read only by the next launch) run
  after K1's front and after K2's shared expert.
- deepseek_v4.py / deepseek_v4_hip.py / deepseek_v4_fused_mhc.py: the seams
  (the load hook, attach, route once a forward, a span at each start point).
  Under a plan the covered layers take DSpark's captures (the residual each
  target layer's boundary rebuilds), so the layers before DSpark's target
  layers keep their post pending for K1. With --enable-monokernel off nothing
  changes.
- Tests: the spec on CPU (test/registered/unit/mono); K1 (each input form: a
  pending post, a settled residual, layer 0, an index source's outputs), K2
  (an index source's and a window-only layer's bit for bit against the K2 of
  the same keys) and the FFN launch under torchrun at TP2 and TP4, with graph
  replay across widths (test/registered/kernels/ops/mono, helpers in
  sglang.test.kernels.mono); serving at TP4 (test/registered/e2e/mono, also a
  nightly job): DSpark verify and plain decode steps past the largest width and
  back (GSM8K, and the covered steps' tokens scored again by the normal
  prefill path, KL), and a fault armed on one rank mid-request: every rank
  stops, no token is sent, the only faults are the deliberate ones at 0x0. The
  chained stress stays manual.
- test/manual/kernels/mono/validate.py runs the CPU tests, the kernel checks,
  the serving tests and a paired decode A/B at 1, 4, 8 and 16 concurrent
  requests against baselines.json, and prints PASS or FAIL for each step.
  .agents/skills/monokernel gives agents the layout, the seams, the invariants
  and how to repair, tune or add a model.

MI350X TP4 DSpark, simulated acceptance 3.51, per-user decode tok/s at 1 / 4 /
8 / 16 concurrent, both arms in one validate.py run: 453.5 / 328.9 / 283.7 /
219.7 without --enable-monokernel, 579.0 / 399.8 / 303.9 / 221.0 with it
(+28% / +22% / +7% / +1%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhaochenyang20 zhaochenyang20 zhaochenyang20 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shuaills @zhaochenyang20