TensorRT Edge-LLM 0.11.1 Release 2026-10-28

We are excited to announce TensorRT Edge-LLM 0.11.1!

Frontend and models

• Added the unified trtedge CLI for HTTP serving, direct engine builds, ONNX export, quantization, and model preparation
• Added interactive text chat with streaming, conversation history, saved settings, and per-response timing metrics
• Added Nemotron 3.5 Bolt E4B with its DFlash Draft model
• Expanded DFlash2 checkpoint support for Qwen3.8 and Muse-Glimmer drafts
• Added Gemma 4 12B DFlash draft export and quantization for deepseek-ai/dflash_gemma4_12b_block7 and z-lab/gemma4-12B-it-DFlash

Runtime and builds

• TensorRT RTX-1.7 supports engine building and inference on Windows through Edge-LLM
• Added checkpoint-backed NVFP4-A16 MoE weight preparation and optional externalization of NVFP4-A16 LM-head weights
• Added TensorRT-RTX JIT-kernel caching to reduce repeated runtime initialization work
• Improved CLI diagnostics, cancellation during checkpoint loading, and handling of read-only tokenizer caches and missing optional dependencies
• Fixed CUDA 11.4 and Windows C++ unit-test compilation

Correctness fixes

• Fixed Gemma 4 raw-prompt BOS handling, multimodal placeholder embeddings, audio prompts, FP8 KV-cache scaling, and INT4 AWQ calibration
• Fixed DFlash version selection for paired engine builds and Muse-Glimmer chat-template handling of disabled thinking
• Fixed DiffusionGemma padded prefill on FMHA-v2 targets, Qwen3-Omni audio configuration, and VLM LoRA decoder selection
• Allowed Cosmos3 reasoning to start when optional policy-generation engines cannot be built
• Fixed a TensorRT-RTX FP8 fully connected fusion issue

TensorRT Edge-LLM 0.11.0 Release 2026-09-29

We are excited to announce TensorRT Edge-LLM 0.11.0!

In this release, we added wheel support, officially support IGX Thor for healthcare industry, added DFlash2 speculative decoding, expanded our support of Muse Glimmer, Pi05, and Tencent Hunyuan. We also launched our first version of In-Flight Batching for OpenAI Compatible Server!

New features and models

• Added published Python wheels for qualified x86-64 and AArch64 platforms, with exact native-payload selection and target-specific local wheel builds
• Added NVIDIA IGX Thor source and wheel support for its SM110 integrated GPU and SM120 discrete GPU
• Added Muse-Glimmer FP16 vision-language inference, NVFP4 text inference, DFlash, DFlash2, streaming, and native tool-call support
• Added opt-in single-rank in-flight batching to the experimental OpenAI-compatible server
• Added Qwen3.8 DFlash2 speculative decoding
• Added experimental pi0.5 FP16 policy inference for LIBERO, DROID, and ALOHA contracts
• Added Hunyuan dense model support for HY-MT1.5-7B and HY-MT2-7B
• Replaced copied chat-template schemas with native C++ rendering of provider-owned Jinja templates for structured, multimodal, reasoning, and tool-calling requests
• Added XGrammar guided decoding for JSON, JSON Schema, regex, EBNF, structural tags, and choices
• Added CodePredictor speculative decoding for Qwen3-Omni and Qwen3-TTS audio generation
• Expanded Nemotron and Gemma speculative decoding with DSpark, greedy trees, reduced-vocabulary chain MTP, and quantized drafts
• Added GPU-free CuTe DSL AOT artifact generation for every supported target architecture
• Added Qwen3.5 tensor-parallel inference for multi-device deployments

Other Important Features

Runtime and performance

• Added native Cosmos3-Edge reasoner video input
• Added fused RGB/NV12 preprocessing and copy-free handoff of caller-owned PCM buffers
• Added pre-tokenized requests and online benchmark scheduling
• Added incremental tool-call streaming
• Added bounded Gemma 4 sliding-window KV storage, DiffusionGemma prompt-KV reuse, and GPU media hashing for context-cache entries
• Improved Blackwell performance with batched DART pruning, ragged paged FMHA, PDL-enabled decode/RoPE/GDN paths, and CuTe DSL LayerNorm
• Added Thor-optimized NVFP4-A16 dense and routed-MoE kernels

Correctness fixes

• Fixed tokenizer special-token handling, Gemma 4 BPE merging, and chat-template resolution for Qwen3-Omni thinking behavior and Nemotron-Omni multimodal messages
• Fixed INT4, NVFP4, FP8 KV-cache, tensor-parallel, audio calibration, and speculative-draft export accuracy issues
• Fixed ragged inference, MTP and tree shape contracts, long-context admission with KV reuse, and speculative accept-length synchronization

TensorRT Edge-LLM 0.10.1 Release 2026-09-03

We are excited to announce the release of TensorRT Edge-LLM 0.10.1!

TensorRT Edge-LLM 0.10.1 adds experimental TP=2 inference across two NVIDIA DGX Spark systems, JetSpec speculative decoding for Qwen3-8B, Qwen3.8-27B DSpark support, end-to-end FP8 ViT attention and DART visual-token pruning. This release also redesigns the experimental OpenAI-compatible server to achieve a 60% - 99% reduction of overall disk usage, and a 50% reduction of cold launch time.

Key Features

• Added experimental TP=2 tensor-parallel inference across two NVIDIA DGX Spark systems using MPI and NCCL.
• Added JetSpec speculative decoding support for Qwen3-8B.
• Added Qwen3.8-27B DSpark support.
• Added end-to-end FP8 ViT attention and DART visual-token pruning to reduce VLM prefill work.
• Added NVFP4 MoE build and runtime support for NVIDIA Nemotron 3 Super 120B-A12B checkpoints with 512 routed experts and top-32 routing.
• Redesigned the experimental OpenAI-compatible server for faster cold launches and lower memory usage.

Other Important Features

Runtime and Performance

• Expanded context-cache reuse for speculative decoding, including hybrid and MTP models.
• Extended MTP verification trees to depth 16.
• Added logit_bias support on speculative decoding paths.
• Added a media artifact cache that retains ViT and audio encoder embeddings across repeated media prefixes.
• Added Blackwell SM110 dense NVFP4-A16 GEMM and GEMV kernels.
• Improved Blackwell performance with single-launch D80 SSD prefill, faster SM12x fused MoE, fused GDN projections, and one fewer per-layer KV copy.
• Added a CuTe DSL RMSNorm plugin.
• Upgraded CuTe DSL to 4.7.0.
• Enabled FMHA-v2 and XQA on Hopper SM90 with JIT-compiled XQA kernels.
• Added mock engine based unit tests.

Export and Quantization

• Added NVFP4 W4A16 export for official LM heads and Qwen3 A16 MoE.
• Fixed export for Gemma 4 Unified quantized checkpoints, Qwen NVFP4 mixed-precision dispatch, fused-MoE MTP drafts, and thinking-default chat templates.

Server and API

• Redesigned the experimental OpenAI-compatible server to utilize the checkpoint-direct builder, enabling single-command model construction and serving alongside profile-aware engine caching.
• Added one-command server deployment for speculative-decoding checkpoints, including MTP, EAGLE3, DFlash, JetSpec, and DSpark.
• Upgraded reasoning capabilities with automatic and selectable parsers for Qwen3, DeepSeek-R1, and Nemotron, featuring incremental streaming via reasoning_content.
• Enabled Nemotron-Omni video input support for the OpenAI Compatible server.

Platform Support and Documentation

• Added architecture-aware Python wheel packaging for x86, Jetson, DRIVE, and DGX Spark.
• Added QNX cross-compilation support
• Added Hopper support for development
• Added TensorRT-RTX as a backend
• Expanded documentation for multi-device inference, the experimental server, memory monitoring, KV-cache reuse, and visual-token pruning.

Bug Fixes

• Fixed Int4GroupwiseGemmPluginV2 accuracy for QKV concat aliasing, split-K workspace isolation, and LoRA insertion.
• Fixed speculative decoding with paged KV Cache issues
• Fixed context reuse for cross-batch image contamination and zero-budget encoder caches.
• Fixed the Cosmos3 Reasoner visual patch layout and batched audio embedding concatenation.
• Fixed cross-platform GPU memory reporting.
• Fixed #162: InternVL3 AWQ export now supports checkpoints that provide PyTorch .bin weights without safetensors. 8B model is removed due to unsupported tokenizer.
• Fixed #163: Qwen2.5-VL and Qwen3-VL AWQ vision export now accepts the num_attention_heads configuration key.
• Fixed #184: Qwen2.5 and Qwen2.5-VL export now accepts standard Hugging Face cache shard symlinks while retaining path-escape validation.
• Fixed #166: Confirmed that Alpamayo-R1-10B export no longer hits the ONNX dynamic-shape assertion reported against 0.9.1.

NVIDIA Contributors

@nvluxiaoz @nvamberl @xiangg-nv @willg-nv @mahu888 @nv-samcheng @duofant @JCalafato @duanyaqi @zhazhang-nv @zhaoyuanh-nvidia @zhijial-nvidia @ever-wong @Jasper-NV @jhalabi-nv @levichen-nvidia @sunghyunp-nvidia @nvyocox @ruocheng-nv @yuanyao-nv @qikail-ctrl @MadFunMaker @kevinfu1

TensorRT Edge-LLM 0.10.0 Release 2026-08-12

We are excited to announce the release 0.10.0 of TensorRT Edge-LLM!

TensorRT Edge-LLM 0.10.0 provides Day-0 support for Qwen3.8-27B, as well as for NVIDIA Nemotron-3.5 Lightning with MTP and DFlash. The release adds support for Cosmos3-Edge, DiffusionGemma, Nemotron-3.5-ASR, and DSpark speculative decoding. It also introduces an experimental direct TensorRT engine builder without ONNX export, multi-turn KV-cache reuse, and video input for the experimental OpenAI-compatible server.

Key Features

• Added Day-0 support for Qwen3.8-27B.
• Added NVIDIA Nemotron-3.5 Lightning support with MTP and DFlash speculative decoding.
• Added an experimental ONNX-less workflow to build TensorRT engines directly from supported checkpoints.
• Added Cosmos3-Edge multimodal reasoning and experimental policy/action-generation support.
• Added DiffusionGemma NVFP4 block-diffusion inference.
• Added experimental Nemotron-3.5-ASR inference support.
• Added DSpark speculative decoding with DDTree drafting.
• Added paged KV-cache reuse for multi-turn LLM requests and media-aware KV reuse for VLM requests.
• Added video input support to the experimental server.
• Expanded Qwen3-TTS with language and instruction conditioning, VoiceDesign, and on-device voice cloning. Thanks to @suharvest for the CustomVoice language-conditioning contribution in GitHub PR #98.

Other Important Features

Runtime and Performance

• Replaced the legacy embedded-cubin FMHA-v2 backend with CuTe DSL FMHA-v2 and removed the checked-in FMHA-v2 cubin artifacts.
• Replaced FFPA with dense FMHA-v2 kernels for head dimension 512.
• Added Blackwell paged FMHA-v2 kernels for head dimension 512.
• Added NVFP4-A16 Marlin MoE and migrated the INT4 AWQ plugin to the CuTe DSL kernel path.
• Added Qwen3-Omni full-loop CUDA Graph execution.
• Added GPU bicubic resizing for ViT image preprocessing.
• Fixed DFlash verification, MTP/GDN accepted-state replay, and Qwen3-Omni video temporal MRoPE correctness.

Export and Quantization

• Added paired-checkpoint MTP support for Gemma 4 12B.
• Added Gemma 4 26B-A4B INT4 AWQ MoE and Qwen3-Omni 30B-A3B INT4 AWQ quantization and export support.

Server and API

• Added Qwen3-Omni streaming audio output, /v1/audio/speech for Qwen3-TTS, and /v1/audio/transcriptions for Qwen3-ASR.
• Added Claude Code and OpenClaw integration to the experimental server.

Examples and Documentation

• Added NeMo Evaluator integration.
• Added an official platform and software support matrix.

NVIDIA Contributors

@nvluxiaoz @nvamberl @xiangg-nv @willg-nv @mahu888 @nv-samcheng @duofant @Caohanwen0 @JCalafato @duanyaqi @zhazhang-nv @zhaoyuanh-nvidia @zhijial-nvidia @ever-wong @Jasper-NV @jhalabi-nv @levichen-nvidia @sunghyunp-nvidia @nvyocox @ruocheng-nv @yuanyao-nv @qikail-ctrl

TensorRT Edge-LLM 0.9.1 Release 07/23/2026

We are excited to announce the release 0.9.1 of TensorRT Edge-LLM!

TensorRT Edge-LLM 0.9.1 expands Gemma 4 multimodal support, DFlash, paged KV cache, and OpenAI-compatible generation controls while improving runtime performance and stability.

Key Features

• Added Gemma 4 E2B, E4B, 12B, 26B-A4B (nvfp4), 31B full modality support with MTP.
• Expanded DFlash with DDTree for Qwen3 and Qwen3.5 hybrid models and draft vocabulary reduction.
• Added paged KV-cache prefill and paged XQA decode support.
• Added online GPU filter-bank preprocessing for Whisper/Qwen3 audio models and Parakeet/Nemotron-Omni, plus Qwen vision video preprocessing.
• Added logprobs across vanilla and speculative decoders and OpenAI-compatible logit_bias requests.

Other Important Features

Runtime and Performance

• Added INT4-FP16 CuTe DSL GEMM/GEMV kernels for SM80+.
• Added CuTe DSL support for the FP16 MoE plugin, including 128/256-expert configurations.
• Improved FMHA for short sequences, partial prefill, and head dimension 256.
• Improved attention configuration validation to fail unsupported configurations cleanly.
• Fixed implicit stream synchronization, streamed engine deserialization, logger teardown, duplicate symbols, and FP8 long-context accuracy.

Export and Quantization

• Added named, per-modality calibration dataset selection
• Added Qwen3-Omni/Qwen3-TTS CodePredictor FP8 quantization.

Server and API

• Added speculative-decoding engine-directory routing through the Python LLM API.
• Fixed greedy sampling fallback handling for speculative decoding.

Examples

• Fixed some llm_bench issues.

NVIDIA Contributors

@nvluxiaoz @nvamberl @xiangg-nv @willg-nv @mahu888 @nv-samcheng @duofant @Caohanwen0 @JCalafato @zhazhang-nv @zhaoyuanh-nvidia @zhijial-nvidia @ever-wong @Jasper-NV @wanghr323 @jhalabi-nv @levichen-nvidia @sunghyunp-nvidia @nvyocox @fans-nv @ruocheng-nv @yuanyao-nv @duanyaqi @poweiw

TensorRT Edge-LLM 0.9.0 Release 2026-07-03

We are excited to announce the release 0.9.0 of TensorRT Edge-LLM!

TensorRT Edge-LLM 0.9.0 officially supports DGX Spark, expands speculative decoding with DFlash, adds Qwen3-Omni-30B-A3B nvfp4 end-to-end support, gemma4 e2b and e4b text inference, C++ audio preprocessing and server audio input for supported audio models. It also fixes a decode performance regression introduced in 0.8.0.

Breaking Changes

• Audio preprocessing has moved into C++ runtime/server paths for supported audio models. The Python tensorrt-edgellm-preprocess-audio package entry point is no longer present.
• NVFP4 MoE backend selection no longer uses --nvfp4-moe-backend or QuantConfig.nvfp4_moe_backend. Export target selection is now handled by EDGELLM_NVFP4_MOE_TARGET.
• Speculative decoding build artifacts now use generic names such as spec_base.engine and spec_draft.engine. Use --specBase and --specDraft; --eagleBase and --eagleDraft remain deprecated aliases.
• Speculative decoding runtime flags now prefer --specVerifySize; --specVerifyTreeSize remains available as a compatibility alias.
• Occasionally, FP8 ViT for some models might have NaN issues. If encountered, falling back to FP16 ViT will resolve the issue.

Key Features

• Officially supports NVIDIA DGX Spark.
• Added Gemma 4 E2B/E4B text inference support.
• Added DFlash speculative decoding support, including export, runtime decode integration, target KV materialization kernels, DFlash draft quantization, and Qwen3/Qwen3.5 validation coverage.
• Added Qwen3-Omni-30B-A3B NVFP4 support for Thinker and Talker quantization/export.
• Fixed a decode performance regression introduced in 0.8.0.
• Expanded Nemotron3 NVFP4 support for Jetson Thor.
• Added C++ audio decoding and mel preprocessing for multimodal audio models and server audio input.
• Improved NVFP4 MoE support for SM100/101/110, including E=256 experts, SM110 split FC1/FC2 CuTeDSL kernels, and SM12x fused MoE plugin coverage.
• Added XQA sliding-window support and head-dim 512 kernels across supported SM targets.

Other Important Features

Runtime and Performance

• Improved CuTeDSL attention performance and added FFPA-style Ampere-floor FMHA CuTeDSL kernels.
• Added Qwen3-Omni TTS standalone streaming and unified Talker/CodePredictor runtime handling.
• Fixed MTP speculative decoding configuration validation.
• Fixed an EAGLE/MTP accept-kernel race on nextTokenIdx.
• Fixed SSD prefill accuracy and SSD TMA bounds issues.
• Migrated llm_bench from legacy runners to EngineExecutor and removed dead legacy runtime code.

Export and Quantization

• Forced ONNX exports to use external data files.
• Fixed Qwen3.5 MoE GPTQ export and symmetric GPTQ checkpoints without zero points.
• Added Qwen3.6-35B-A3B NVFP4 export.
• Fixed InternVL3.5 visual weight-key prefix handling for quantized checkpoints.

Server and API

• Added experimental server audio input handling for OpenAI-compatible input_audio, audio_url, and local audio content forms.
• Added server response metadata fields such as created, model, and token usage placeholders.
• Expanded speculative decoding server wording and handling beyond EAGLE/MTP to include DFlash.
• Added TTS streaming latency reporting for time-to-first-codec and time-to-first-playable-audio.

Documentation

• Added DFlash speculative decoding documentation and supported draft-model mappings.
• Added a new Qwen3-Omni workflow covering quantization, export, engine build, text/audio/image input, and speech output.
• Updated MoE documentation for Nemotron3 NVFP4 and NVFP4 MoE export behavior.
• Updated ASR and server documentation for C++ audio preprocessing.
• Updated 0.8.0 performance data

NVIDIA Contributors

@fans-nv @nvluxiaoz @nvamberl @ruocheng-nv @poweiw @Jasper-NV @charllll @mahu888 @xinrzhang-star @JCalafato @ever-wong @zhijial-nvidia @willg-nv @levichen-nvidia @xiangg-nv @nv-samcheng @jhalabi-nv @nvmbreughe

TensorRT Edge-LLM 0.8.0 Release 2026-06-02

We are very excited to announce release 0.8.0 of TensorRT Edge-LLM!

TensorRT Edge-LLM 0.8.0 completes the transition to the checkpoint-based export workflow. The tensorrt_edgellm package is now the official front-end for quantization and ONNX export.

This release has been validated on the primary NVIDIA edge platforms, including Jetson Orin and Thor-class devices, and adds support to the just released Jetpack 7.2, which can be downloaded from Nvidia’s official site. As a general sizing guideline, a model is expected to run when the target device has at least the model size plus approximately 3 GB of available memory for runtime overhead. Please see our updated Quick Start Guide, Supported Model List, and model examples for the latest deployment guidance.

Breaking Changes

• The legacy ONNX export workflow has been removed. The checkpoint-based workflow is now the official TensorRT Edge-LLM export path.
• experimental/llm_loader and experimental/quantization functionality has moved into the tensorrt_edgellm package.
• Use the unified CLI commands:
  • tensorrt-edgellm-quantize
  • tensorrt-edgellm-export
  • tensorrt-edgellm-merge-lora
  • tensorrt-edgellm-reduce-vocab
  • tensorrt-edgellm-preprocess-audio
• Older per-component export commands such as tensorrt-edgellm-export-llm, tensorrt-edgellm-export-visual, and python -m llm_loader.export_all_cli should be replaced by tensorrt-edgellm-export.

Key Features

• Promoted tensorrt_edgellm as the official checkpoint-based quantization and ONNX export workflow, with unified CLI commands for export, quantization, LoRA, vocabulary reduction, and audio preprocessing
• Added Qwen3/Qwen3.5/Qwen3.6 MoE NVFP4 export
• Reduced engine build memory for INT4 FFN, INT4 MoE, and LM-head weights
• Added OpenAI-compatible server tool-calling support, per-request stop strings, and Qwen3-TTS streaming improvements
• Updated dependency pins, including PyTorch 2.12, Transformers 5.9, and ModelOpt 0.44

Other Important Features

Runtime and Performance

• Added pluggable decoding support for vanilla, EAGLE, and MTP runtime paths
• Added Qwen3-ASR export and quantization workflows, including independent LLM and audio-encoder precision selection, including FP8 Qwen3 ASR Audio Encoder
• Added fused gate+up and XQA kernel support for new MoE configurations
• Fixed XQA kernel-loader thread-safety and CUDA 13.3 MoE top-k softmax build issues

Export and Quantization

• Improved mixed-precision quantization handling for fused QKV and gate/up projections

Server and API

• Added experimental Dockerfiles for containerized development
• Expanded high-level Python API and server validation for LLM, VLM, and streaming flows

Documentation

• Updated documentation for checkpoint export, quantization, LoRA, FP8 KV cache, FP8 embedding, speculative decoding, ASR, TTS, and server usage

NVIDIA Contributors

@fans-nv @nvluxiaoz @nvamberl @ruocheng-nv @poweiw @yuanyao-nv @Jasper-NV @charllll @mahu888 @xinrzhang-star @JCalafato @nvmbreughe @nvxingkaiz @ever-wong @zhijial-nvidia @willg-nv @levichen-nvidia @xiangg-nv @nvyocox @nv-samcheng

TensorRT Edge-LLM 0.7.1 Release 2026-05-19

We are very excited to announce release 0.7.1 of TensorRT Edge-LLM!

• We welcome community contribution. We added Alpamayo-1-10B support in our software and thanks for @Turoad to raise #67. This MR provides insights for our implementation.
• We are excited to launch our pages on Jetson AI Lab. Please check more tutorials and model deployment guidelines there.

Deprecation Notice

• The original workflow used to quantize and export ONNX is replaced by the new checkpoint-based workflow to quantize and export ONNX. This workflow was released as "experimental" in prior versions, but is now the official front-end into TensorRT Edge-LLM. The original tensorrt_edgellm is deprecated and will be removed in 0.8.0.

Key Features

• Added Qwen3.5 MTP support
• Added Alpamayo-1-10B support
• Added Qwen3-TTS streaming support
• Added FP8 ViT and Qwen3-TTS support for experimental loader
• Added migration for customization and usage with the experimental loader
• Improved Mamba prefill kernel performance
• Rearchitected runtime with composable stacks
• Fixed #81 and other bugs

NVIDIA Contributors

@fans-nv @nvluxiaoz @nvamberl @ruocheng-nv @poweiw @yuanyao-nv @Jasper-NV @charllll @mahu888 @xinrzhang-star @JCalafato @nvmbreughe @nvxingkaiz @ever-wong @zhijial-nvidia @willg-nv @levichen-nvidia @xiangg-nv @maggie-j-liu @nvyocox @nv-samcheng

TensorRT Edge-LLM 0.7.0 Release 2026-04-28

We are very excited to announce release 0.7.0 of TensorRT Edge-LLM!

• We welcome community contribution and have merged #68 and #42. Thanks the contributors @matiaslin and @victoroliv2.
• We are excited to launch our pages on Jetson AI Lab. Please check more tutorials and model deployment guidelines there.

Deprecation Notice

• The experimental workflow is expected to reach full parity of the original workflow. The original tensorrt_edgellm is deprecated and will be removed in 0.8.0.

Key Features

• Introduced Day 0 Support of NVIDIA Nemotron 3 Nano Omni. Please review Jetson AI Lab on how to run TensorRT Edge-LLM for this model.
• Introduced an experimental High Level Python API and OpenAI Compatible Server. Please follow our Quick Start Guide, Introduction and Code.
• Implemented an agent-friendly experimental new workflow to quantize and convert models to ONNX. The quantized checkpoints are now compatible with major frameworks like TensorRT-LLM, vLLM and sgLang, and the ONNX conversion has 0 GPU consumption saves > 70% total memory consumption.
• Expanded pre-quantized models into our model support lists. Please see our updated [supported model lists].(https://nvidia.github.io/TensorRT-Edge-LLM/0.7.0/user_guide/getting_started/supported-models.html)
• Added AGENTS.md to assist development and users. Automatic agents to debug accuracy is added.
• Added performance dashboard for selected models. The dashboard will keep updating for newer versions and models.

Other Important Features

Model Support

• Added Qwen3.5 LLM/VLM support
• Added NVIDIA Nemotron Nano 3 30B A3B NVFP4 support
• Added Qwen3-ASR quantization workflow

Performance Improvements

• Added FP8 embedding support to reduce embedding-table memory
• Reduced runtime memory by sharing TensorRT execution context memory
• Added multi-batch cutedsl prefill kernels
• Improved performance by cutedsl-based Mamba SSD kernels

Runtime Extensions

• Unified LLM runtime execution paths for vanilla and EAGLE3 decoding
• Added per-slot streaming with the StreamChannel API
• Replaced LinearKVCache with per layer HybridCacheManager to support more hybrid models

Workflow Improvements

• Improved build workflow with automatic TensorRT detection and cutedsl kernel binary shipment
• Upgraded Transformers support to 5.x

NVIDIA Contributors

@fans-nv @nvluxiaoz @nvamberl @ruocheng-nv @poweiw @yuanyao-nv @Jasper-NV @charllll @mahu888 @xinrzhang-star @JCalafato @nvmbreughe @nvxingkaiz @ever-wong @zhijial-nvidia @willg-nv @levichen-nvidia @xiangg-nv @maggie-j-liu

TensorRT Edge-LLM 0.6.1 Release 2026-04-15

• Added DriveOS 7.2.4 official support
• Fixed EAGLE draft model weights loading issue to retrieve acceptance rate