SGLang v0.5.22 Released: DeepSeek-V4.1 Throughput Jumps 2.3x on H200 and 1.9x on MI355X, InstantTensor Loads Models 10x Faster

Background and the Problem

As next-generation MoE foundation models like DeepSeek-V4.1, Qwen3.8, and GLM-5.3 scale into production, enterprise serving stacks face severe memory bandwidth bottlenecks and slow initialization times across heterogeneous hardware clusters (NVIDIA H200/B200/B300 and AMD MI355X). Furthermore, long-context prompt prefilling severely degrades interactive decoding latencies without architectural isolation. The open-source serving engine SGLang released v0.5.22, featuring 950 PRs from 245 contributors that systematically overhaul serving efficiency across the stack.

Architecture and How It Works

1. DeepSeek-V4.1 Kernel and DSpark Acceleration (PR #41251, #41308):

• Delivers up to 2.3x higher throughput per GPU on NVIDIA H200 through optimized memory reuse kernels, and up to 1.9x throughput on AMD Instinct MI350X/MI355X via DSpark integration;

1. InstantTensor 10x Loader (PR #40453):

• Introduces zero-copy pinned memory weight ingestion, slashing large model load times by 10x for rapid container autoscale;

1. PD Disaggregation & Speculative Decoding (PR #40703, #43735):

• Prefill-Decode disaggregation lifts long-prompt throughput by 27% and cuts p95 TTFT by 50%. GLM-5.3 combines decode context parallelism with MTP for up to 2.1x faster token generation;

1. Broad Platform Support:

• Supports Apple Silicon Macs natively via standard PyTorch alongside MLX. Adds ComfyUI diffusion integration for MiniMax-H3 and Qwen-Image 2.1 with GGUF and LoRA support.

Benchmarks and Measured Results

Official benchmarks across NVIDIA H200/B200/B300 and AMD MI355X confirm:

• DeepSeek-V4.1 Throughput: 2.3x per GPU on H200; 1.9x throughput on AMD MI355X via DSpark;
• Next-Gen Accelerators: Qwen3.8-Flash-Next decodes 12% faster on B200; GLM-5.3-Flash runs 10% faster on B300, and MiMo-V2.6-Pro achieves 1.85x decode throughput with DFlash on B300;
• Interactive Latencies: PD disaggregation slashes p95 TTFT by 50% while increasing throughput by +27% on long prompts;
• Model Coverage: Day-0 serving for Clef, PPLX-Decider v1.1, EmbeddingGemma 2, LLaDA2.2, HunyuanImage-3.0-Instruct, and SANA-Video 2.0.

Getting Started for Developers

• Upgrade Command:

  pip install --upgrade "sglang[all]>=0.5.22"
  

• Production Tip: Enable --enable-instant-tensor to slash Kubernetes pod restart times by 90%, and deploy --enable-pd-disaggregation on long-context agent pipelines to cut p95 response latencies in half.