SGLang v0.5.21 Unveils Dynamic Prefill-Decode Switching and Native Rust Prefix Caching for Extreme LLM Serving

核心背景与行业痛点

Serving modern large language models under heavy concurrency requires navigating fundamentally divergent hardware requirements: Prefill is compute-bound, demanding raw tensor FLOPS to ingest prompt contexts, while Decode is memory-bandwidth-bound, iteratively fetching KV caches. While Prefill-Decode (PD) disaggregation addresses this asymmetry, traditional deployments assign instance roles statically. Under volatile workloads, static clusters suffer from severe capacity skew and underutilized GPUs. Concurrently, prefix caching management implemented in Python frequently creates lock contention and memory fragmentation under dense multi-tenant traffic.

架构亮点与底层机制

SGLang v0.5.21 introduces transformative architectural improvements:

1. Zero-Restart Dynamic PD Role Switching: Instances dynamically transition between Prefill and Decode roles on the fly without service interruptions, enabling schedulers to adapt to traffic patterns in milliseconds.
2. Native Rust-Core Prefix Cache: The Radix Tree prefix cache engine is ported to a native Rust implementation and enabled by default, eliminating Python GIL bottlenecks and delivering sub-microsecond cache lookups.
3. Dedicated Decisions and Score APIs: Introduces /v1/decisions and /v1/score endpoints, empowering developers to utilize models as ultra-low-latency discrete classifiers and multi-candidate rerankers without autoregressive overhead.
4. Frontier Hardware & Model Expansion: Validates GLM-5.3-Flash across AMD MI355X accelerators with native FP8/MXFP4 MoE kernels and multi-token prediction (MTP) speculative decoding.

权威 Benchmark 与实测跑分对比

Benchmarked across industrial production traces:

1. 22% Lower TTFT on DeepSeek-V4.1: Yields a 22% reduction in time-to-first-token across 32k-128k long-context scenarios under distributed serving.
2. 20.6% Higher Prefill Throughput on Kimi K3: Demonstrates a 20.6% throughput gain under disaggregated PD serving.
3. Layer-Boundary Native Communication: Resolves tensor parallel discrepancies under PP, DP, and context parallelism (CP), stabilizing extended rollouts.

开发者实战落地与开箱指南

SGLang v0.5.21 is available via uv pip install --prerelease=allow sglang==0.5.21 and prebuilt Docker images on Docker Hub for CUDA 13 and ROCm 10. Platform engineering teams can deploy dynamic PD switching and the Rust cache engine to achieve over 20% aggregate throughput gains out of the box.