vLLM and NVIDIA ship a specialized Kimi K3 decode path on GB300: 842 tok/s at batch 1, 1.3–1.9x faster without speculation

Core Background & Industry Pain Points

For huge MoE models like Kimi K3, low-concurrency decode speed decides how responsive a coding agent feels. Small-batch inference is bound by memory bandwidth and kernel launch overhead, and stock vLLM launches 23 kernels per K3 layer. Meanwhile agent-written optimizations now outpace maintainers' line-by-line review, while merging blindly risks production bugs.

Architecture Highlights & Mechanism

With coding-agent assistance, the team built a unified CuTe DSL decode path for Blackwell (8 or 16 GPUs in one NVLink domain, batch 1–64): batch-aware small-matrix or split-K kernels, attention output projection fused with all-reduce and the residual step, router scoring split across TP ranks and exchanged over NVLink, a separate stream for the shared expert, and weight prefetch during communication. Decode context parallelism handles long contexts. It follows the fast-track RFC (#59665): keep changes inside model code, bound the scope, and validate correctness and performance systematically.

Benchmarks & Measured Performance

All figures are from vLLM/NVIDIA. With DSpark (7 draft tokens, synthetic acceptance length 6) on 16 GPUs, batch-1 decode hits 842 tok/s vs 488 stock, 1.73x; this is a controlled synthetic setting and real acceptance will be lower. Without speculation, TP16 reaches 256.4 tok/s at batch 1 (1.88x) and 3,606 tok/s at batch 64 (1.62x); TP8 gains 1.31–1.58x. AgentX P90 interactivity is 403.2 vs 223.7 tok/s/user, but the baseline is an earlier 8-GPU run.

Developer Quick Start

PR #61867 was merged into vLLM on Oct 16 and targets GB300/Blackwell multi-GPU deployments; per the sibling DeepSeek PR it is enabled through the opt-in decode_backend: latency_specialized switch, so check the PR for exact flags. The DeepSeek-V4.1-Flash port (#61868) is still under review, and a GLM-5.3 version is promised later.