Kimi K3 Hits 842 tok/s on NVIDIA GB300 NVL72: NVIDIA and vLLM Unveil 12,000-Line CuTe DSL Decode Megakernel with 1.8x Higher AgentX Interactivity

Core Background & Industry Pain Points

As frontier Mixture-of-Experts (MoE) foundation models scale into the multi-trillion-parameter regime—exemplified by Moonshot AI's Kimi K3 (reported by SemiAnalysis at 2.8 trillion parameters)—low-latency token decoding on rack-scale NVIDIA GB300 NVL72 superpods hits a wall dominated by kernel launch overhead and fine-grained inter-GPU synchronization. In interactive coding agent workloads, GPU Tensor Cores are frequently starved while waiting on 23 separate CUDA kernel launches per transformer layer across attention, MoE routing/GEMMs, and cross-GPU NVLink collectives.

Architectural Highlights & Core Mechanisms

Today, vLLM (@vllm_project) and NVIDIA unveiled a unified CuTe DSL Decode Megakernel for Kimi K3 authored by NVIDIA (CentML) engineer Benjamin Chislett, alongside an independent hardware benchmark analysis by SemiAnalysis (@SemiAnalysis_):

• 12,000-Line CuTe DSL Megakernel Fusion: Written in NVIDIA's CuTe DSL, the new decode path fuses attention, MoE expert computation, and multi-GPU communication into a unified execution flow.
• Per-Layer Latency Slashed from 130 µs to 69 µs: Across each Kimi K3 transformer layer, kernel count drops from 23 kernels down to 13 kernels, cutting per-layer execution time by 46.9% (from 130 µs to 69 µs).
• DSpark Speculative Decoding & Upcoming DeepSeek-V4.1-Flash / GLM-5.3 Support: The megakernel composes directly with vLLM's DSpark speculative decoding stack while passing strict long-context numerical evaluations. Equivalent megakernel pipelines are already underway for DeepSeek-V4.1-Flash (vLLM PR #61868) and GLM-5.3.

Authoritative Benchmarks & Empirical Comparisons

Evaluated on NVIDIA GB300 NVL72 across batch sizes, context lengths, and parallelism topologies:

• 842 tokens/s at Batch 1 with DSpark Speculative Decoding: On 16 GB300 GPUs, single-request decode speed jumps from 488 tokens/s on stock vLLM to 842 tokens/s (1.73x faster).
• 1.58x–1.88x Pure Decode Speedups Across Batch Sweeps: Without speculative decoding, batch=1 throughput reaches 193.3 tok/s on 8 GPUs (1.58x) and 256.4 tok/s on 16 GPUs (1.88x); at batch=64, throughput hits 2,375.9 tok/s on 8 GPUs (1.31x) and 3,606.1 tok/s on 16 GPUs (1.62x).
• Up to 1.8x Higher P90 Interactivity on SemiAnalysis AgentX: On the SemiAnalysis AgentX workload, vLLM reports P90 interactivity rising from 223.7 to 403.2 tokens/s per user (1.8x), while @SemiAnalysis_ highlights a 423 tok/s per user milestone (1.68x faster on identical hardware).

Developer Practical Guide & Quickstart

Infrastructure teams deploying Kimi K3 on NVIDIA GB300 NVL72 can review the full benchmark sweep and parallelism configurations in the official write-up (https://vllm.ai/blog/2026-10-16-kimi-k3-megakernel) and track vLLM PR #61868 for upcoming DeepSeek-V4.1-Flash and GLM-5.3 CuTe DSL megakernel integration.