vLLM Serves DeepSeek-V4.1-Flash on NVIDIA Vera Rubin NVL72 at 1.7M Tokens/s per GPU with 387 Tokens/s per User, 58x GB300 NVL72 at That Point and 2.7x at Peak Throughput

Background and the problem

Agent workloads mean many sessions, long prefixes and high output speed for every user. In this high-interactivity range, throughput on older racks collapses, which is a core inference-cost problem. vLLM had already done PD disaggregation and Wide-EP on GB300 NVL72, and this post moves the same stack to NVIDIA's new Vera Rubin NVL72.

Architecture and how it works

Each Rubin GPU has two memory domains and SMs read their own much faster, so vLLM splits MoE weights and the residual stream by domain. SM107 adds a K64 FP8xFP4 MMA that does twice the work per instruction, giving MegaMoE 8-11% more throughput at 8K+ tokens per rank. All 72 GPUs share one NVLink domain, so TP8 can span two trays at almost no extra all-reduce cost. Dynamo routes requests by uncached work, keeping the prefix hit rate at 96.5-97%.

Benchmarks and measured results

All figures are vLLM's own, on the SemiAnalysis AgentX workload: 1.7M tokens/s per GPU at a P90 of 387 tokens/s per user (58x GB300), 2.52M tokens/s per GPU at peak (2.7x), and 1,810 tokens/s for a single session. The 58x reflects how sharply GB300 throughput drops at high interactivity, not a 58x overall speedup. Rubin numbers are preview results with no third-party replication yet.

Getting started for developers

Vera Rubin is not yet widely available, so most teams cannot use this setup soon. The methods still transfer: pick an operating point from a per-user-speed versus per-GPU-throughput curve for your session load, replace session affinity with a shared KV pool plus workload-aware routing, and start with vLLM's PD disaggregation and Wide-EP recipes on GB300 or B300.