SemiAnalysis Discloses Google & Inferact TPUv7 Submission: TorchTPU vLLM Megakernels Drive 2.8T Kimi K3, Surpassing GB200 NVL72 and B300 in High-Interactivity TCO

Background and the Problem

As massive ~2.8T-parameter Mixture-of-Experts (MoE) agent models like Moonshot Kimi K3 enter production, serving costs and user interactivity (tokens/s per user) become critical battlegrounds. For interactive coding agents, developers demand streaming latencies exceeding 200–400 tok/s to enable real-time flow states. However, traditional GPU clusters suffer steep TCO degradation at ultra-high interactivity due to communication overhead and memory bandwidth saturation. SemiAnalysis released an exclusive analysis covering Google and Inferact's (makers of vLLM) first preliminary TPUv7 submission on the InferenceX platform.

Architecture and How It Works

1. Hardware Cost Advantage ($1.21/chip-hour):

• Based on large-hyperscaler volume TCO calculations, Google TPUv7 costs just $1.21 per chip-hour, compared to AMD MI355X ($1.50), NVIDIA B200 ($1.73), GB200 ($1.86), B300 ($2.26), and GB300 ($2.31);

1. TorchTPU vLLM Stack & Megakernel Execution:

• Led end-to-end by Inferact's Jeff Ma, George Novack, and Google, the stack leverages fused megakernels customized for TPUv7 MXUs alongside the TPU-Sync KVCache transfer and offloading library for prefill-decode (PD) disaggregation, demonstrating first-class production readiness for the PyTorch/vLLM stack on Google silicon.

Benchmarks and Measured Results

From the Kimi K3 2.8T Agentic Total Tokens per $1 TCO vs. P90 Interactivity benchmark curves:

• Ultra-High Interactivity Regime (~390 tok/s per user):
• TPUv7 achieves ~2.2M tokens per $1 TCO, outperforming NVIDIA GB200 NVL72, B200, B300, and AMD MI355X;
• AMD MI355X truncates around 130 tok/s and NVIDIA B300 around 270 tok/s, whereas TPUv7 smoothly sustains interactivity up to ~400 tok/s;
• Comparison with GB300 NVL72:
• While GB300 leads at low interactivity (31M tokens/$1 at 25 tok/s), at ~400 tok/s (~2.5M tokens/$1 for GB300), TPUv7's lower cost per chip-hour ($1.21 vs $2.31) renders it highly competitive on a TCO basis.

Getting Started for Developers

• Ecosystem Shift: Proves that vLLM has matured into a truly portable runtime across NVIDIA (CUDA), AMD (ROCm), and Google (TPU) hardware;
• Deployment Strategy: Enterprise clusters serving massive MoE models can leverage the open TorchTPU vLLM stack to exploit Google Cloud TPUv7 pods, unlocking compelling TCO efficiencies for interactive coding agents.