vLLM and Ant Group open-source Humming, one quantized-inference kernel library for many formats and every NVIDIA GPU since Turing, lifting DeepSeek-V4.1-Flash end-to-end throughput by 14-43%

Background and the pain point

Quantization is the main way to cut LLM serving costs, and MoE models need it most: each token activates only some experts, but all weights must sit in GPU memory. The trouble is the growing number of schemes, WNA16, W8A8, W4A8, W4A4, plus formats such as INT4, NVFP4 and MXFP4 with different scaling. Each combination needs a tuned kernel on each GPU generation before saved memory turns into speed, and the vLLM blog says building them one by one has become a heavy maintenance burden. Ant Group and the vLLM community open-sourced Humming to fix that.

Architecture and mechanism

Humming first converts models from different quantization tools into one standard format, describing weight format, scaling granularity and zero points as separate parameters. Three configs (layer, compute and tuning) then describe a kernel, which NVRTC or NVCC compiles just in time. Data movement, dequantization, Tensor Core math and output handling are reusable components, so a new format or architecture only means extending the relevant component. In vLLM's test one W4A16 kernel compiled in 1.82 s versus 23.35 s for CUTLASS with NVCC, and preparing kernels for every batch size of one layer during warmup takes about 2-4 s. On Hopper, which lacks native MXFP4 Tensor Cores, it converts MXFP4 weights to FP8 inside the kernel so H200 can run W4A8-MXFP4 efficiently.

Benchmarks

All figures come from the vLLM community's own tests. Dense W4A16 matches or beats vLLM Marlin in most configs, reaching about 2x Marlin on large-batch H200 and over 3x in some B300 cases; the B300 MoE Grouped path reaches about 5x Marlin. W4A8-MXFP4 at small and medium batch sizes is about 4.2x DeepGEMM W8A8 on H20 and 4.5x on H200. End to end, DeepSeek-V4.1-Flash on 8x H20 at TP=8 with MXFP4 weights unchanged gained 14-43% throughput and 25-32% lower mean TTFT when MoE switched from Marlin W4A16 to Humming W4A8; at concurrency 64 output rose from 442.37 to 633.94 tok/s. These results cover one model on one GPU type, so re-test on your own setup.

Getting started

Install standalone with pip install humming-kernels (about a 500 KB wheel, kernels compiled on demand), or humming-kernels[cu12] / [cu13] if no CUDA compiler is present. Code is at GitHub vllm-project/humming under Apache 2.0. vLLM 0.29.0+ CUDA builds already include it; add --linear-backend humming --moe-backend humming to enable it, and use --quantization humming to load formats vLLM lacks natively, such as GPTQ INT3. Teams self-hosting MoE models, especially MXFP4 models on H20 or H200, should A/B it first. Planned next are autotuning, more Rubin features and more T-Head devices.