vLLM v0.32.0: 814 commits, built-in /v1/systemone structured-decision endpoint, NVFP4 KV cache on more GPUs, and 43–85% more req/s with multiple API workers

Core Background & Industry Pain Points

Open-source inference engines now compete on memory efficiency, cold-start time and standard interfaces rather than just day-0 model support. Long-context and MoE models make the KV cache the main memory cost, and decision models that return option probabilities in one forward pass previously needed custom glue. vLLM shipped v0.32.0, 814 commits, less than two weeks after v0.31.0 (Oct 5).

Architecture Highlights & Mechanism

The release serves POST /v1/systemone by default, matching the endpoint already in Ollama and llama.cpp, for choice, noul and score questions. A generic hf parser reads each checkpoint's response_template, so new models need no bespoke parser. NVFP4 KV cache now covers SM8x/SM90/SM12x and ROCm adds a 4-bit KV cache. Model Runner V2 gains a GPU n-gram speculator and Elastic EP and becomes the ROCm default.

Benchmarks & Measured Results

All figures come from the PRs (vendor figures): FP8 KV on SM100 halves GLM-5.x KV memory per rank with 2.3–5.5% more throughput and 7–11% higher TTFT; fp8_ds_mla on SM90 gives GLM-5.3-Flash up to 31% more output throughput; multiple API workers deliver 43–85% more req/s on short requests. No head-to-head comparison with other engines was published.

Developer Adoption Guide

Upgrade with pip install vllm or docker pull vllm/vllm-openai:v0.32.0. Before upgrading, note that Python 3.10 is dropped, default CORS no longer allows every origin, and Responses API reasoning stream events were renamed. Use vllm download-kernels and vllm preload --preload-autotune to cut restart time.