Release10d ago
OpenBMB open-sources MiniCPM-V 4.7: a 35B-A3B MoE multimodal model that runs on-device at 13+ tokens/s on iPhone 17 Pro Max
On 2026-10-12 OpenBMB released MiniCPM-V 4.7, an Apache-2.0 open multimodal family with a 35B-A3B variant (the series' first MoE) and a 1B variant. Its Lookahead Expert Scheduling (LES) prefetches experts across layers and, combined with layer-wise mixed-precision quantization and DSpark speculative sampling, lifts SSD-offloaded decoding on an iPad Pro M4 (16GB) from 2.97 to 17.39 tokens/s (5.86x), per OpenBMB. Weights are on Hugging Face, ModelScope and Ollama, with vLLM, SGLang, llama.cpp and iOS/Android/HarmonyOS support.
Tooling2d ago
vLLM and Ant Group open-source Humming, one quantized-inference kernel library for many formats and every NVIDIA GPU since Turing, lifting DeepSeek-V4.1-Flash end-to-end throughput by 14-43%
On Oct 21 the vLLM community and Ant Group open-sourced Humming (Apache 2.0), a quantized-inference library that JIT-compiles GEMM kernels on demand. It supports every NVIDIA architecture from Turing on plus T-Head's Zhenwu 810E, and covers weight and activation combinations from INT1-INT8 and FP4-FP8 for dense and MoE workloads. In vLLM's microbenchmarks it runs 2-5x faster than vLLM Marlin, FlashInfer or DeepGEMM in some cases; on 8 H20 GPUs, switching DeepSeek-V4.1-Flash MoE from Marlin W4A16 to Humming W4A8 raised throughput 14-43%. vLLM 0.29.0 ships it built in.
Tooling4d ago
vLLM v0.32.0: 814 commits, built-in /v1/systemone structured-decision endpoint, NVFP4 KV cache on more GPUs, and 43–85% more req/s with multiple API workers
vLLM shipped v0.32.0 on Oct 18 with 814 commits from 363 contributors (121 new). Highlights: a default-served POST /v1/systemone structured-decision endpoint and a generic hf tool/reasoning parser; NVFP4 KV cache on SM8x/SM90/SM12x and FP8 KV by default for GLM-5.x on SM100; Model Runner V2 gains a GPU n-gram speculator and Elastic EP and becomes the ROCm default; three security advisories are fixed, default CORS is restricted, and Python 3.10 is dropped.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.