AI Coding Live News

LM Studio posted on X at 15:32 UTC on Oct 29 that users can now download a new oMLX engine inside LM Studio and run it with MLX models they already have. The official changelog lists it in Bionic 1.1.10, alongside speculative decoding for MLX models and an in-app terminal. oMLX is an open-source Apple Silicon inference server maintained by developer jundot (Apache 2.0, about 23,000 GitHub stars) built around continuous batching and a two-tier RAM and SSD KV cache. LM Studio has not published speed comparisons for the engine.

Zoom

Hours after DeepSeek open-sourced DeepInfer, the vLLM project team released official benchmark parity data serving DeepSeek-V4.1-Flash. Utilizing DeepSeek's identical decoding setup (131K context window, nextn=3 speculative decoding), vLLM closed its throughput performance delta to within 6% of DeepInfer across batch sizes 8 through 256 on NVIDIA GB200 systems. On GB300 hardware, vLLM attained up to 1,000,000 tokens/second per GPU and 826 tokens/second per user, generating $38.96 in hourly GPU revenue at DeepSeek off-peak rates. vLLM simultaneously shipped day-0 runtime support for FastH3-Omni and Spark-X2.6-4B.

Zoom

DeepSeek officially open-sourced DeepInfer under Apache 2.0, its production-grade inference engine serving DeepSeek-V4.1-Flash at scale. Originating from vLLM with architectural refinements from SGLang and TensorRT-LLM, DeepInfer natively targets NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. It integrates specialized kernels including FlashMLA, DeepGEMM, DeepEP, and TileKernels. The engine features DSpark confidence-scheduled speculative decoding with 5-token drafts, SWA bounded replay over a 140-token sliding window, and balanced MoE expert parallelism. DeepSeek simultaneously open-sourced deepseek-recipe, delivering an end-to-end open ecosystem across models, kernels, and inference runtimes.

Zoom

On Oct 20 (UTC) DeepSeek open-sourced DeepInfer (Apache 2.0), the in-house inference stack that serves DeepSeek-V4.1-Flash at scale. The release covers offline inference with Attention DP/CP and MoE EP+EPLB parallelism, DSpark speculative decoding, SWA bounded replay, batch-invariant inference, deterministic sampling and structured outputs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The KV cache system, prefill/decode disaggregation and load balancing are not included. DeepSeek also released deepseek-recipe (MIT), which converts Chat Completions, Messages and Responses requests and encodes them the same way the official API does.

Zoom

Traditional speculative decoding relies on static target hidden features that fail to supply varying contextual needs across draft positions. HKU researchers introduced ReaDy (arXiv:2610.20503). Designed specifically for Hybrid Linear Attention (HLA) architectures combining Transformers with recurrent linear attention states, ReaDy leverages compact recurrent states as dynamic context for drafting. On a single NVIDIA B200 GPU serving Qwen3.8-27B, ReaDy achieves an acceptance length of 4.53 and 302.78 TPS, accelerating inference by 2.4x over standard autoregressive decoding.

Zoom

vLLM and NVIDIA unveiled a unified CuTe DSL decode megakernel (~12,000 LoC authored by NVIDIA CentML's Benjamin Chislett) for Moonshot AI's frontier Kimi K3 model on NVIDIA GB300 NVL72. By fusing attention, MoE, and cross-GPU communication, the optimization cuts per-layer latency from 130 µs (23 kernels) to 69 µs (13 kernels), delivering 842 tokens/s at batch=1 with DSpark speculative decoding (1.73x faster than stock vLLM) and up to 1.8x higher P90 per-user interactivity (403.2–423 tok/s/user) on SemiAnalysis AgentX, with DeepSeek-V4.1-Flash (PR #61868) and GLM-5.3 support coming next.

Zoom

A vLLM blog post by NVIDIA engineer Benjamin Chislett details an agent-assisted CuTe DSL decode path for Kimi K3 on NVIDIA GB300 that fuses attention, MoE and NVLink communication, cutting per-layer time from 130 µs across 23 kernels to 69 µs across 13. With DSpark speculative decoding on 16 GPUs it reaches 842 tok/s at batch 1, 1.73x stock vLLM; PR #61867 was merged on Oct 16.

Zoom

On 2026-10-15 NVIDIA released Nemotron 3.5 Super VL, an open multimodal agent model with 120B total / 12B active parameters: a Mamba-2–Transformer hybrid MoE with a C-RADIO vision encoder, text/image/document/video input, up to 1M context, and built-in MTP plus LiLiCorr speculative decoding. Weights, datasets and recipes ship under OpenMDW-1.1 on Hugging Face, OpenRouter, ModelScope and NIM. Alongside it, NVIDIA published Bolt 74B-A9B 24GB, a natively 2-bit (QAT) local agent model for RTX/DGX Spark; the 4GB Bolt E4B follows on Oct 23.

Zoom

Coding agents frequently reproduce existing code, error logs, and multi-turn refactoring traces, making retrieval-based speculative decoding (SD) an ideal acceleration paradigm. However, conventional retrieval drafters fail in agent workflows: candidate code on disk diverges from the agent's emission formats (e.g., diff blocks, escape characters), and static draft lengths ignore inter-agent turn drift. KAIST researchers introduce AgSpec, a speculative decoding framework tailored for coding agents. AgSpec constructs format-aligned corpora across session trajectories, workspaces, and global repositories, coupling offline draft caps with online feedback-adaptive draft length control. Across repository-level multi-agent benchmarks, AgSpec outperforms five retrieval drafters and EAGLE-3, boosting generation throughput up to 4.76x.

Zoom

Inferact open-sourced inferact/tpu-megakernels on 2026-09-23: a Pallas megakernel collection for TPU v7. Their blog reports Kimi K3 at 709 tok/s on 16× TPU v7 Ironwood vs 452 tok/s on 16× GB200 under DSpark speculative decoding (acceptance length 6), and roughly 1.4–2× the GB200 baseline at batch sizes 1–8 without speculation. All 92 MoE layers of Kimi K3 run in a single kernel with cross-layer weight prefetch.

Zoom