AI Coding Live News

The official Kimi account @Kimi_Moonshot announced the Kimi K3.1 Pro model weights at 16:00 UTC on November 4; the moonshotai/Kimi-K3.1-Pro repository is public and not gated. The model card describes a natively multimodal MoE with 2.8T total and 104B active parameters and a 1M-token context, shipped as MXFP4 weights from quantization-aware training, about 1.56TB in total; vLLM, SGLang and TokenSpeed are the recommended engines. The license allows free use and modification, but Model-as-a-Service businesses with over $20M in annual revenue need a separate agreement.

Zoom

On 2026-10-23 (15:41 UTC) the official @NVIDIAAI account announced Nemotron 3.5 Bolt E4B 4GB, a small open model for local assistants, interactive characters and robots that can hold a conversation and use tools. The official Hugging Face model card says it is post-trained from Gemma-4-E4B-it, has 8B total parameters (4B effective), ships in Q4_K_M quantization, supports up to 64K input and 64K output tokens, and has a switchable thinking mode, plus a DFlash speculative-decoding drafter and a BF16 version. Weights, data and recipes are open under the OpenMDW-1.1 license, with commercial use allowed. On the model card the 4GB build scores 55.60 on MultiChallenge (Gemma4 E4B-it at the same quantization 39.36, Qwen3.5 4B 52.23), but trails Qwen3.5 4B on Tau2-Bench (52.32) and BFCL-v3 (67.31).

Zoom

The Ornith AI team officially released Ornith-1.5-122B under the permissive MIT license, a 122-billion parameter mixture-of-experts (MoE) reasoning model optimized to execute on a single GPU node (such as a single DGX). The model is immediately available across vLLM (0.19.1+), SGLang (0.5.9+), Hugging Face, and Ollama (ornith-1.5:122b) with native FP8, GGUF, and MLX quantization. Across independent agentic evaluations in the OpenHands harness, Ornith-1.5-122B registered 84.5 on SWE-bench Verified, 76.9 on SWE-bench Pro (surpassing Qwen3.8-Flash-Next's 43.9 by 33 points), 81.2 on TerminalBench 2.1, and 74.6 on MCP-Atlas.

Zoom

In small language models, vocabulary embedding tables often account for over 60% of total parameters, posing severe bottlenecks for edge deployment and ternary quantization. A new study (arXiv:2610.22428) replaces trainable embeddings with a single frozen 16 MB table of 512-bit binary codes derived via PCA and sign binarization. On Gemma 3 270M, trainable parameters collapse from 268.1M to 101.0M (-62.3%) while matching or exceeding original performance across HellaSwag (30.4%), LAMBADA (23.2%), and PIQA (61.5%).

Zoom

On 2026-10-12 OpenBMB released MiniCPM-V 4.7, an Apache-2.0 open multimodal family with a 35B-A3B variant (the series' first MoE) and a 1B variant. Its Lookahead Expert Scheduling (LES) prefetches experts across layers and, combined with layer-wise mixed-precision quantization and DSpark speculative sampling, lifts SSD-offloaded decoding on an iPad Pro M4 (16GB) from 2.97 to 17.39 tokens/s (5.86x), per OpenBMB. Weights are on Hugging Face, ModelScope and Ollama, with vLLM, SGLang, llama.cpp and iOS/Android/HarmonyOS support.

Zoom

Large language model serving exhibits bifurcated hardware constraints: prefill is compute-bound, benefiting from high-throughput low-precision tensor operations (e.g., NVFP4), while decode is memory-bandwidth-bound, requiring ultra-compact weight storage (e.g., 2-3 bit weight-only) to minimize token-generation memory traffic. Uniform quantization forces suboptimal compromises across both phases. Researchers from IST Austria introduce Disaggregated Quantization (DQ, arXiv:2609.26333), decoupling computation formats, quantization precision, and physical memory placement across prefill and decode. Paired with Offloaded Disaggregated Prefill (ODP) to stream NVFP4 prefill weights asynchronously from NVMe SSDs, DQ delivers a 1.78x time-to-first-token (TTFT) speedup on Qwen3.8-27B at 8K prompt length within llama.cpp, while supporting disaggregated serving in vLLM on architectures up to 2.8T parameters.

Zoom

Deploying 70B+ LLMs under tight memory constraints necessitates extreme 4-bit weight, activation, and KV-cache quantization (W4A4KV4). However, current methods predominantly focus on suppressing activation outliers, overlooking their geometric alignment with the quantizer grid. Researchers introduce PrismQuant, a quantizer-aware rotation framework that mathematically aligns the leading activation eigenspace with the constant group subspace of asymmetric grouped INT4 quantization. Formulated as a Ky Fan trace maximization problem, PrismQuant derives a closed-form, provably optimal rotation solution without requiring expensive gradient descent. Benchmarked across Llama, Qwen, and Mistral architectures up to 70B parameters and 30B MoE variants, PrismQuant sets new SOTA marks: on Llama-3.1-70B under W4A4KV4, it achieves 72.46% zero-shot accuracy—merely 0.22 percentage points below full FP16 precision. Deployment benchmarks demonstrate a 1.51x prefill speedup and 56.34% reduction in peak decoding VRAM with code open-sourced on GitHub.

Zoom

Addressing the fundamental tension where prefill is compute-bound while decode is memory-bandwidth-bound, researchers from ISTA and Neural Magic introduced Disaggregated Quantization (DQ). By assigning compute-native NVFP4 weights to prefill and compact 2-3 bit weights to decode, paired with Offloaded Disaggregated Prefill (ODP) that streams prefill weights from SSD, DQ cuts Time-To-First-Token (TTFT) by 1.78x at 8K context and lifts 1-bit decode accuracy by 32.5 points on MMLU-Pro.

Zoom

Unsloth AI released version v0.1.815-beta alongside official prebuilt Linux x86_64 CUDA 13 wheels for Flash-Attention 2.8.4, Causal-Conv1D 1.7.0, and Mamba-SSM 2.3.2.post1 targeting PyTorch 2.13/2.14 on Python 3.13. The release enables fast local deployment of Alibaba's Qwen-Image-2.1 across both consumer and datacenter GPUs using Fast FP8 and GGUF quantization (defaulting to Q4_K_M) for combined image generation and precise image editing. Crucially, the Unsloth workspace introduces custom Agent Skills compatible with Claude Code and .agents formats, featuring @skill tagging in chat and 2x faster reasoning blocks (boosting UI rendering from 30 FPS to 60 FPS).

Zoom