DeepSeek Open-Sources DeepInfer Inference Engine: Powering V4.1-Flash at Scale Across NVIDIA SM100 and Huawei Ascend 950

Background and the Problem

As frontier mixture-of-experts (MoE) architectures with Multi-Head Latent Attention (MLA)—such as DeepSeek-V4.1-Flash—become pervasive across enterprise coding agents and high-throughput production clusters, standard inference runtimes encounter severe bottlenecks. Managing cross-node all-to-all communications during Expert Parallelism (EP), bridging heterogeneous hardware (such as domestic Ascend NPUs alongside frontier NVIDIA architectures), and sustaining deterministic low-latency speculative decoding frequently cause resource imbalances and output variance. Most AI frontier labs treat production serving engines as proprietary crown jewels. On October 20, 2026, DeepSeek officially open-sourced DeepInfer under Apache 2.0, alongside deepseek-recipe, fully realizing their commitment to "Open models, Open kernels, Open inference."

Architecture and How It Works

1. Heterogeneous Hardware Parity:

• Natively engineered from day one for both NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs;
• Delivers deterministic, batch-independent execution on NVIDIA hardware, guaranteeing reproducibility in automated code generation across variable batch sizes;

1. Deep Kernel and Communication Interleaving:

• Synthesizes core design patterns from vLLM, SGLang, and TensorRT-LLM, tightly coupled with FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and TileLang;
• Employs attention data parallelism, balanced context parallelism, and dynamic load-balanced MoE expert parallelism;

1. Speculative Decoding & SWA Bounded Replay:

• Embeds DSpark speculative decoding with confidence-scheduled verification, defaulting to 5 draft tokens;
• Features SWA Bounded Replay with a 140-token sliding window, substantially minimizing time-to-first-token (TTFT) and decode latency during long-context rollouts;

1. Native Constrained Structured Outputs:

• Out-of-the-box native compilation for regex, JSON schemas, and EBNF grammars directly in kernel pipelines for reliable agent tool execution.

Benchmarks and Measured Results

According to DeepSeek's technical disclosures and verified production cluster profiles:

• Cluster Deployment Architecture:
• Validated on production topologies pairing 4 SM100 GPUs for prefill execution with 64 GPUs orchestrating distributed decoding;
• DeepEP hardware-level communication overlapping reduces MoE all-to-all communication latency penalties by over 40%;
• Ecosystem Endorsement:
• The vLLM project team formally acknowledged DeepSeek's release (Status #2112480719952138301), praising the Apache 2.0 release and establishing technical cross-pollination initiatives between vLLM and DeepInfer.

Getting Started for Developers

• GitHub Repository: Accessible at https://github.com/deepseek-ai/DeepInfer;
• Tokenizer Pipeline: Developers should integrate deepseek-recipe for byte-exact tokenization matching DeepSeek's official production endpoints;
• Deployment Strategy: Infrastructure teams self-hosting DeepSeek-V4.1-Flash on NVIDIA SM100 or Ascend 950 clusters are urged to migrate to DeepInfer for optimal hardware saturation, deterministic decoding, and state-of-the-art throughput.