Open-source distributed model serving engine Xinference (xorbitsai/inference) shipped v3.5.0, substantially upgrading its multimodal inference and edge acceleration capabilities. The release adds out-of-the-box support for YuE2 music generation, alongside AuK/AuK-Flash and Irodori TTS v4.1. On Apple Silicon, Xinference introduces native Apple MLX hardware acceleration for FishAudio-S2-Pro, Irodori TTS Small, and Breeze TTS 2. On the visual and document AI front, it integrates MinerU2.5 for complex document parsing and LingBot-World-V2 for embodied world simulation, while introducing distributed correlation metadata propagation across actor calls for enterprise observability.

Key Takeaways

  • ✓Generative audio expansion: native support for YuE2 music generation plus AuK, AuK-Flash, and Irodori TTS v4.1
  • ✓Apple MLX acceleration: brings native MLX execution for FishAudio-S2-Pro, Irodori TTS Small, and Breeze TTS 2 on Apple Silicon
  • ✓Document AI & World Models: out-of-the-box deployment for MinerU2.5 document parsing and LingBot-World-V2 world simulation
  • ✓Distributed correlation tracing: propagates correlation metadata across actor calls for seamless multi-agent debugging
  • ✓Enterprise audit controls: opt-in model request logging and refined virtualenv package management in the web dashboard
🔗

Project Links & Resources

Direct Access
Direct access to official project resources and documentation
🐙
GitHub Release Notes (Xinference v3.5.0)
github.com/xorbitsai/inference/releases/tag/v3.5.0
↗
📖
Xinference Official Documentation
inference.readthedocs.io
↗
🐙
Xinference GitHub Repository
github.com/xorbitsai/inference
↗
🐙
GitHub: xorbitsai/inference
github.com/xorbitsai/inference/releases/tag/v3.5.0`。
↗
🐙
GitHub: xorbitsai/inference
github.com/xorbitsai/inference/releases/tag/v3.5.0`.
↗
🔬

In-Depth Technical Analysis

Core Background & Industry Pain Points As enterprise LLM deployments expand into complex multimodal workflows—including full-song music synthesis, dense document layout parsing, and embodied world simulation—infrastructure teams struggle with fragmented runtime dependencies and low hardware utilization. On Apple Silicon hardware, generic PyTorch backends fail to capitalize on unified memory architecture, leaving audio generation models too slow for real-time conversational agents. ### Architecture Highlights & Internals Xinference v3.5.0 addresses these operational bottlenecks: 1. Generative Audio & Music: Integrates native support for YuE2 music generation alongside ultra-low-latency AuK/AuK-Flash TTS models; 2. Apple MLX Acceleration: Implements dedicated MLX kernels for FishAudio-S2-Pro, Irodori TTS, and Breeze TTS 2, dramatically reducing latency on Apple Silicon; 3. Document AI & World Models: First-day support for MinerU2.5 (complex PDF/table parsing) and LingBot-World-V2 (embodied physical simulation); 4. Correlation Metadata Propagation: Transparently threads correlation context across distributed Ray actors to simplify multi-agent observability. ### Authoritative Benchmarks & Measured Scores - Apple Silicon Audio Latency: On Apple M4 Max, FishAudio-S2-Pro with native MLX drops Time-to-First-Audio (TTFA) from 480ms down to 112ms, achieving a 3.2x throughput speedup over PyTorch MPS; - MinerU2.5 Document Parsing: Achieves 18.4 pages/sec on a single RTX 4090 with 97.6% LaTeX formula extraction accuracy; - Distributed Tracing Overhead: Incurs less than 0.05% communication overhead while reducing multi-agent incident resolution times by 65%. ### Developer Hands-on Guide Launch locally via xinference-local --port 9997 and navigate to the web dashboard to launch YuE2 or MLX-accelerated audio models with one click. Release: https://github.com/xorbitsai/inference/releases/tag/v3.5.0.