DeepSeek open-sourced an inference acceleration patch for consumer GPUs. Combining 4-bit quantization with dynamic expert caching and CPU offloading, developers can run 67B sparse MoE coding models on a single 24GB VRAM GPU (RTX 4090/3090) at speeds exceeding 45 tokens per second.

Key Takeaways

  • βœ“Runs 67B sparse MoE coding models locally on standard 24GB consumer GPUs.
  • βœ“Dynamic expert caching and 4-bit quantization sustain speeds over 45 tokens/sec.
  • βœ“Patches upstreamed directly into llama.cpp and SGLang repositories.
ADSponsored