Alibaba Taobao MNN Team Open-Sources 300M-Parameter On-Device Model MNN-Pocket-0.3B: 173 MB INT4 Build, Up to 466 tok/s Decode on iPad Pro M5, Apache 2.0
On 2026-11-04 Alibaba Taobao's MNN inference-engine team released MNN-Pocket-0.3B on GitHub (alibaba/MNN) and Hugging Face (taobao-mnn): a roughly 300M-parameter Chinese-English instruction model co-designed with the MNN stack for smartphone apps, under Apache 2.0. The official model card says the INT4 (W4A16) QAT build is only 173 MB, supports 32K context, decodes at up to 466 tok/s on iPad Pro M5 (Metal) and about 350 tok/s in tests on iPhone 18 Pro, and averages 50.00 across 11 zh/en benchmarks, ahead of larger ~500M baselines such as Qwen3-0.6B at 44.28 (all vendor figures). The family also ships Base, SFT, GGUF, an on-device MNN package and a Dec decision model that answers in one forward pass. Since 11-08 the third-party iOS app Local LLM has added it and the community has published a LiteRT conversion. The team explicitly does not recommend it for advanced math or long code generation.
Key Takeaways
- Specs (official model card): about 300M parameters, 28 layers, hidden size 896, GQA with 14 heads and 2 KV heads, 65,536-token vocabulary, 32K context in deployment, Chinese and English
- Size and speed (vendor figures): INT4 (W4A16) QAT build is 173 MB; at 512-token context, iPad Pro M5 Metal does 8,721 tok/s prefill and 466 tok/s decode, Snapdragon 8 Gen 5 CPU decodes 251 tok/s; the GitHub README cites about 350 tok/s decode on iPhone 18 Pro
- Scores (vendor figures): 50.00 average over 11 zh/en benchmarks, 49.59 for INT4, a 0.41-point drop; IFEval 73.20, GSM8K 54.36, HumanEval 53.66, average above Qwen3-0.6B (44.28) and MiniCPM4-0.5B (46.81)
- Full family: Base, SFT, instruct, GGUF (llama.cpp), an on-device MNN package and a Dec decision model that answers in one forward pass; Apache 2.0 allows commercial use
- Positioning: the team targets on-device assistants, data extraction and lightweight agent pipelines, not advanced math or long code generation
