Alibaba Taobao MNN Team Open-Sources 300M-Parameter On-Device Model MNN-Pocket-0.3B: 173 MB INT4 Build, Up to 466 tok/s Decode on iPad Pro M5, Apache 2.0

Background & Industry Pain Points

Putting a language model inside a phone app is not only about parameter count: install size, memory, power and first-token latency all matter. Many ~0.5B open models run on phones but weigh hundreds of megabytes and fall short of real-time chat speed. On November 4, 2026, Alibaba Taobao's MNN inference-engine team announced MNN-Pocket-0.3B in the News section of GitHub alibaba/MNN and on Hugging Face (taobao-mnn), positioning it as a mobile-app-native small model under Apache 2.0. From November 8 the third-party iOS app Local LLM said it had added the model, and a community developer demoed a LiteRT conversion on a Samsung phone NPU, so discussion is still growing.

Architecture Highlights & Mechanisms

The official model card lists about 300M parameters, 28 layers, hidden size 896, GQA with 14 attention heads and 2 KV heads (head_dim 64), FFN size 2560, a 65,536-token vocabulary and tied embeddings; positions go to 40,960 with 32K supported in deployment. The team says the architecture, training and inference engine were designed together around phone-app tasks and resource budgets. Quantization-aware training produces an INT4 (W4A16, block 64) MNN build of only 173 MB. The family also includes Base, SFT, GGUF for llama.cpp, and a Dec decision model, a LoRA adapter plus pointer head that answers multiple-choice style questions in one forward pass.

Authoritative Benchmarks & Measured Scores

All figures are the vendor's. The 11-benchmark zh/en average is 50.00, and 49.59 for INT4, a 0.41-point loss. Baselines: Qwen3-0.6B 44.28, MiniCPM4-0.5B 46.81, ERNIE-4.5-0.3B 31.99, Gemma-3-270M 6.42. Strong spots are IFEval 73.20, IFEval-zh 74.59, GSM8K 54.36 and HumanEval 53.66, but C-Eval 38.86 and CMMLU 40.11 trail MiniCPM4-0.5B (67.98, 66.04), and MATH500 35.40 trails Qwen3-0.6B at 45.20. On-device speed at 512-token context with INT4: iPad Pro M5 Metal does 8,721 tok/s prefill and 466 tok/s decode; Snapdragon 8 Gen 5 decodes 138 tok/s on OpenCL and 251 tok/s on CPU. No independent third-party reruns yet.

Developer Hands-on Guide

To try it on a server or laptop, load taobao-mnn/MNN-Pocket-0.3B with Transformers; the chat template ships in the repo, and llama.cpp users can grab the GGUF build. For phones, download the official MNN-Pocket-0.3B-MNN package or export your own with llmexport from the MNN repo, then build MNN's llm_demo to run it. The team recommends on-device assistants, data extraction, intent detection and lightweight agent pipelines, and does not recommend advanced math or long code generation, so test instruction following and extraction accuracy on your own samples before adopting it.