TokenRouter: Tsinghua Unveils NeurIPS 2026 Serving System for Token-Level LLM Routing, Boosting Throughput by up to 64x
Token-level LLM routing—routing easy tokens to small models and switching to frontier models for critical reasoning tokens—presents an optimal trade-off between output quality and inference cost. However, current LLM serving engines (such as vLLM and TGI) are architected for single-model execution, suffering severe step desynchronization, batch admission latency, and extreme orchestration complexity when handling high-frequency token transfers across heterogeneous models. Researchers from Tsinghua University's NICS Lab present TokenRouter (arXiv:2610.12242, accepted at NeurIPS 2026, code: github.com/thu-nics/TokenRouter), an efficient and developer-friendly serving system engineered for token-level routed LLM inference. TokenRouter introduces a 'request-centric programming and model-centric execution' paradigm, enabling developers to write routing logic intuitively per request while the runtime handles asynchronous execution across subservers. Incorporating a Delayed-Batching Scheduler parameterized via discrete-time Markov chain modeling, a decoupled tri-loop architecture, and an instant handoff-resume mechanism, TokenRouter achieves 2.01x to 64.15x higher decoding throughput compared to existing serving platforms across diverse routing workloads, with complete code open-sourced.
Key Takeaways
- Tsinghua University open-sources TokenRouter (NeurIPS 2026), an optimized serving architecture for token-level LLM routing
- Introduces request-centric programming and a delayed-batching scheduler, delivering 2.01x to 64.15x higher decoding throughput
- Decoupled tri-loop execution cuts P99 inter-token latency by 43.6%, reducing enterprise cloud GPU costs by 50% to 70%
