On November 4 Together AI launched Together Compiler (TKCC), which pairs a swarm of AI agents with a persistent inference engine to keep rewriting inference kernels for real traffic, models and hardware, as part of what it calls Polymorphic Inference. The company says that within 24 hours it reached up to 2.6x higher TPS on models such as Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash, up to 50% faster on customer workloads, and it already serves trillions of tokens a day in production.
Key Takeaways
- ✓Together Compiler recasts inference optimization from a classic compiler problem into an agentic search: agents propose kernel changes, a persistent engine checks speed and correctness, and only verified changes reach production
- ✓Together says that within 24 hours it reached up to 2.6x higher TPS on Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash, and up to 50% faster on customer workloads
- ✓On InferenceX low-latency batch size 1 it reports 1,111 TPS on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash (company-reported); SemiAnalysis said it validated one of the results
- ✓The persistent engine hot-reloads candidate forward passes, cutting time to serving to 9 seconds versus 467 seconds for restarting an engine, with 252x higher experiment throughput than baseline agentic optimization systems
- ✓It already serves trillions of tokens a day in Together's production stack, where 40+ models are now optimized by agents
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and the Industry Pain Point
Open models are shipping faster and with more varied architectures: Together cites OpenRouter data showing 80% of traffic goes to models released in the last three months. Traffic shapes and hardware (B200, GB300 and soon Vera Rubin) are changing quickly too, and hand-tuning each deployment takes researchers days or weeks. Together AI calls this situation Polymorphic Inference.
Architecture Highlights and Mechanism
Together Compiler (TKCC) takes the traffic shape, model and hardware as input and uses a swarm of agents to generate optimizations for the inference engine. The key piece is a persistent engine: it starts once and hot-reloads candidate forward passes without restarting. Fast engines load a few layers for quick iteration, while slower engines load many layers to check that an optimization holds on real data. Every change is checked for speed and correctness against the stock engine, and only verified changes go to production. When traffic drifts, the agents can run the optimization again, which Together calls continual compilation.
Benchmarks and Measured Results
According to the official blog, within 24 hours TKCC reached up to 2.6x higher TPS on Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash. On InferenceX low-latency settings it reports 1,111 TPS on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash, 1.7x faster than open-source megakernels on Llama 1B at batch size 1, and 9 seconds to serving versus 467 seconds. These are company-reported numbers; SemiAnalysis said on X that it validated one of the InferenceX/AgentX results.
Getting Started for Developers
TKCC is an internal capability of the Together platform, not a downloadable open-source tool. Developers using Together inference endpoints get the agent-tuned speed without code changes, and teams with dedicated deployments can look at Together's Polymorphic Inference page for optimization tuned to their own traffic.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.