Together Compiler Hits 2.6x Speedups on Frontier Model Inference
Together Compiler: Continual Compilation for AI Inference
An agent loop that continuously recompiles inference engines against live traffic could shift how serving providers keep pace with rapidly changing model architectures, replacing static hand-tuned kernels. Teams running inference infrastructure or building MoE serving stacks can study the tiered persistent engine design and context-management ablations for their own optimizer setups. The results also benchmark directly against open source engines on Chinese frontier models like DeepSeek v4.1 Flash and Kimi K3, a useful comparison point when evaluating serving providers.
Together AI has built TKCC (Together Compiler), a continual agent-driven compilation loop for AI inference optimization, detailed in a new arXiv paper (arXiv:2611.14681). The system outperforms hand-written Megakernels on the Llama 1B workload by up to 1.7x and beats the best open source engines on frontier models including Kimi K3, GLM 5.3, MiniMax M3.1 Flash and DeepSeek v4.1 Flash by up to 2.6x higher tokens per second on the InferenceX benchmark.
The core technical problem is a throughput-fidelity tradeoff in existing agent-based compilation. Kernel-level optimizers using offline benchmarks produce low-fidelity timings because of data-dependent behavior in primitives like mixture-of-experts, while end-to-end engine optimizers are slowed by long engine launch times. TKCC addresses this with a Tiered Persistent Engine that decouples engine launch overhead from forward pass computation, plus a cost model that selects parallelism configurations to seed the optimization loop. The team also reports ablations on agent context management and production deployment insights.
TKCC sets state-of-the-art batch-size-1 performance with 1111 TPS on M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash, generalizes beyond frontier language model inference, and already serves trillions of tokens across Together AI's production fleet.
Together Compiler: Continual Compilation for AI Inference
Today's AI workloads are rapidly changing: new models are released more often, model architectures are increasingly diverse, and traffic distributions change frequently. The challenge we face at Together AI is continual optimization for these changing workloads. Compiling a deployment into an optimized forward pass is a challenge that is well-suited for agent hill-climbing. Existing systems struggle with continual compilation due to a tradeoff between measurement throughput and fidelity. Many systems use agents to optimize individual kernels from the inference forward pass using offline benchmarks, but timings are low-fidelity because of data-dependent behavior in primitives like mixture-of-experts (MoE). On the other hand, systems use agents to directly optimize end-to-end engines but suffer from slow engine launch times. To address these challenges, we build Together Compiler (TKCC), the first truly continual agent loop designed for AI inference optimization. To address the throughput-fidelity tradeoff, we propose: (1) a Tiered Persistent Engine that decouples engine launch overheads from forward pass computation; (2) a cost model to select the parallelism configurations that seed the optimization process. We also conduct extensive ablations on agent context management and present our insights on deploying TKCC in production. TKCC outperforms highly-optimized hand-written Megakernels on the Llama 1B workhorse workload by up to 1.7x and the best open source engines on the latest frontier models (Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash) across hardware platforms by up to 2.6x higher TPS on the InferenceX benchmark. TKCC also sets the state-of-the-art performance at batch size 1, with 1111 TPS on M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash. Finally, TKCC generalizes beyond frontier language model inference. TKCC serves trillions of tokens across our production fleet.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Together AI, DeepSeek, Kimi, inference optimization.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority. Cross-source corroboration: 7 independent sources have covered this development (including Together AI, Lxfater, berryxia), confirming industry-wide signal strength.
