Muse Spark Open Hits 433 tok/s Per User on TPU v7 Megakernel
The notable part is that a 816B MoE model now runs end-to-end as one Pallas program on a single TPU host through SGLang and TorchTPU, not through a closed inference stack. Teams building long-context agent products should study the deep dive for fp8 KV and session-aware cache strategies they can replicate. Compared with CUDA-only megakernel efforts, this demonstrates the PyTorch-to-TPU path has matured enough for production serving.
A community megakernel for Muse Spark Open on Google TPU v7 reaches 433 tokens per second per user and up to 25.9K tokens per second per chip on the AgentX benchmark at 1M context. The result, highlighted by the PyTorch team, was built by radixark and runs on the SGLang serving stack via the TorchTPU PyTorch backend.
All 62 layers of the 816B-parameter Mixture-of-Experts model execute as a single Pallas program on one TPU v7 host. The path delivers 1.94x faster per-token speed than the previous implementation, and DFlash2 speculative decoding adds a further 2.0x per token. Long agent sessions are supported through fp8 KV cache compression, HiCache, and session-aware eviction.
The result shows large MoE models can now be served efficiently on TPUs using the mainstream PyTorch toolchain rather than proprietary stacks. Teams running million-token agent workloads gain a documented open-source recipe for high-throughput inference, and the integration of SGLang with TorchTPU lowers the barrier for labs already invested in PyTorch to target TPU hardware.
Great to see the open source community pushing inference performance forward on PyTorch:
A megakernel for Muse Spark Open on TPU v7 got 433 tok/s per user and up to 25.9K tok/s per chip on AgentX (1M context). All 62 layers of the 816B MoE run as one Pallas program on a single TPU v7 host, with SGLang on the TorchTPU PyTorch backend.
- 1.94x faster per token than our previous path - DFlash2 speculative decoding: 2.0x per token - fp8 KV, HiCache and session-aware eviction for long agent sessions
Read the full technical deep dive here: https://t.co/xmxOjlrZtK
@radixark
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Muse Spark Open, PyTorch, SGLang, TPU.
Source attribution: curated from PyTorch (@PyTorch).
