LMSYS 用 megakernels 把解码速度从 517 提到 1,109 tokens/s
LMSYS 放出完整技术拆解：8 块 GB300 上把解码干到 1,109 tokens/s，每步 kernel 启动从 1,450 砍到 97。prefetch、指令缓存预热、跨卡通信这些招数都写了，还带 CUDA/PTX 代码。做推理优化的工程师直接照着学就行。
LMSYS 发布 Muse Spark Open 技术博客，在 8× GB300 上用 SGLang megakernels 将解码速度从 517 提升到 1,109 tokens/s。每个 decode step 的 kernel 启动次数从 1,450 降到 97，实现每层只启动一个 kernel。方法上，作者 Pranjal Shankhdhar 解释了三个关键机制：提前把权重 prefetch 进共享内存和 L2、预热指令缓存避免大 kernel 取码停顿、用数据本身作为跨 GPU 通信的就绪信号。配套优化还包括融合 draft model，使 drafting 速度快 2.6×。文章附带实测时间线、CUDA/PTX 代码示例，以及为什么停在每层一个 megakernel 而不做更大的取舍分析。
这类 megakernel 思路的本质是削减 kernel launch 开销，把计算、内存读写和通信重叠在同一层的大 kernel 里完成。对做推理优化的工程师来说，博客里的实测数据和方法拆解可以直接参考复现，也说明了在 GB300 这类新硬件上系统层调优仍有很大空间。
Muse Spark Open: 517 → 1,109 tokens/s on 8× GB300 with SGLang megakernels. One kernel launch per layer, overlapping compute, memory loads and communication. Total launches per decode step drop from 1,450 to 97.
https://t.co/YXb3pGDKH3
In our new technical post, Pranjal Shankhdhar explains what makes this work: - Prefetching weights into shared memory and L2 before they’re needed. - Warming the instruction cache so large kernels don’t stall fetching their own code. - Using data itself as a readiness signal for cross-GPU communication. - Fusing the draft model to make drafting 2.6× faster.
The post also includes measured timelines, CUDA/PTX examples, and the tradeoffs behind stopping at one megakernel per layer.
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include SGLang, GB300, megakernels, 推理优化.
Source attribution: curated from LMSYS Org (SGLang) (@lmsysorg).
