NVIDIA 提出自适应 Megakernel 方案,Nemotron 3.5 Transcribe 推理吞吐最高达 6367 倍实时
High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion
NVIDIA 给自家 Nemotron 3.5 Transcribe 换了个自适应推理引擎,每个 pass 自动在 megakernel 和融合 kernel 之间切换,B200 上跑到 6367 倍实时,比 vLLM 快 64%。做语音转写或推理优化的人都该看看这套 per-pass 切换的思路。
NVIDIA 发布面向 Nemotron 3.5 Transcribe(Hugging Face Open ASR 开源模型榜第一)的高吞吐推理引擎。在 H100 SXM GPU 上,精度无损的推理吞吐峰值达 4,345 倍实时,为调优后 vLLM 峰值的 1.39 倍;B200 上峰值达 6,367 倍实时,在最高并发下为 vLLM 的 1.64 倍,WER 保持在 1.91-1.93%,与 vLLM 的 1.88-1.93% 基本持平。
技术上,连续批处理会在每次迭代改变工作负载的规模与构成,单一融合粒度无法一直最优。该引擎针对每个 pass 在离线测得的交叉点处选择持久化 megakernel 或融合 kernel 链:128 并发时比始终用 megakernel 高 23% 吞吐,8 并发时比始终用融合 kernel 高 9%;在 400 ms p95 端到端时延预算内可服务的并发请求数是 vLLM 的 8 倍。B200 上还可开启贪心 MoE 层跳过,对离线任务额外提升 13-16% 吞吐,代价是 WER 略升。
对部署语音转写服务的企业与推理工程师,这套 per-pass 切换思路可直接迁移到其他生成式模型的 serving 优化。
High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion
Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR leaderboards. Serving it efficiently takes more than fast operators: continuous batching changes the amount and mix of work at every iteration, and the best fusion granularity changes with it. We present a serving engine that, for every pass, chooses between persistent megakernels and chains of fused kernels at crossovers measured offline. On an H100 SXM GPU, throughput of our accuracy-preserving serving peaks at 4,345x real time, 1.39x the peak of tuned vLLM, and is up to 4.39x faster at low concurrency; on B200 it peaks at 6,367x, 1.64x vLLM at the highest concurrency we measured. Word error rate (WER) is 1.91 to 1.93% against 1.88 to 1.93% for vLLM. Within a 400 ms budget for p95 end-to-end completion time, it serves 8x as many concurrent requests. Choosing per pass beats either level alone: 23% higher throughput than always using megakernels at 128 concurrent requests, and 9% higher than always using fused kernels at 8. Opt-in switches, including greedy MoE layer skipping, add 13 to 16% throughput on B200 for offline jobs that can accept higher WER.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Nemotron, vLLM, NVIDIA, speech recognition.
Source attribution: curated from arXiv cs.LG (@Aditya Chatterjee, Sanjay Singh Chauhan, Mayank Jain, Arushi Dhingra, Sahin Chowdhury, Rahul Mittal).