SGLang Low-Latency Mode Launches with NVIDIA and AMD Support
The same serving stack now optimises both NVIDIA and AMD paths, giving operators a concrete reason to benchmark MI355X deployments against GB300 rather than defaulting to CUDA. Teams running agentic inference traces can enable the mode through SGLang and expect P90 latency gains without changing models. The plugin system also means third-party kernels — starting with GLM-5.3, K3, and MiniMax M3 — will arrive without waiting for a core release.
SGLang introduced a Low-Latency Mode built jointly with NVIDIA and AMD. On DeepSeek-V4.1-Flash with DSpark, it delivers up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in SemiAnalysis's AgentX trace replay. On NVIDIA hardware, overlapping kernels cut decoder layer time from 96.8 to 68.4 µs; on AMD, two persistent kernels per layer replace dozens of small ones with cross-GPU communication overlapped inside. A new plugin system lets custom kernels plug in from external libraries with automatic fallback, and low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress.
Introducing SGLang Low-Latency Mode, built with @NVIDIAAI and @AMD.
On DeepSeek-V4.1-Flash with DSpark, it delivers up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s) and 1.56× on AMD MI355X (332 to 516 tok/s) in @SemiAnalysis_'s AgentX trace replay.
Highlights: - NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs - AMD: 2 persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside them. - New plugin system: custom kernels plug in from external libraries, with automatic fallback. - Low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress and will plug in next.
