SGLang让GLM5.2在B300上单用户跑500 tok/s，交互延迟几乎不随上下文增长，对多轮Agent任务很实用。
SGLang通过优化在8块B300 GPU上实现bs=1时每用户500+ tok/s的吞吐量，单用户交互性相比基线提升18%至34%，高并发峰值吞吐提升6%至11%。新推出的TopK-V2内核在80K输入长度下加速2.33倍，在1M长度下加速10.17倍，使交互性在1M token范围内基本持平。GLM-5.2模型采用IndexShare架构于DSA层和更强的MTP头，结合SGLang的服务优化共同达成这一性能。
Serving GLM5.2 NVFP4 Agentic Workload with SGLang:…
Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1
In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens.
Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations.
Special thanks to @NVIDIAAI for the help in day-0 support of GLM-5.2 NVFP4, and to @Zai_org for IndexShare in SGLang!