AITOPAI热报
ModelsSelected

Ant International Boosts GLM-5.3-Flash Throughput on SGLang

官方账号LMSYS Org (SGLang)@lmsysorg
Post
View the post
Why we picked it

The non-obvious takeaway is that most of the decode speedup depends on how much prefix-sharing your traffic actually has, so the 4.32× figure only applies to workloads like batch document or code review with near-identical prompts. Inference and platform engineers running vLLM or SGLang can benchmark the same cache-reuse and speculative decoding stack on H200 fleets. Ant International is Singapore-headquartered, and the blog offers a reference for ASEAN teams building local GLM serving capacity instead of routing through overseas APIs.

Ant International AI Infra, working with the SGLang community, published a blog reporting 2.11–4.28× higher end-to-end serving throughput for GLM-5.3-Flash on H200 GPUs. The headline results include 2.06× prefill throughput on 4×H200 at a 64K/10 token configuration with no shared prefix, and 4.32× decode throughput on 4×H200 at 64K/1K tokens with a 99.9% shared prefix.

Body

The gains come from three combined techniques: cache reuse that exploits shared prefixes across requests, speculative decoding to accelerate token generation, and kernel-level tuning on H200 hardware. The end-to-end benchmark ran on 16×H200 nodes at 64K/1K tokens across three different prefix-sharing ratios, showing how much each ratio affects the speedup.

Long-context serving at 64K tokens is a common production pain point, and prefix caching plus speculative decoding are becoming standard levers for inference cost reduction. Teams running open-source stacks like SGLang can replicate the approach rather than paying premium closed-endpoints rates.

Original · LMSYS Org (SGLang)

New blog: 2.11–4.28× Higher GLM-5.3-Flash Serving Throughput on H200

Ant International AI Infra brings faster long-context serving to SGLang. How much do cache reuse, speculative decoding, and kernel tuning add up to? Engineering Result: - 2.06× prefill throughput on 4×H200: 64K/10 tokens, no shared prefix. - 4.32× decode throughput on 4×H200: 64K/1K tokens, 99.9% shared prefix. - 2.11–4.28× end-to-end throughput on 16×H200: 64K/1K tokens across three prefix-sharing ratios.

Thanks to Ant International AI Infra and the SGLang community.

Read the full blog below

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Models" desk, focusing on Southeast Asia and the ASEAN-6 (Singapore, Malaysia, Indonesia, Thailand, Vietnam, Philippines) digital economy. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include GLM, SGLang, H200, speculative decoding.

Source attribution: curated from LMSYS Org (SGLang) (@lmsysorg).

#GLM#SGLang#H200#speculative decoding#inference optimization

Keep reading

Recent stories with the same tag or section
PapersBuildersMulti-sourceSelected05:24
Stanford AI Lab Introduces Joules-to-GDP Framework for AI Energy Efficiency

The framework shifts efficiency analysis from raw benchmark scores to value-per-joule, a metric that will matter as data-centre energy costs and grid constraints increasingly cap AI deployment. Teams choosing models or hardware can use it to justify smaller models like Gemma 4 31B over larger ones when value per joule outweighs per-task output. It pairs with GDPWorks for organisations benchmarking agentic work beyond the original 44 occupations.

#Gemma 4#GLM 5.2#NVIDIA H100#GDPval
Event
Stanford AI Lab@StanfordAILab11 sourcesSource
PapersBuildersMulti-sourceSelected10:49
GC-MoE Operator Accelerates MoE Inference with CUDA Green Contexts

The technique shows that partitioning a single GPU with CUDA Green Contexts removes the resource contention between dispatch warps and GEMM warps that limits fused MoE operators like Mega MoE. Practitioners serving DeepSeek-V4.1-Flash or GLM-5.2 on Hopper or Blackwell clusters can adopt the open-source operator to cut decode latency without extra hardware. The results on H20 GPUs, a China-market chip, make it especially relevant for deployments where H100-class hardware is unavailable.

#GC-MoE#DeepSeek#GLM#CUDA Green Contexts
Event
arXiv cs.AI@Yong Li et al. (3)11 sourcesSource