AITOPAI热报
ProductsSelected

SGLang Summit 首日：SGLang v1.0 与 Miles v0.2 发布

Official accountLMSYS Org (SGLang)@lmsysorg
Post
View the post
Why we picked it

LMSYS 在自家 Summit 上连发两个工具：SGLang 升到 v1.0，推理更快更省内存；Miles v0.2 把 RL 训练铺到更多硬件上。做模型部署或后训练的工程师可以关注，和 vLLM、TRL 对比着看更清楚。

SGLang Summit 第一天，LMSYS 团队在主舞台公布两项更新。Ying Sheng 宣布 SGLang v1.0 正式发布，重点在推理速度、内存使用效率和面向生产环境的灵活扩展能力。

Body

Banghua Zhu 同步介绍了 Miles v0.2，将生产级强化学习（RL）训练支持扩展到更多硬件和更多工作负载场景。两者组合形成从推理服务到 RL 训练的完整开源基础设施栈。

SGLang 是目前主流的开源推理引擎之一，与 vLLM 在生产部署中直接竞争。v1.0 版本号意味着 API 和架构进入稳定期，对需要长期维护的团队部署决策有参考价值。

Original · LMSYS Org (SGLang)

Day 1 at #SGLangSummit going strong! Just now on the Main Stage: • Ying Sheng (@ying11231) announces SGLang v1.0: faster inference, better memory use, & flexible scaling for prod • Banghua Zhu (@BanghuaZ) introduces Miles v0.2: Production RL across more hardware and workloads https://t.co/9bxNh54a6V

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Products" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include SGLang, LMSYS, Miles, 推理引擎.

Source attribution: curated from LMSYS Org (SGLang) (@lmsysorg).

#SGLang#LMSYS#Miles#推理引擎#开源模型

Keep reading

Recent stories with the same tag or section
ProductsBuildersSelected00:51
AI Agents Help Integrate SGLang with IBM Spyre in a Week

The notable point is the integration velocity: a 1,500-line plugin plus AI coding agents replaced what is typically a multi-month porting effort. Teams evaluating non-GPU accelerators for LLM inference can now consider IBM Spyre as a drop-in option with vLLM-parity performance. Engineering leaders building serving infrastructure should watch how plugin-based hardware abstraction in SGLang lowers the cost of supporting new silicon.

#SGLang#IBM Spyre#vLLM#PyTorch
Official accountPyTorch@PyTorchSource
How-toBuildersSelected01:21
SGLang Megakernels Lift Muse Spark Open Decoding to 1,109 Tokens/s

Kernel-launch overhead, not compute, is often the real ceiling in multi-GPU inference, and this write-up shows concrete numbers and code proving it. Engineers building or tuning serving stacks on Blackwell-class hardware get CUDA/PTX examples and measured timelines they can adapt. Teams benchmarking against vLLM or TensorRT-LLM deployments will find the launch-count reduction a useful reference point.

#SGLang#Muse Spark Open#GB300#inference optimization
Official accountLMSYS Org (SGLang)@lmsysorgSource
PapersBuildersSelected10:41
SlackTier Moves DeepSeek-V4.1 Engram Tables Off Host DRAM

The non-obvious insight is that per-layer prefetch slack, not raw bandwidth, decides whether lookup tables fit on NVMe, DRAM, or remote memory — static hot sets transfer poorly for agent traffic. Inference engineers running DeepSeek-V4.1 or similar n-gram-augmented models on SGLang can reclaim roughly 180 GiB of host memory and convert it into KV cache, directly lifting throughput. Compared with simply scaling up DRAM, the RDMA option lets one table copy serve two instances, cutting per-node memory cost.

#DeepSeek#SGLang#Engram#KV cache
arXiv cs.LG@Wen Huang, Jack LiuSource
PapersBuilders10:41
Chinese content论文揭示前缀缓存的智能体服务存在亚稳态：一次突发可致吞吐腰斩70分钟

跑过 vLLM 智能体服务的人应该都见过那种莫名的性能崩塌，这篇论文把它解释清楚了：一次几小时的突发流量就能让命中率掉到 10%，吞吐腰斩还自己好不了。论文给了能用的预算器方案，不用调参，直接把 KV 池撑爆的可能性掐掉。

#vLLM#SGLang#prefix caching#KV缓存
arXiv cs.LG@Wen Huang, Jack LiuSource
ModelsBuildersSelected01:02
Muse Spark Open Hits 433 tok/s Per User on TPU v7 Megakernel

The notable part is that a 816B MoE model now runs end-to-end as one Pallas program on a single TPU host through SGLang and TorchTPU, not through a closed inference stack. Teams building long-context agent products should study the deep dive for fp8 KV and session-aware cache strategies they can replicate. Compared with CUDA-only megakernel efforts, this demonstrates the PyTorch-to-TPU path has matured enough for production serving.

#Muse Spark Open#PyTorch#SGLang#TPU
Official accountPyTorch@PyTorchSource
ModelsBuildersSelected03:49
Ant International Boosts GLM-5.3-Flash Throughput on SGLang

The non-obvious takeaway is that most of the decode speedup depends on how much prefix-sharing your traffic actually has, so the 4.32× figure only applies to workloads like batch document or code review with near-identical prompts. Inference and platform engineers running vLLM or SGLang can benchmark the same cache-reuse and speculative decoding stack on H200 fleets. Ant International is Singapore-headquartered, and the blog offers a reference for ASEAN teams building local GLM serving capacity instead of routing through overseas APIs.

#GLM#SGLang#H200#speculative decoding
Official accountLMSYS Org (SGLang)@lmsysorgSource