AITOPAI热报
IndustryMulti-sourceSelected

SGLang 成立联盟，NVIDIA、AMD、Google、Meta 等 8 家公司加入

Official accountLMSYS Org (SGLang)@lmsysorg
Post· 11 sources
View the post
Why we picked it

SGLang 推理框架拉了 NVIDIA、AMD、Google、Meta 等 8 家组联盟一起维护，以后新模型、新芯片的支持会更快跟进。自己搭推理服务的工程师和做私有化部署的团队应该留意它的路线图更新。

SGLang 在 SGLang Summit 上宣布成立 SGLang Consortium（联盟），NVIDIA、AMD、Google、Meta、Intel、Cloudflare、Modal 和 RadixArk 加入，与项目维护者共同制定路线图。

Body

SGLang 目前运行在数十万块 GPU 上，支撑全球多个大型生产级推理部署。联盟将把生产需求转化为共享工程任务，方向包括支持前沿模型、优化推理与多模态负载、扩展分布式推理、跨硬件平台优化。

开源推理框架由多家芯片与云厂商共同维护，意味着新硬件（如 AMD、Intel 加速器）的适配会更快进入主线。对依赖 SGLang 做部署的团队来说，路线图决策的透明度和硬件覆盖面都会提升。

Original · LMSYS Org (SGLang)

Introducing the SGLang Consortium at SGLang Summit.

@nvidia , @AMD, @Google, @Meta, @intel, @Cloudflare, @modal, and @radixark are joining SGLang’s maintainers to help shape the project’s roadmap and advance open-source AI inference.

@sgl_project runs on hundreds of thousands of GPUs and powers some of the largest production inference deployments in the world.

The consortium will turn production needs into shared engineering work: supporting frontier models, improving reasoning and multimodal workloads, scaling distributed inference, and optimizing across hardware platforms.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Industry" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SGLang, NVIDIA, 开源模型, 推理框架.

Source attribution: curated from LMSYS Org (SGLang) (@lmsysorg). Cross-source corroboration: 11 independent sources have covered this development (including Sebastian Raschka, vLLM, PyTorch), confirming industry-wide signal strength.

#SGLang#NVIDIA#开源模型#推理框架

Keep reading

Recent stories with the same tag or section
ProductsBuildersSelected00:51
AI Agents Help Integrate SGLang with IBM Spyre in a Week

The notable point is the integration velocity: a 1,500-line plugin plus AI coding agents replaced what is typically a multi-month porting effort. Teams evaluating non-GPU accelerators for LLM inference can now consider IBM Spyre as a drop-in option with vLLM-parity performance. Engineering leaders building serving infrastructure should watch how plugin-based hardware abstraction in SGLang lowers the cost of supporting new silicon.

#SGLang#IBM Spyre#vLLM#PyTorch
Official accountPyTorch@PyTorchSource
How-toBuildersSelected01:21
SGLang Megakernels Lift Muse Spark Open Decoding to 1,109 Tokens/s

Kernel-launch overhead, not compute, is often the real ceiling in multi-GPU inference, and this write-up shows concrete numbers and code proving it. Engineers building or tuning serving stacks on Blackwell-class hardware get CUDA/PTX examples and measured timelines they can adapt. Teams benchmarking against vLLM or TensorRT-LLM deployments will find the launch-count reduction a useful reference point.

#SGLang#Muse Spark Open#GB300#inference optimization
Official accountLMSYS Org (SGLang)@lmsysorgSource
PapersBuildersSelected10:41
SlackTier Moves DeepSeek-V4.1 Engram Tables Off Host DRAM

The non-obvious insight is that per-layer prefetch slack, not raw bandwidth, decides whether lookup tables fit on NVMe, DRAM, or remote memory — static hot sets transfer poorly for agent traffic. Inference engineers running DeepSeek-V4.1 or similar n-gram-augmented models on SGLang can reclaim roughly 180 GiB of host memory and convert it into KV cache, directly lifting throughput. Compared with simply scaling up DRAM, the RDMA option lets one table copy serve two instances, cutting per-node memory cost.

#DeepSeek#SGLang#Engram#KV cache
arXiv cs.LG@Wen Huang, Jack LiuSource
PapersBuilders10:41
Chinese content论文揭示前缀缓存的智能体服务存在亚稳态：一次突发可致吞吐腰斩70分钟

跑过 vLLM 智能体服务的人应该都见过那种莫名的性能崩塌，这篇论文把它解释清楚了：一次几小时的突发流量就能让命中率掉到 10%，吞吐腰斩还自己好不了。论文给了能用的预算器方案，不用调参，直接把 KV 池撑爆的可能性掐掉。

#vLLM#SGLang#prefix caching#KV缓存
arXiv cs.LG@Wen Huang, Jack LiuSource
ModelsBuildersSelected01:02
Muse Spark Open Hits 433 tok/s Per User on TPU v7 Megakernel

The notable part is that a 816B MoE model now runs end-to-end as one Pallas program on a single TPU host through SGLang and TorchTPU, not through a closed inference stack. Teams building long-context agent products should study the deep dive for fp8 KV and session-aware cache strategies they can replicate. Compared with CUDA-only megakernel efforts, this demonstrates the PyTorch-to-TPU path has matured enough for production serving.

#Muse Spark Open#PyTorch#SGLang#TPU
Official accountPyTorch@PyTorchSource