AITOPAI热报
ProductsSelected

借助 AI 编程智能体，SGLang 一周内完成 IBM Spyre 适配

Official accountPyTorch@PyTorch
Post
View the post
Why we picked it

IBM Spyre 这种新芯片，以前接入推理框架可能要几个月，现在一个 1500 行插件加 AI 编程智能体一周搞定，吞吐还比参考实现高 3.6 倍，性能追平 vLLM。做推理部署、要挑国产或非 GPU 硬件的团队，这篇技术拆解值得仔细看。

PyTorch 官方宣布，SGLang 推理框架借助 AI 编程智能体，用不到一周时间和一个 1500 行的插件完成了对 IBM Spyre 加速芯片的集成。通过复用 PyTorch 已有的 Spyre 支持和 SGLang 现有扩展点，新实现的吞吐量达到参考版 torch-spyre 实现的约 3.6 倍。

Body

性能层面，在 Granite 3.3 8B 和 Gemma 4 26B A4B 两类工作负载上，SGLang 的表现与 vLLM 持平。RadixCache 前缀缓存机制无需任何硬件专属改动即可直接运行，说明整个适配过程主要靠软件层的模块化设计完成。

这一案例的价值在于验证了开源接口标准和可插拔架构的实际效率：新硬件进入主流推理栈的周期可以压缩到一周以内，对需要扩展硬件选型的企业部署团队有直接参考意义。

Original · PyTorch

With help from AI coding agents, SGLang was integrated with @IBM Spyre in under a week using a 1,500-line plugin. Reusing existing PyTorch Spyre support and SGLang extension points delivered roughly 3.6x the throughput of the reference torch-spyre implementation.

Performance matched vLLM across Granite 3.3 8B and Gemma 4 26B A4B workloads, with RadixCache prefix sharing working out of the box without hardware-specific changes. This shows how open source AI standards and modular architectures allow developers to bring new silicon online quickly and expand hardware choice across the stack.

Read the full technical breakdown: https://t.co/doMoB9xUSJ

@gal_bloch @RaghuGanti @mudhakar @alxnails

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Products" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SGLang, vLLM, IBM Spyre, PyTorch.

Source attribution: curated from PyTorch (@PyTorch).

#SGLang#vLLM#IBM Spyre#PyTorch#推理框架

Keep reading

Recent stories with the same tag or section
How-toBuildersSelected01:21
SGLang Megakernels Lift Muse Spark Open Decoding to 1,109 Tokens/s

Kernel-launch overhead, not compute, is often the real ceiling in multi-GPU inference, and this write-up shows concrete numbers and code proving it. Engineers building or tuning serving stacks on Blackwell-class hardware get CUDA/PTX examples and measured timelines they can adapt. Teams benchmarking against vLLM or TensorRT-LLM deployments will find the launch-count reduction a useful reference point.

#SGLang#Muse Spark Open#GB300#inference optimization
Official accountLMSYS Org (SGLang)@lmsysorgSource
PapersBuildersSelected10:41
SlackTier Moves DeepSeek-V4.1 Engram Tables Off Host DRAM

The non-obvious insight is that per-layer prefetch slack, not raw bandwidth, decides whether lookup tables fit on NVMe, DRAM, or remote memory — static hot sets transfer poorly for agent traffic. Inference engineers running DeepSeek-V4.1 or similar n-gram-augmented models on SGLang can reclaim roughly 180 GiB of host memory and convert it into KV cache, directly lifting throughput. Compared with simply scaling up DRAM, the RDMA option lets one table copy serve two instances, cutting per-node memory cost.

#DeepSeek#SGLang#Engram#KV cache
arXiv cs.LG@Wen Huang, Jack LiuSource
PapersBuilders10:41
Chinese content论文揭示前缀缓存的智能体服务存在亚稳态：一次突发可致吞吐腰斩70分钟

跑过 vLLM 智能体服务的人应该都见过那种莫名的性能崩塌，这篇论文把它解释清楚了：一次几小时的突发流量就能让命中率掉到 10%，吞吐腰斩还自己好不了。论文给了能用的预算器方案，不用调参，直接把 KV 池撑爆的可能性掐掉。

#vLLM#SGLang#prefix caching#KV缓存
arXiv cs.LG@Wen Huang, Jack LiuSource
ModelsBuildersSelected01:02
Muse Spark Open Hits 433 tok/s Per User on TPU v7 Megakernel

The notable part is that a 816B MoE model now runs end-to-end as one Pallas program on a single TPU host through SGLang and TorchTPU, not through a closed inference stack. Teams building long-context agent products should study the deep dive for fp8 KV and session-aware cache strategies they can replicate. Compared with CUDA-only megakernel efforts, this demonstrates the PyTorch-to-TPU path has matured enough for production serving.

#Muse Spark Open#PyTorch#SGLang#TPU
Official accountPyTorch@PyTorchSource
ModelsBuildersSelected03:49
Ant International Boosts GLM-5.3-Flash Throughput on SGLang

The non-obvious takeaway is that most of the decode speedup depends on how much prefix-sharing your traffic actually has, so the 4.32× figure only applies to workloads like batch document or code review with near-identical prompts. Inference and platform engineers running vLLM or SGLang can benchmark the same cache-reuse and speculative decoding stack on H200 fleets. Ant International is Singapore-headquartered, and the blog offers a reference for ASEAN teams building local GLM serving capacity instead of routing through overseas APIs.

#GLM#SGLang#H200#speculative decoding
Official accountLMSYS Org (SGLang)@lmsysorgSource