ProductsSelected10月22日 01:23

SGLang-TorchTPU Brings PyTorch Model Serving to Google TPUs

官方账号LMSYS Org (SGLang)@lmsysorg

Post

Why we picked it

TPU support through PyTorch-native tooling lowers the barrier for teams that want to serve agents without rewriting stacks around XLA or JAX. Developers building agentic services can now evaluate TPU serving within the same SGLang setup they already run on GPUs. It is a practical alternative for cost-sensitive inference workloads where GPU supply is constrained.

LMSYS announced SGLang-TorchTPU, a new plugin that extends the SGLang inference framework to run models on TPUs. The project lets developers use familiar PyTorch tools and stacks to deploy models on TPU hardware and serve agentic workloads. The team published a write-up covering how the plugin works, its feature and model coverage, and early exploration of training on TPU.

SGLang-TorchTPU Brings PyTorch Model Serving to Google TPUs — Image · LMSYS Org (SGLang)

Original · LMSYS Org (SGLang)

Welcome SGLang-TorchTPU to the family!

We're excited to offer a new path for developers to use familiar tools and stacks to support models on TPU, and serve them for agentic workloads!

Check out our write-up on how SGLang-TorchTPU works as a plugin for SGLang, its feature and model coverage, and our early exploration of training on TPU!

https://t.co/1Cq9sk62cB

#SGLang #LMSYS #TPU #PyTorch #agents

Keep reading

Recent stories with the same tag or section

Models Builders03:25

Chinese contentMiniMax M3.1 Flash 搭配 SGLang Omni 打造可打断的语音智能体

MiniMax 把自家 M3.1 Flash 和 SGLang Omni 的流式、打断处理能力拼在一起，做能随时打断的语音智能体，做语音方向的朋友可以看看这套组合。

#MiniMax #SGLang #语音智能体 #M3.1 Flash

MiniMax_AI@MiniMax_AISource

How-to BuildersMulti-sourceSelected01:01

Anyscale session covers Ray Serve LLM deployment and inference scaling

The session bundles inference infrastructure decisions most teams learn the hard way: when disaggregated prefill/decode and KV-cache-aware routing pay off, and whether self-hosting beats token-based API pricing. Engineers deploying open-source models with vLLM or SGLang can register to get a hands-on walkthrough of Ray Serve LLM and agentic-platform integration. Registration is open now via the Anyscale link.

#Ray Serve #vLLM #SGLang #inference

Event

Anyscale@anyscalecompute10 sourcesSource

Products BuildersOfficial14:29

Chinese contentGoogle、Red Hat 与 IBM 发布 llm-d 1.0 开源分布式推理栈

做 LLM 推理部署的可以看看，llm-d 1.0 开源了，能管多租户、不停机升级，还支持 TPU 和 GPU 混用。

#llm-d #vLLM #SGLang #Kubernetes

Event

官方一手marktechpost@Asif Razzaq2 sourcesSource

Products Builders09:38

Chinese contentEAGLE-4 发布：无需改动模型，LLM 文字生成速度提升近 3 倍

不用改模型，装个 eagle_harness 配合 vLLM 就能让 Qwen 这类模型出字快近 3 倍，实测数据都给了，自己搭服务的可以试试。

#EAGLE-4 #Qwen #vLLM #SGLang

Gorden Sun@Gorden_SunSource

How-to Builders05:54

Chinese content推测解码开源工具让大模型生成文本更快

用小模型猜 token、大模型验证，一次能吐好几个 token，代码开源了，vLLM 和 SGLang 都能跑，推理党可以试试。

#vLLM #SGLang #推测解码 #推理优化

Ate-a-Pi@svpinoSource

Products BuildersMulti-sourceSelected08:42

SGLang Low-Latency Mode Launches with NVIDIA and AMD Support

The same serving stack now optimises both NVIDIA and AMD paths, giving operators a concrete reason to benchmark MI355X deployments against GB300 rather than defaulting to CUDA. Teams running agentic inference traces can enable the mode through SGLang and expect P90 latency gains without changing models. The plugin system also means third-party kernels — starting with GLM-5.3, K3, and MiniMax M3 — will arrive without waiting for a core release.

#SGLang #NVIDIA #AMD #DeepSeek-V4.1-Flash

Event

官方账号LMSYS Org (SGLang)@lmsysorg11 sourcesSource