AITOPAI热报
ProductsSelected

vLLM TorchTPU backend brings Google Cloud TPUs into vLLM ecosystem

Xianbao Qian@Xianbao_QIAN
Post
View the post
Why we picked it

Serving infrastructure built around NVIDIA GPUs now has a maintained TPU path, lowering the switching cost for teams that want Google's accelerator pricing and capacity. Contributors can apply for the free TPU pool to test and port models. Practitioners running vLLM in production should evaluate TPU serving costs against their current GPU fleet.

vLLM and Google released vLLM TorchTPU, a new vLLM backend for Google Cloud TPUs. Models that already run on vLLM can now be served on TPUs through the same ecosystem. The release includes open-source Pallas kernels, multi-host parallelism, KV cache offloading and disaggregated serving. A free TPU capacity pool is being opened for community contributors.

Original · Xianbao Qian

I’ve worked with Google’s internal AI infrastructure, including TPU, since the TensorFlow 1.0 era. It’s an excellent system, but it was hindered by layers of overlays and complexity that none of us could fully control.

My dream has always been to make TPU as accessible as GPUs. I’ve never been more excited than I am today, because we’ve finally found the path: opening up the knobs and plugging TPU into the community, end to end.

Today we’re releasing vLLM TorchTPU with Google, a new vLLM backend for Google Cloud TPUs. By leveraging vLLM ecosystem, more models that run in vLLM today will be served on TPU. It also brings open-source Pallas kernels, multi-host parallelism, KV cache offloading, and disaggregated serving. We’re even opening a free TPU capacity pool for community contributors.

This only happened because of the amazing Inferact and Google teams. They are some of the sharpest and most generous collaborators I’ve worked with. I’m honored to contribute, and I’m excited to see what the community builds. Let's cook

Blog: https://t.co/YR5M2x9Gvh

#vLLM#Google Cloud TPU#Pallas kernels#inference#open-source models

Keep reading

Recent stories with the same tag or section
ProductsBuildersMulti-sourceSelected04:07
PyTorchCon Demo Theater: 13 Sessions on Training, Inference and Agents

The lineup reveals where the PyTorch ecosystem is concentrating effort: inference speed for video models, MoE expert parallelism on NVL72, and fault-tolerant training that survives link and node failures live. Engineers choosing accelerator backends can use these demos to compare AWS Trainium, Intel AI PCs, TPUs and Rebellions hardware on real workloads. The Huawei CRCR CI relay is also a practical pattern for teams maintaining forks against upstream PyTorch.

#PyTorch#PyTorchCon#Wan 2.2#vLLM
Event
官方账号PyTorch@PyTorch11 sourcesSource
ProductsBuildersSelected02:54
vLLM-TorchTPU brings native PyTorch serving to Google TPUs

The practical effect is that vLLM's optimizations — prefix caching and speculative decoding — now apply to TPU hardware, giving teams a second accelerator option beyond NVIDIA GPUs for self-hosted inference. Organisations already locked into the PyTorch and vLLM stack can evaluate TPU economics without rewriting their serving code. It also signals Google's continued push to open TPU inference beyond JAX.

#vLLM#TPU#PyTorch#Google
官方账号vLLM@vllm_projectSource
ProductsBuildersSelected02:01
FlyDSL kernel DSL now supports PyTorch on AMD MI350 GPUs

AMD MI350 support matters because GPU kernel tooling has historically centred on NVIDIA's stack, and this gives teams serving open models a Python-native path to tune kernels without CUDA-only tools. Benchmark and deployment engineers running vLLM on AMD accelerators can adopt the DSL directly from PyTorch's ecosystem and measure the 3.01x serving gain against their own baselines.

#FlyDSL#PyTorch#AMD#vLLM
官方账号PyTorch@PyTorchSource
IndustryBuildersSelected01:07
Cohere's Rachad Alao on open science for foundation models

The talk links open model releases to concrete security practices like independent auditing and dynamic safeguards, rather than treating openness and safety as opposing goals. Researchers and ML engineers can follow the named stack — PyTorch, TorchTitan and vLLM — to reproduce or audit open models themselves. Cohere's stance is notable as a US-listed enterprise lab that publicly backs open science. origin

#Cohere#PyTorch#TorchTitan#vLLM
官方账号PyTorch@PyTorchSource