AITOPAI热报
ProductsSelected

vLLM-TorchTPU brings native PyTorch serving to Google TPUs

官方账号vLLM@vllm_project
Post
View the post
Why we picked it

The practical effect is that vLLM's optimizations — prefix caching and speculative decoding — now apply to TPU hardware, giving teams a second accelerator option beyond NVIDIA GPUs for self-hosted inference. Organisations already locked into the PyTorch and vLLM stack can evaluate TPU economics without rewriting their serving code. It also signals Google's continued push to open TPU inference beyond JAX.

vLLM announced vLLM-TorchTPU, built with Google and Inferact, which enables serving models on TPUs using native PyTorch tooling. The release includes prefix caching, speculative decoding, and disaggregated serving. It lets teams run inference on TPUs through the same vLLM workflow they already use on GPUs.

Original · vLLM

vLLM 🤝 TPU

🚀 Native PyTorch serving on TPUs is here! Built with @Google and @inferact, vLLM-TorchTPU brings familiar PyTorch tools to TPU inference, with prefix caching, speculative decoding, and disaggregated serving.

🛠️ Excited to keep building with the community!

#vLLM#TPU#PyTorch#Google#inference

Keep reading

Recent stories with the same tag or section
ProductsBuildersMulti-sourceSelected04:07
PyTorchCon Demo Theater: 13 Sessions on Training, Inference and Agents

The lineup reveals where the PyTorch ecosystem is concentrating effort: inference speed for video models, MoE expert parallelism on NVL72, and fault-tolerant training that survives link and node failures live. Engineers choosing accelerator backends can use these demos to compare AWS Trainium, Intel AI PCs, TPUs and Rebellions hardware on real workloads. The Huawei CRCR CI relay is also a practical pattern for teams maintaining forks against upstream PyTorch.

#PyTorch#PyTorchCon#Wan 2.2#vLLM
Event
官方账号PyTorch@PyTorch11 sourcesSource
ProductsBuildersSelected02:01
FlyDSL kernel DSL now supports PyTorch on AMD MI350 GPUs

AMD MI350 support matters because GPU kernel tooling has historically centred on NVIDIA's stack, and this gives teams serving open models a Python-native path to tune kernels without CUDA-only tools. Benchmark and deployment engineers running vLLM on AMD accelerators can adopt the DSL directly from PyTorch's ecosystem and measure the 3.01x serving gain against their own baselines.

#FlyDSL#PyTorch#AMD#vLLM
官方账号PyTorch@PyTorchSource
IndustryBuildersSelected01:07
Cohere's Rachad Alao on open science for foundation models

The talk links open model releases to concrete security practices like independent auditing and dynamic safeguards, rather than treating openness and safety as opposing goals. Researchers and ML engineers can follow the named stack — PyTorch, TorchTitan and vLLM — to reproduce or audit open models themselves. Cohere's stance is notable as a US-listed enterprise lab that publicly backs open science. origin

#Cohere#PyTorch#TorchTitan#vLLM
官方账号PyTorch@PyTorchSource
How-toBuildersMulti-sourceSelected01:01
Anyscale session covers Ray Serve LLM deployment and inference scaling

The session bundles inference infrastructure decisions most teams learn the hard way: when disaggregated prefill/decode and KV-cache-aware routing pay off, and whether self-hosting beats token-based API pricing. Engineers deploying open-source models with vLLM or SGLang can register to get a hands-on walkthrough of Ray Serve LLM and agentic-platform integration. Registration is open now via the Anyscale link.

#Ray Serve#vLLM#SGLang#inference
Event
Anyscale@anyscalecompute10 sourcesSource
ModelsBuildersMulti-sourceSelected00:47
vLLM adds Day-0 support for Recursive's Flint model

A sparse 2.5B-active architecture hitting 500 TPS on one GB200 shows how far inference efficiency now goes without speculative decoding, a benchmark point worth tracking against dense models of similar quality. Teams serving long-context workloads can run Flint in vLLM immediately in either NVFP4 or BF16. Builders evaluating open-weight model serving stacks should compare its throughput on their own GPUs.

#vLLM#Flint#NVFP4#open-source models
Event
官方账号vLLM@vllm_project8 sourcesSource
IndustryBuildersSelected00:43
IBM Spyre joins PyTorch compiler and vLLM, Red Hat builds CI/CD

Hardware vendors gaining first-class PyTorch and vLLM support means open-source inference stacks are extending beyond NVIDIA GPUs to dedicated AI accelerators. Teams building multi-backend deployments should watch llm-d and the out-of-tree accelerator CI/CD work, which lower the cost of adding new hardware backends. Spyre's path contrasts with closed proprietary stacks like NVIDIA's CUDA ecosystem.

#IBM#Spyre#Red Hat#PyTorch
Event
官方账号PyTorch@PyTorch2 sourcesSource