AITOPAI热报
ProductsSelected

PyTorch keynote details torch.accelerator, FlexShard and torch_remat

官方账号PyTorch@PyTorch
Post
View the post
Why we picked it

FlexShard's split between storage sharding and compute sharding points to a shift in how PyTorch abstracts distributed memory, potentially simplifying multi-device setups. Engineers running large training jobs or porting to new accelerators can watch the keynote for concrete tooling they can adopt now.

Meta's Alban Desmaison and Edward Yang presented a PyTorch keynote at PyTorchCon 2026 on simplifying hardware integration and large-scale training. They demonstrated torch.accelerator and backend testing tools designed to catch issues before costly training runs. The talk also introduced torch_remat for activation checkpointing and FlexShard for separating storage sharding from compute sharding. A keynote livestream recording is available online.

Original · PyTorch

Alban Desmaison @albanDesmaison and Edward Yang @ezyang (@Meta) discuss simpler hardware integration and more reliable large-scale training with PyTorch at #PyTorchCon 2026.

They cover torch.accelerator, backend testing, and tools to catch problems before expensive training runs, alongside torch_remat for activation checkpointing and FlexShard for separating storage sharding from compute sharding.

Watch the keynote livestream: https://t.co/6TtaEkvsHz

#PyTorch#FlexShard#torch_remat#distributed training

Keep reading

Recent stories with the same tag or section
ProductsBuildersMulti-sourceSelected04:07
PyTorchCon Demo Theater: 13 Sessions on Training, Inference and Agents

The lineup reveals where the PyTorch ecosystem is concentrating effort: inference speed for video models, MoE expert parallelism on NVL72, and fault-tolerant training that survives link and node failures live. Engineers choosing accelerator backends can use these demos to compare AWS Trainium, Intel AI PCs, TPUs and Rebellions hardware on real workloads. The Huawei CRCR CI relay is also a practical pattern for teams maintaining forks against upstream PyTorch.

#PyTorch#PyTorchCon#Wan 2.2#vLLM
Event
官方账号PyTorch@PyTorch11 sourcesSource
How-toBuildersSelected03:43
PyTorch Examines Linear Algebra Kernels Behind Faster Optimizers

The speed of an optimizer depends as much on kernel-level linear algebra as on step count, a factor most training guides ignore. Engineers tuning large-scale training can apply the QR, Cholesky and eigendecomposition implementation patterns to cut data movement on GPUs. The talk ties directly to GPU MODE's competition, giving concrete reference implementations to study.

#PyTorch#optimizers#GPU#linear algebra
官方账号PyTorch@PyTorchSource
ProductsBuildersSelected02:54
vLLM-TorchTPU brings native PyTorch serving to Google TPUs

The practical effect is that vLLM's optimizations — prefix caching and speculative decoding — now apply to TPU hardware, giving teams a second accelerator option beyond NVIDIA GPUs for self-hosted inference. Organisations already locked into the PyTorch and vLLM stack can evaluate TPU economics without rewriting their serving code. It also signals Google's continued push to open TPU inference beyond JAX.

#vLLM#TPU#PyTorch#Google
官方账号vLLM@vllm_projectSource
ProductsBuildersSelected02:01
FlyDSL kernel DSL now supports PyTorch on AMD MI350 GPUs

AMD MI350 support matters because GPU kernel tooling has historically centred on NVIDIA's stack, and this gives teams serving open models a Python-native path to tune kernels without CUDA-only tools. Benchmark and deployment engineers running vLLM on AMD accelerators can adopt the DSL directly from PyTorch's ecosystem and measure the 3.01x serving gain against their own baselines.

#FlyDSL#PyTorch#AMD#vLLM
官方账号PyTorch@PyTorchSource