PyTorch Examines Linear Algebra Kernels Behind Faster Optimizers
The speed of an optimizer depends as much on kernel-level linear algebra as on step count, a factor most training guides ignore. Engineers tuning large-scale training can apply the QR, Cholesky and eigendecomposition implementation patterns to cut data movement on GPUs. The talk ties directly to GPU MODE's competition, giving concrete reference implementations to study.
At PyTorchCon 2026, Mark Saroufim (@marksaroufim) examined the linear algebra kernels behind better optimizers. He focused on QR, symmetric eigendecomposition and Cholesky, drawing lessons from GPU MODE's linear algebra competition. His talk explains how sequential dependencies constrain GPU execution. He shows how implementations expose parallel work, overlap independent stages and combine updates to reduce data movement. The core point: fewer training steps do not necessarily mean faster training.
@marksaroufim Fewer training steps don’t necessarily mean faster training. At #PyTorchCon 2026 (@coreauto) examines the linear algebra kernels behind better optimizers.
Mark focuses on QR, symmetric eigendecomposition, and Cholesky, drawing lessons from GPU MODE’s linear algebra competition.
He explains how sequential dependencies constrain GPU execution, and how implementations expose parallel work, overlap independent stages, and combine updates to reduce data movement.
