Systolic Linear Attention Optimises Gated Delta Attention for Google TPUs
Linear attention layers built for GPU tensor cores waste half of each TPU matrix tile, so porting code without redesign leaves large speedups on the table. Teams pre-training on TPUs can adopt SLA-style key width, gate tiling and kernel fusion to cut training time without changing perplexity. Against KDA on TPU v7x, the 11%-24% training speed gain makes this a concrete alternative for MoE runs.