Softmax Attention Study Identifies Abstention and Noise Filtering as Missing Primitives
Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Abstention explains nearly all gating gains at 10M parameters while noise filtering dominates by 350M, a concrete scaling rule for teams training or fine-tuning small language models on standard softmax attention.
An arXiv paper (2609.22005) argues that gating the value pathway of softmax attention improves language model pretraining because it adds two primitives standard attention lacks: abstention and noise filtering. In matched models from 10M to 350M parameters, the authors implement abstention with a learned per-head sink logit and noise filtering with a gate on each value. The measured validation-loss benefit of abstention shrinks as models grow, while the benefit of noise filtering grows with scale, so abstention accounts for nearly all the gating gain at 10M and filtering for most of it at 350M. The best model at every scale combines both primitives, and injected-interference tests show each gate form has a characteristic blind spot, with negligible added parameters and full key-value cache compatibility.
Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering. The first is abstention, which allows an attention head to output nothing, bypassing the requirement that attention weights must sum to one. The second is noise filtering, which allows the value pathway of an attention head to suppress interference from superposed features in the residual stream. In our experiments in matched models from 10M to 350M parameters, we supply abstention through a learned per-head sink logit in the softmax and noise filtering through a gate on each value. We report three empirical findings. First, the benefit of abstention, measured as the reduction in validation loss relative to a matched baseline, declines as models grow, whereas the benefit of noise filtering increases with scale. In particular, abstention accounts for nearly all of the gain from gating at 10M and filtering for most of it at 350M. Second, the best model at every scale is the one with both primitives built in. Third, injecting controlled interference into the values a head reads confirms that the gate removes such interference, and reveals that each of the two gate forms we study has a characteristic blind spot. Supplying both primitives adds negligible parameters and remains compatible with the key-value cache.