研究：注意力门控带来弃权与噪声过滤，收益随模型规模此消彼长
Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
论文把门控为何有效讲透了：10M 小模型靠弃权，350M 靠滤噪，两个都用最好，几乎不加参数。
一篇 arXiv 论文提出，对 attention 的 value 通路加门控之所以能改善预训练，是因为补上了 softmax attention 缺失的两种原语：弃权与噪声过滤。实验在 10M 到 350M 参数的匹配模型上进行，弃权通过 softmax 中的 per-head sink logit 实现，噪声过滤通过每个 value 上的门控实现。结果显示，弃权的收益随规模增大而下降，在 10M 模型上几乎解释了门控带来的全部验证损失下降，噪声过滤则在 350M 模型上贡献了大部分收益。每个规模上表现最好的模型都同时内置两种原语；注入受控干扰的实验证实 value 门控能滤除 residual stream 中叠加特征的干扰，两种门控形式各有盲区。整套方案新增参数几乎可忽略，并与 KV cache 兼容。
Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention
Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering. The first is abstention, which allows an attention head to output nothing, bypassing the requirement that attention weights must sum to one. The second is noise filtering, which allows the value pathway of an attention head to suppress interference from superposed features in the residual stream. In our experiments in matched models from 10M to 350M parameters, we supply abstention through a learned per-head sink logit in the softmax and noise filtering through a gate on each value. We report three empirical findings. First, the benefit of abstention, measured as the reduction in validation loss relative to a matched baseline, declines as models grow, whereas the benefit of noise filtering increases with scale. In particular, abstention accounts for nearly all of the gain from gating at 10M and filtering for most of it at 350M. Second, the best model at every scale is the one with both primitives built in. Third, injecting controlled interference into the values a head reads confirms that the gate removes such interference, and reveals that each of the two gate forms we study has a characteristic blind spot. Supplying both primitives adds negligible parameters and remains compatible with the key-value cache.