这篇论文解释了为啥RoPE比Sinusoidal位置编码更受青睐——它减少了参数空间的对称性，让Transformer表达力更强。如果你好奇背后的理论，值得一看。
这篇论文研究了Transformer中注意力机制的函数等价性，重点分析了sinusoidal和旋转位置编码（RoPE）两种变体。作者发现sinusoidal编码保留了普通注意力的等价结构，而RoPE显著减小了对称群，从而增强了表达力。这一发现为RoPE在实践中的流行提供了理论解释。论文还讨论了位置编码如何影响线性模式连接性，并通过对齐算法证明连接性的存在和变化关键依赖于位置编码。实验表明使用RoPE的Transformer在参数空间具有更少的函数等价性，有助于优化和泛化。
Functional Equivalence in Attention: A Comprehensive Study with Applications to Linear Mode Connectivity
Neural network parameter spaces are inherently non-injective, as distinct parameter configurations can realize identical functions through functional equivalence. While this symmetry is well understood in classical fully connected and convolutional models, it becomes substantially more intricate in modern attention-based architectures. Existing analyses of multihead attention have largely focused on the vanilla formulation, overlooking positional encodings that fundamentally reshape architectural symmetries. In this work, we provide a formal study of functional equivalence in Transformers with positional encodings. Focusing on the two most widely used variants--sinusoidal and rotary positional encodings (RoPE)--we show that sinusoidal encodings preserve the equivalence structure of vanilla attention, whereas rotary encodings significantly reduce the symmetry group, thereby enhancing expressivity. This offers a principled explanation for the growing prominence of RoPE in practice. We further examine how positional encodings affect linear mode connectivity, and through an alignment algorithm, empirically demonstrate that the presence and variability of connectivity across Transformer settings crucially depend on the positional encoding.