arXiv 论文分析排斥性自注意力的非平衡动力学：混沌、注意力凝聚与局域化
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
把 transformer 注意力当成动力系统来分析的物理论文，找出了混沌、凝聚这些相变结构，做理论或注意力机制研究的可以看看。
该论文研究一个极简循环 Transformer，其中 Q=K=I 且值映射 V=-I，N 个归一化 token 在排斥反馈下持续重组表示几何与注意力网络。在 d=2 情形下，token 构成圆上的正多边形不动点，注意力反馈强度 γ 增大后经翻转分岔进入周期二运动和混沌。注意力凝聚在 β∼N² 的缩放区间出现，而 N→∞ 时有限 β 下注意力保持弥散。在 d=N→∞ 的高维情形，模拟显示 β=O(1) 时存在凝聚转变，随 γ 变化呈现弥散单纯形态、共识翻转、混沌活跃路由和碎片化簇翻转等相。
Nonequilibrium Phases of Repulsive Self-Attention: Chaos, Attention Condensation, and Emergent Locality
We study the nonequilibrium dynamics of a minimal recurrent transformer with $N$ normalized tokens, $Q=K=I$, and a negative value map $V=-I$. Similarity-based attention selects nearby representations, while the negative value map drives tokens away from the selected field. This feedback can continually reorganize both the representation geometry and the attention network. For $d=2$, the tokens lie on a circle, where the regular polygon is an exact fixed point. As the attention feedback strength $γ$ is increased, the polygon loses stability through a flip bifurcation, giving rise to period-two motion, chaos, and cluster-exchange or cluster-flip states. Despite this temporal complexity, attention remains diffuse as $N\to\infty$ at finite fixed softmax sharpness $β$. Attention condensation instead emerges in the scaling regime $β\sim N^2$. In the hard-routing limit, repulsive updates amplify local perturbations and routing-partner switches transmit them ballistically, producing an emergent butterfly cone in representation space. High-dimensional geometry provides a distinct route to localization. For $d=N\to\infty$, simulations from Gaussian initial conditions provide evidence for a condensation transition at $β=O(1)$, driven by dynamically generated finite overlap gaps. Depending on $γ$, the resulting phases include diffuse simplex-like states, consensus flips, condensed active routing with signatures of chaos, and fragmented cluster flips. These results establish temporal activity, attention condensation, and geometric clustering as distinct collective phenomena, and show that sparse attention can sustain persistent dynamics rather than freeze it.
