这篇论文用Pythia模型实测了Transformer内部动力学的红外组织现象，发现记忆核呈标度无关性，挺硬核的理论研究。
该研究在Pythia语言模型中提取Transformer层雅可比矩阵的松弛谱，直接测量了弛豫态密度（TDOS）、记忆自能、遗忘间隙、记忆核和红外临界指数。测量发现慢弛豫模式呈现红外累积，产生近似平坦的TDOS（ρ(λ)~λ^{-0.1}）和标度无关的记忆核（K(t)~t^{-1}）。记忆自能在早期优化中出现瞬态最大值，随后松弛到近临界亚稳态，此时遗忘间隙最小、集体磁化率最大。这些定量实验证据表明Transformer动力学受红外集体组织支配，且该行为在不同训练阶段、网络深度和模型规模下均可复现。
Infrared Organization and Critical Cognitive Field Formation in Transformer Dynamics
Large language models exhibit remarkable emergent behaviors, yet the physical mechanism governing their collective dynamics remains poorly understood. Cognitive Field Theory predicts that learning reorganizes the time-scale density of states (TDOS) through the infrared accumulation of slow relaxation modes, thereby enhancing the memory self-energy, reducing the cognitive forgetting gap, and strengthening the collective susceptibility. Using publicly available Pythia language models, we extract relaxation spectra directly from Transformer layer Jacobians throughout training, network depth, and model scale, allowing the TDOS, memory self-energy, forgetting gap, memory kernel, and infrared critical exponent to be measured quantitatively. The measurements reveal progressive infrared accumulation of slow relaxation modes, producing an approximately flat infrared TDOS with \( ρ(λ)\simλ^{-0.1} \) and scale-free memory kernels \( K(t)\sim t^{-1}. \) The memory self-energy exhibits a pronounced transient maximum during early optimization before relaxing toward a metastable near-critical regime, corresponding to the smallest cognitive forgetting gap and the largest collective susceptibility predicted by Cognitive Field Theory. These observations provide quantitative experimental evidence that Transformer dynamics are governed by infrared collective organization. The reproducibility of the same dynamical behavior across training, network depth, and model scales suggests that infrared slow-mode organization represents a universal collective principle of Transformer dynamics.