Papers

结构-残差归纳偏置改进语言模型预训练,GPT-2 Medium 上 NLL 降低 1.44%

Structure--Residual Inductive Biases for Language Model Pretraining

Why we picked it

这篇论文挑战了下词预测包办表示学习的默认做法,把结构和细节拆开当显式偏置。BBC 和 CC 一个管前层一个管后层,300M token 就能看到 1.44% 的 NLL 改善,还开箱即用给了代码。做预训练或架构研究的人可以直接拿来跑实验。

一篇 arXiv 论文提出结构-残差(structure–residual)原则:让表示显式地把结构与实现细节分开,灵感来自 Kolmogorov 复杂度的两部分编码视角。作者给出两种实现:Boundary Bottleneck Concept(BBC)让信息经由边界载体在 token 段之间路由,同时保留局部自回归计算;Continuous Concept(CC)通过正则化表示几何,把表示谱分解为低频结构与高频残差。

Body

实验显示两种方法对网络深度偏好截然不同:BBC 在前几层最有效,CC 则在后面的层最有效。在 300M 训练 token 时,BBC 与 CC 结合使 GPT-2 Medium 的 NLL 降低 1.44%,且增益在最长 20B token 的训练中持续存在。诊断结果也符合机制预期:BBC 提高了边界状态的相对预测重要性,CC 降低了谱残差能量。论文还推导了两者的编码界,证明当段内路径能量趋于零时,多重性修正的边界注意力变得精确。代码已在 GitHub 开源。

Original · arXiv cs.AI

Structure--Residual Inductive Biases for Language Model Pretraining

Next-token prediction is relied upon to learn both predictive behavior and internal representations. We ask whether representation structure itself should be an explicit inductive bias. Motivated by the two-part coding view of Kolmogorov complexity, we study a structure--residual principle in which a representation separates structure from realization detail. We instantiate this principle in two complementary ways. Boundary Bottleneck Concept (BBC) routes information from token segments through boundary carriers while preserving local autoregressive computation. Continuous Concept (CC) regularizes representation geometry and induces a spectral decomposition into low-frequency structure and high-frequency residual. We derive coding bounds for both views and show that multiplicity-corrected boundary attention becomes exact as the within-segment path energy vanishes. Empirically, BBC and CC exhibit strikingly different depth preferences: BBC works best in early layers, whereas CC is most effective in later layers. At 300M training tokens, combining BBC and CC improves NLL by 1.44\% on GPT-2 Medium. These gains persist with training runs up to 20B tokens. Representation diagnostics show that BBC increases the relative predictive importance of boundary states, while CC reduces spectral residual energy. Together, these results suggest that structure--residual biases can improve language-model pretraining beyond next-token prediction. Code: https://github.com/haihuang-ml/structure-residual

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include GPT-2, pretraining, representation learning, inductive bias.

Source attribution: curated from arXiv cs.AI (@Hai Huang, Bartosz Żuk, Rassul Magauin, Jiaxin Huang, Xue Liu).