做模型训练或理解 scaling law 的团队值得一读——这篇论文把大模型涌现能力的机制讲清楚了，不是玄学而是容量与干扰的数学问题。
来自斯坦福、MIT、哈佛和 Anthropic 的联合研究揭示了大型语言模型能学会小模型无法掌握的技能的根本原因：大模型在训练过程中更不容易遗忘稀有技能。其额外容量能保护弱学习信号，而小模型的有限神经元会被常见任务占据，导致稀有任务在学习信号出现足够多次之前就被覆盖。研究通过控制实验和 OLMo 模型（4M 到 4B 参数）验证了这一结论，发现大模型在低频任务上表现更好，能保留更多任务特征，且梯度干扰更小。该论文为模型规模与能力涌现之间的关系提供了清晰的训练层面解释。
Great Stanford + MIT + Harvard + Anthropic paper. …
Great Stanford + MIT + Harvard + Anthropic paper.
Gives a clear training-based reason for why larger models learn abilities smaller models miss.
Says bigger AI models learn rare skills because they forget them less during training, their extra space protects weak learning signals.
The authors say the issue is not just whether a small model could represent the task, but whether training lets it keep that task while many common tasks keep pushing on the same limited parts.
Their core idea is that common tasks take up the model’s neurons first, so rare tasks get overwritten before they appear often enough to build into stable knowledge.
In a crowded data mixture, common patterns get first claim on the model’s internal machinery.
Small models may briefly pick up a rare signal, but the next wave of common-task updates overwrites it before the signal appears again.
They tested this first with controlled toy tasks where they could change how rare and complex each task was, then with OLMo language models from 4M to 4B parameters.
The main result is that bigger models learned low-frequency tasks much better, kept more task features inside their representations, and showed less gradient interference, which means common-task updates disturbed rare-task learning less.
Larger models can remember weak rare signals long enough to turn them into real learned skills.
----
Link – arxiv. org/abs/2605.29548
Title: "Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention"