这项研究用OLMo模型从4M到4B参数实验，直观解释了为什么小模型学不会稀有任务——不是数据不够，是神经元竞争导致梯度被覆盖。
研究人员证明某些任务需要模型缩放而非数据缩放，小模型即使拥有无限数据也无法学会这些任务。原因在于神经元竞争：频繁任务优先抢占容量，稀有任务的梯度在下次更新前被覆盖，导致模型陷入学习后遗忘的循环。团队预训练了参数从4M到4B的OLMo模型，在训练中注入可控频率的新任务。只有最大的4B参数模型成功学会了稀有任务，干扰梯度几乎消失。
We finally know why bigger models are smarter. It'…
We finally know why bigger models are smarter. It's not the data.
More training data was supposed to fix small models.
A new paper shows why it cannot. Researchers proved some tasks need model scaling, not data scaling.
A small model fails them even with infinite data. The cause is competition over neurons.
Frequent tasks grab capacity first and keep it. Rare task updates get overwritten before the next example arrives.
The model learns, then forgets, in an endless loop. Scaling breaks the loop in three steps: 1. Common tasks get fully learned 2. Their gradients fade to nothing 3. Rare features accumulate safely The team pretrained OLMo models from 4M to 4B parameters.
They injected novel tasks at controlled frequencies during training. Only the largest models learned the rare ones.
Interference between their gradients nearly disappeared. How many tasks is your model silently skipping?