预训练新趋势:放弃学习率衰减,改用恒定学习率加检查点平均
Sebastian Raschka 把主流模型的 LR 调度分成了三代:余弦、WSD,再到 Kolibri 和 DatologyAI 的恒定 LR 加检查点平均。好处是训练长度可以晚点定,不用提前押注。自己跑预训练或做复现的人值得看看这个分类。
Sebastian Raschka 在社交平台发文指出,新的预训练配方可能正在形成趋势:不再使用学习率衰减阶段,而是全程保持恒定学习率,再对最后 N 个检查点做平均(或使用滑动权重平均)。这一做法在 Kolibri-1 和 DatologyAI 的模型中已有体现。
他基于公开训练细节做了一个简单归类:Llama 3、OLMo 2/3、GLM-4.5、GLM-5、Kimi K3、Phi-4 仍采用经典余弦退火;DeepSeek V3、DeepSeek V4、Kimi K2、Nemotron 3 系列、SmolLM3、MiniMax M2 等使用 WSD(Warmup-Stable-Decay)调度;Kolibri-1 和 DatologyAI 则完全不设衰减阶段。
Raschka 认为,去掉衰减阶段的一个实际好处是训练团队不必预先确定总训练长度,可以灵活决定何时停止。从余弦到 WSD 再到无衰减加权重平均,头部实验室的调度选择正呈现三代演进,对计划训练或复现模型的团队有直接参考价值。
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.)
Instead: constant LR + averaging the last N checkpoints (or a running weight average).
I guess that makes it easier in practice because you don't have to commit to a total training length upfront.
A little meta analysis based on those who share training details:
- Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4
- WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2
- No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include pretraining, Kolibri, DeepSeek, open-source models.
Source attribution: curated from Sebastian Raschka (@rasbt).