CMU 等提出统一训练：预训练、SFT、RL 可合并为单一算法
CMU 这篇论文把 SFT 和 RL 揉进一个损失函数里训练，124M 和 0.76B 的小实验都打赢了传统分阶段流程，做训练的可以看看。
CMU 等机构论文提出 unified training，把预训练、midtraining、SFT 和 RL 统一到一个基于似然的算法下，核心思路是把 RL 目标视为对模型生成目标的最大似然拟合。他们从零训练了一个 124M 模型做推理谜题，最佳统一训练结果在 RL 任务和未见过的更大谜题上都超过分阶段训练。在 0.76B 模型的数学 RL 实验中，混入 0% 到 5% 的 midtraining 数据后，数学准确率保持不变，保留率提升，其中 5% 混合比例效果最好。持续学习实验中，数学训练后所有统一训练模型在最终编码准确率上都超过分阶段训练。作者承认实验规模较小，尚未找到普适最优的混合方案。
Highly recommended paper from CMU and others on how foundation models get trained. It fundamentally asks whether pretraining, midtraining, SFT, and RL really need to be separate stages. They show that this separation isn't fundamental. All of these stages can run under a single likelihood-based algorithm, which they call unified training. The key step treats suitable RL objectives as maximum-likelihood fitting to model-generated targets. RL generations and static data then share one loss and one trainer. This changes the design problem. You now choose which data to train on, in what proportions and when. Staged training becomes one special case. They first trained a 124M model from scratch on reasoning puzzles. The best unified runs beat staged training on the RL tasks. They also beat it on larger puzzles the model never saw. Those runs reused the midtraining corpus up to 4 times. Staged training lost accuracy when midtraining ran past its best budget. Under a fixed compute budget, unified training had the best accuracy at the smallest and largest budgets. Staged training led at intermediate budgets. The real-world test used a 0.76B model on math RL, with 0% to 5% midtraining data mixed in. Math accuracy stayed similar while retention improved. Coding accuracy dropped as the mix grew. Continual learning gave the clearest result. After math training, all unified models beat staged training on final coding accuracy, despite starting lower. The 5% mix did best. The experiments are small-scale. The authors have not found a mixture or schedule that works best across the board. arxiv.org/abs/2610.40564 Chat with Paper: academy.dair.ai/papers/trainin… 💬 17 🔄 10 ❤️ 58 👀 4345 📊 29 ⚡
