研究重新审视“RL 泛化、SFT 记忆”：SFT 数据质量与规模是关键

在数据规模上，当 SFT 与 RL 匹配正确 rollout 数量时，SFT 泛化能力大幅提升；匹配总 rollout 预算时，SFT 能达到与 RL 相当的泛化水平；两者结合后 SFT 甚至超过 RL。作者还用 LLM 标注分析训练 rollout 中的解题方法，发现更大数据集覆盖更多尾部方法，这些尾部方法提供了可泛化的推理信号。

论文最后通过分析浅层 Transformer 在 RL 与 SFT 下的训练动态，为上述实证结论提供理论支撑。

Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data

Large language models trained with Reinforcement Learning (RL) with verifiable rewards exhibit strong reasoning ability and broad generalization, whereas models trained with Supervised Fine-Tuning (SFT) are often viewed as more prone to memorization and limited transfer. This paper rethinks this distinction through the lens of SFT training data on widely used Qwen2.5 and Qwen3 series models. First, we study the role of data source and show that it is critical: a carefully mixed SFT dataset substantially outperforms data generated solely by a larger model. Second, we study the role of data scale and show that matching the number of correct rollouts between SFT and RL greatly improves SFT generalization, while matching the total rollout budget enables SFT to generalize as well as RL. Combining these two factors further enables SFT to generalize even better than RL. Third, by using LLM annotations to characterize the solution methods in training rollouts, we show that larger datasets cover more tail methods and that these tail methods provide generalizable reasoning signals. Finally, we support these empirical findings theoretically by analyzing the training dynamics of shallow transformers under both RL and SFT.

This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SFT, RL, Qwen2.5, Qwen3.

Source attribution: curated from arXiv cs.LG (@Yunlong Hou, Fengzhuo Zhang, Yuan Cheng, Jiachun Pan, Xingyao Li, Zhuoran Yang). Cross-source corroboration: 2 independent sources have covered this development (including elvis), confirming industry-wide signal strength.