这篇论文发现了SFT过训练会搞崩GRPO训练的秘密，还给出了诊断方法来提前止损。做RLHF或强化学习训练的可以看看。
一项研究分析了SFT（监督微调）的过度训练对RLVR（基于强化学习的验证）训练的影响。使用Qwen2.5-Coder-3B和DeepSeek-Coder-6.7B模型，发现SFT深度增加时，预RL的pass@1上升，但GRPO的pass@10从0.806降至0.481（3种子均值，n=20）。预RL熵与GRPO结果正相关（ρ=+0.69）。研究者提出一个两阶段诊断方法，结合预RL熵筛选和早期GRPO熵监控，可标记高风险检查点。简单KL正则化和标签平滑无法挽救已崩溃的检查点。
SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR
The standard heuristic of selecting the SFT checkpoint with the highest pass@1 for GRPO can fail when SFT compresses the rollout distribution. For binary rewards, the expected within group advantage variance is $p(1{-}p)(g{-}1)/g$; when early GRPO drives $p$ below $p^*(g)$, most groups have identical rewards and provide no group relative signal. We study SFT depth ladders for Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B. We test Qwen2.5-Coder-3B across five depths and three seeds, and DeepSeek-Coder-6.7B across four matched depths and three seeds. On Qwen, pre RL pass@1 rises with SFT depth, but peak GRPO pass@10 falls from $0.806$ to $0.481$ (3 seed mean, $n{=}20$); pre RL entropy is positively associated with the GRPO outcome ($ρ{=}{+}0.69$). On DeepSeek, pass@1 remains far above $p^*(8){=}0.083$, and GRPO outcomes compress rather than invert. A two stage diagnostic, combining pre RL entropy triage with an early GRPO entropy monitor, flags high risk checkpoints and can stop failing runs early. Simple KL to reference regularisation and label smoothing variants do not rescue the collapsed Qwen checkpoint in our setting, suggesting the failure is not a trivial GRPO hyperparameter artefact.