OpenAI等机构常用方法被挑战，论文证明先OPD后RL的两阶段训练效果更好，还给出了实用切换信号。
该论文对比了强化学习可验证奖励(RLVR)与在线策略蒸馏(OPD)两种大模型推理训练方法。研究发现简单的两阶段方案(OPD-then-RL)在逻辑和数学推理基准上表现优于纯OPD、纯RLVR及所有联合基线方法。OPD扩大学生对教师支持解决方案的覆盖范围，而RL则在此范围内进行优化，两者联合优化会导致信号干扰。
Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR
Reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) have emerged as two dominant methods for post-training reasoning LLMs. Prior work uses OPD's dense token-level supervision to complement the sparse RL reward, fusing the two signals within a single step: either as a \emph{weighted-additive combination} or a \emph{teacher-modulated rescaling} of the RL advantage. In this paper, we show that a simple two-stage scheme, OPD-then-RL, consistently outperforms pure OPD, pure RLVR, and all such joint baselines across logic and math reasoning benchmarks. Beyond the empirical results, we further provide a systematic understanding of this through pass@$k$ behavior, learning dynamics, and parameter updates, yielding a consistent explanation: OPD expands the student's coverage of teacher-supported solutions and RL sharpens within that support, while jointly optimizing the two signals causes them to interfere.To provide a practical recipe, we find that the OPD validation score is the key signal for when to switch to RL, and that OPD is a better cold start for RL than SFT. Together, our results establish OPD-then-RL as a simple yet strong way to combine the two methods, turning two entangled signals into complementary stages.