RetireOPD：一种用于强化学习智能体的自退火策略蒸馏方法
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
朋友，这是篇挺有意思的论文，讲的是怎么用一种叫RetireOPD的新方法来训练AI智能体，效果比直接用强化学习好不少，在几个任务上都有明显提升。
研究提出了一种名为RetireOPD的自退火策略蒸馏方法，用于训练强化学习智能体。该方法通过优化一个与技能条件解耦的教师模型，并采用自适应退休机制，让学生模型在自身与教师模型差异停止缩小且成功率达到目标阈值后，自行放弃教师监督，仅依靠强化学习继续训练。在Qwen2.5模型上，该方法在ALFWorld任务上比基线提升了14.1%至18.8%，在WebShop任务上提升了11.8%至19.0%。
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.
