RetireOPD: Self-Retiring On-Policy Distillation for Agentic RL
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
This research presents a novel method for improving the performance of large language models in reinforcement learning tasks, which could be relevant for developers working on AI agents.
A new method called RetireOPD is proposed for training multi-turn agents with reinforcement learning.
It uses a skill-conditioned teacher to provide dense supervision and an adaptive retirement mechanism where the student stops using the teacher once its performance stabilizes.
This approach was tested on Qwen2.5 models ranging from 1.
5B to 7B parameters and showed improved success rates on ALFWorld and WebShop benchmarks compared to a standard RL baseline.
RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and AI regulation, compliance frameworks, and sovereign compute policy. Key entities and technical topics under continuous tracking include RetireOPD, Qwen2.5, reinforcement learning.
Source attribution: curated from arXiv cs.AI (@Yan Yu, Zhengxi Lu, Yizhou Liu, Yichen Pan, Aozhe Wang, Qipeng Chen, Hua Yang, Wenqi Zhang, Weiming Lu, Qianglong Chen, Yongliang Shen).
