Agentic-OPD：从被丢弃的 RL 检查点中回收智能体能力
Agentic-OPD: Recovering Forgotten Agent Capabilities via On-Policy Distillation for Agentic Tasks
做 RL 训练智能体的朋友值得看：你每次只留 best checkpoint，其实丢掉了一堆各有所长的老师。这篇用 on-policy distillation 把它们榨出来，4B 小模型在 deep search 上干过了 GPT-5.6 和 DeepSeek-V4.1，还不花额外训练成本。
一篇 arXiv 论文提出 Agentic-OPD，用于恢复 GRPO 训练中被丢弃的检查点所蕴含的智能体能力。论文指出，标准做法只保留最佳验证检查点，但同一轮训练中早期检查点已解决的问题，最佳检查点反而会失败，所有检查点解决问题的并集远超任何单一检查点。
方法上，论文把最佳验证检查点当作学生，其余所有已保存检查点作为冻结教师候选，用两个模块解决"向谁学、学什么"：HCN（Highest Coverage Next）策略根据学生未解决问题上的加权覆盖率动态选教师，LAD（Locality-Aware Distillation）模块只在教师更强的问题上、且仅针对学生失败的 rollout 进行蒸馏，避免覆盖学生已掌握的行为。在五个智能体任务域、24 个基准与任务子集上，Agentic-OPD 相比初始化检查点最多提升 7.4 分，一个 4B 学生模型在深度搜索任务上超过了 GPT-5.6 和 DeepSeek-V4.1。
这项工作的意义在于把一轮已完成的 RL 训练变成可复用的策略池，且教师池无需额外训练成本即可获得，将"后 RL 能力回收"确立为与 RL 算法改进正交的一个训练阶段，对训练资源有限的团队尤其有参考价值。
Agentic-OPD: Recovering Forgotten Agent Capabilities via On-Policy Distillation for Agentic Tasks
Reinforcement learning is now the mainstream paradigm for training LLMs on agentic tasks, yet standard practice keeps only the best-validation checkpoint and discards the rest. We show that this is lossy: in GRPO training of LLM agents, the best-validation checkpoint fails on problems that earlier checkpoints from the same run had already solved, and the union of problems solved across checkpoints far exceeds any single one. The discarded checkpoints thus form a teacher pool obtained at no additional training cost, which on-policy distillation (OPD) can mine to recover the lost coverage. Mining this pool raises two challenges: i) which checkpoint to distill from, as no checkpoint dominates the others and complementarity to the student is problem-specific; ii) where to trust it, as even a well-chosen teacher is only locally superior, so indiscriminate distillation overwrites behavior the student has already mastered. We propose Agentic-OPD, which treats the best-validation checkpoint as the student and all saved checkpoints as frozen teacher candidates, using the student's evolving capability gaps to guide both teacher selection and supervision filtering: the HCN policy (Highest Coverage Next) dynamically selects teachers by weighted coverage of the student's unsolved problems, while the LAD module (Locality-Aware Distillation) confines transfer to failed student rollouts on problems where the teacher performs better. Across five agentic task domains and 24 benchmarks and task subsets, Agentic-OPD improves over its initialization checkpoint by up to 7.4 points, with a 4B student surpassing the latest frontier models GPT-5.6 and DeepSeek-V4.1 on deep search. Agentic-OPD recasts a finished RL run as a reusable pool of policies and establishes post-RL capability recovery as a stage orthogonal to improvements in the RL algorithm itself.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Agentic-OPD, GRPO, on-policy distillation, 智能体.
Source attribution: curated from arXiv cs.AI (@Tingcheng Bian, Kangye Gong, Jinchang Luo, MingQuan Cheng, Feiran Liu, Ni Li, Xiaoling Xia, Haiwei Wang, Hui Li).
