做强化学习研究的团队终于有了一个能同时解决分布偏移和灾难性遗忘的通用框架——COOPO 的循环设计让离线数据复用和在线探索形成正向循环，D4RL 上效果显著，建议做 RL 算法开发的同学点开看理论证明和实验细节。
COOPO 是一种新型强化学习框架，通过循环交替进行约束离线训练和在线微调，解决了离线强化学习中的分布偏移和在线学习中的高交互成本问题。该算法在每个循环中先用 KL 正则化的优势加权更新锚定策略，再用任意策略优化进行在线微调，定期回归离线训练可消除遗忘和漂移。理论证明 COOPO 在标准覆盖假设下能实现单调改进，在线样本效率优于纯在线 RL。在 D4RL 基准测试中，COOPO 相比最先进的混合方法减少了在线交互次数，同时提升了最终回报，且对不同离线算法和在线优化器具有鲁棒性。
COOPO: Cyclic Offline-Online Policy Optimization Algorithm
Offline reinforcement learning struggles with distributional shift and constrained performance due to static dataset limitations, while online RL demands prohibitive environment interactions. The recent advent of hybrid offline-to-online methods bridges these domains but suffers from distribution drift during transitions and catastrophic forgetting of offline knowledge. We introduce COOPO (Cyclic Offline-Online Policy Optimization), a generalized framework that repeatedly cycles between constrained offline training and online fine-tuning. Each cycle first anchors the policy to the dataset via KL-regularized advantage-weighted offline updates to minimize distributional shift and then fine-tunes it online using any policy optimization for stable exploration. Crucially, periodically returning to offline training eliminates forgetting and drift while maximizing dataset reuse. The cyclic behavior also helps reduce the online environment interactions. Theoretically, COOPO achieves better online sample efficiency, surpassing pure online RL, with guaranteed monotonic improvement under standard coverage assumptions. Extensive D4RL benchmarks demonstrate COOPO reduces online interactions versus state-of-the-art hybrids while improving final returns, maintaining robustness across diverse offline algorithms and online optimizers. This looped synergy sets new efficiency and performance standards for adaptive RL.