做强化学习训练的团队可以省下从头调参的功夫——用现有基线策略做跳板，训练效率更高且最终策略更强，值得在连续控制任务上试试。
该论文提出了一种新的强化学习训练方法，通过嵌入已有的基线策略来提升训练效率。方法在训练初期依赖基线策略，逐步将控制权转移给可训练的学习策略，最终使学习策略独立运行。理论分析证明了该方法在目标到达概率上的优势，实验表明其在连续控制任务中表现优于或持平于现有方法，且全程保持高目标到达率。
An Agency-Transferring Model-Free Policy Enhancement Technique
Training reinforcement learning (RL) policies from scratch is costly: it requires careful reward and environment design, extensive tuning, and substantial computation. Yet many control problems already have a functional but suboptimal policy available as a baseline. This paper proposes a method for embedding such a baseline into the RL training process, simultaneously improving training efficiency relative to from-scratch methods and producing a learning policy that outperforms the baseline. At each step, the method arbitrates between the baseline policy and a trainable learning policy, initially relying strongly on the baseline policy and then progressively transferring agency to the learning policy. By the end of training, the learning policy is a standalone neural network that operates without baseline policy support. The paper formalizes what it means for the baseline policy to be functional: under this policy, the agent reaches a goal set and remains there with high probability. The proposed arbitration mechanism is designed to exploit this property during training, yielding high goal-reaching rates right from the beginning of training. A theoretical analysis provides a formal interpretation of this behavior under stated assumptions and extends it to the final baseline-free regime, where explicit lower bounds are derived for the goal-reaching probability of the standalone learning policy. Empirical results on continuous-control benchmarks show that the proposed method achieves returns that match or exceed those of competitive approaches, while maintaining the highest goal-reaching rates throughout training among the compared methods -- including in the final stage, where the learning policy operates without any baseline support.