这篇论文把稀疏奖励变密集，让RL学得快。机器人实验证明比原来快很多，实用性强。
该论文提出通过成功访问匹配（Success Visitation Matching）将稀疏的结果奖励（0/1）转化为密集的过程奖励。方法训练一个判别器来区分成功和失败的轨迹，并激励RL策略匹配成功轨迹的状态-动作访问。理论证明该方法不改变最优策略。在机器人控制策略微调中，模拟和真实操作任务上的收敛速度均显著快于直接使用稀疏奖励的基线。
Learning Process Rewards via Success Visitation Matching for Efficient RL
In many modern applications of reinforcement learning (RL), the natural reward for a task of interest is inherently sparse: a reward of 0 is given everywhere except when the task is completed, when a reward of +1 is given. Training a policy to maximize such a sparse reward requires solving a challenging credit assignment problem, leading to slow or ineffective RL improvement. We propose a simple approach to transform a sparse outcome reward into a dense process reward. Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes, and using this discriminator to incentivize the RL-learned policy to match the state-action visitations of successful episodes, while avoiding those of unsuccessful episodes. By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion, and, we show, provably achieves this without changing the optimal policy. Focusing on finetuning of robotic control policies, we demonstrate that our approach leads to significantly faster RL finetuning performance on both simulated and real-world manipulation tasks, as compared to simply maximizing the sparse outcome reward.