DeepMind团队提出NSPER方法，结合新颖性和惊喜信号，让强化学习更高效探索。
本文提出NSPER方法，在图像强化学习中整合新颖性和惊喜信号来优化经验回放。NSPER+R进一步将这些信号作为内在奖励，共同提升回放质量和探索效果。实验在DeepMind Control Suite任务上显示，NSPER和NSPER+R比现有方法提高了训练效率和收敛速度。
Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning
Sample efficiency is a central challenge in reinforcement learning (RL), particularly in image-based domains where agents must learn from high-dimensional visual inputs. Traditional sampling often relies on random or suboptimal experience selection, leading to redundant updates and slow learning. Improving efficiency requires mechanisms that prioritize informative experiences while also encouraging effective exploration. Prioritized Experience Replay (PER) addresses part of this challenge by reusing high-value transitions, while intrinsic rewards promote the exploration of novel or uncertain states. However, their integration has not been extensively studied. This paper introduces Novelty and Surprise Prioritized Experience Replay (NSPER), which uses novelty to capture underrepresented states and surprise to expose gaps in the agent's understanding of the environment. We further extend this with NSPER+R, integrating these signals as intrinsic rewards to jointly improve replay quality and exploration. Experiments on DeepMind Control Suite tasks show that NSPER and NSPER+R improve training efficiency and convergence speed compared to existing methods in image-based RL.