这篇论文给HRL的高层智能体装上了'风险雷达'，用粗动力学的不确定性来选子目标，比传统方法更稳更准，值得做强化学习的看看。
该论文提出S3方法，在分层强化学习（HRL）中引入基于粗粒度动力学（Coarse Dynamics）的内在动机，以稳定高层智能体的子目标选择。通过最小化由混合密度网络（MDN）建模的预测不确定性，该方法在长时域非平稳环境中引导高层策略。实验表明，这种稠密的内在奖励机制使子目标选择更倾向于风险规避，并在多个基准任务上超越现有SOTA HRL方法（如HIRO、HAC）。
S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning (HRL) intends to separate strategic planning from primitive execution. It has been widely successful in solving long-horizon and complex tasks, where flat-RL algorithms have difficulty in learning. However, while the low-level agent in HRL benefits from dense feedback and abundant trial opportunities, the high-level agent receives sparse, delayed feedback from the environment and its performance depends on the low-level execution capability. In this paper, we study whether subgoal selection by the high-level agent can be performed more strategically, by providing it with dynamics-aware intrinsic motivation. Since motivation based on primitive transition dynamics would require broad coverage of the state-action space, we propose to use coarse dynamics, i.e., environment transitions aggregated over multiple steps at the temporal scale at which the high-level agent operates. This approach stabilizes the high-level policy by learning to minimize the predictive uncertainty associated with the coarse dynamics, and provides a guided structure for navigation. We model the predictive uncertainty by evaluating different dispersion metrics as approximated by a Mixture Density Network (MDN). Empirically, we observe that a dense, dynamics-aware intrinsic reward leads to risk-averse subgoal selection, enabling it to outperform state-of-the-art HRL methods in non-stationary long-horizon environments.