这篇论文提出用世界模型加分层策略搞安全RL，在长期任务上比现有方法成功率和安全性都高不少，值得一看。
本研究提出Imagine方法，结合可学习世界模型与高层、低层两种互补策略，解决强化学习中的安全探索问题。高层策略生成中间子目标，引导探索偏向安全区域；低层策略利用世界模型的想象展开减少不安全行为。在长时域高维动作空间的导航和操作任务上，该方法在成功率和约束满足方面显著优于现有安全RL基线，一致满足预设安全预算。
Imagine to Ensure Safety in Hierarchical Reinforcement Learning
This work investigates the safe exploration problem in reinforcement learning, where an agent must maximize cumulative performance while simultaneously satisfying safety constraints. This challenge becomes even more pronounced in long-horizon tasks, where existing safe methods face fundamental limitations due to compounding estimation errors and restricted exploration capabilities. To address this problem, we propose a method that combines a learnable world model with two complementary policies a high-level policy and a low-level policy to promote safety at both hierarchical levels. The high-level policy generates intermediate subgoals that bias exploration toward safe regions, while the low-level policy uses imagined rollouts in the learned world model to reduce unsafe behaviors when reaching these subgoals. The proposed method was evaluated on challenging long-horizon navigation and manipulation tasks with high-dimensional action spaces, where it significantly outperforms existing Safe RL baselines in both success rate and strong empirical constraint satisfaction, consistently meeting the prescribed safety budget across seeds, while prior approaches fail to effectively solve these complex long-horizon scenarios.