研究提出新方法提升强化学习策略对抗输入扰动的鲁棒性
Robust Policy Optimization via Adversarial Importance Sampling
这是篇关于如何提升强化学习模型鲁棒性的研究论文，作者开源了实现代码，对做相关研究的人可能有参考价值。
本文提出Adversarial Importance Sampling（Advis）方法，通过重要性采样估计和优化可验证的最坏情况回报，无需额外环境交互或辅助网络。同时开发了advrl库，提供现有鲁棒性方法和攻击的干净实现。研究还指出，在评估对抗性攻击时，最优超参数不适用于不同代理，因此使用更多配置（比先前工作多6-14倍）来评估策略。最终在连续控制环境中验证了该方法的有效性。
Robust Policy Optimization via Adversarial Importance Sampling
Significant progress has been made in safeguarding deep reinforcement learning (DRL) policies against input perturbations. Developing robust DRL involves three main stages: algorithm design, implementation, and evaluation. In this work, we identify and address a key limitation at each stage. First, we introduce Adversarial Importance Sampling (Advis), a method that uses importance sampling over trajectories from standard training to estimate and optimize verifiable worst-case returns. Advis satisfies three desirable criteria not jointly achieved by prior work: it requires no additional environment interactions, no auxiliary networks, and captures long-term robustness. Second, we introduce advrl, a modular PyTorch library that provides clean, single-file implementations of existing robustness methods and adversarial attacks, facilitating rapid prototyping and enabling reproducible and traceable evaluations. Third, we revisit evaluation under learned adversaries and show that optimal adversarial hyperparameters do not transfer across agents, which can lead to an overestimation of robustness when using a limited set of attacker configurations. Accordingly, we evaluate policies against a large and diverse set of attackers, using 6-14x more configurations than prior work. Finally, we evaluate our approach on continuous control environments, demonstrating its effectiveness relative to existing baselines. The code is available at: https://github.com/AmineAndam04/advrl
