Robust Policy Optimization via Adversarial Importance Sampling
This paper provides a new method and library for improving the robustness of reinforcement learning models, which is a key area of research in AI safety and reliability.
This academic paper introduces Advis, a method for optimizing deep reinforcement learning policies against input perturbations.
It uses importance sampling over standard training trajectories to estimate worst-case returns without extra environment interactions or auxiliary networks.
The authors also created the advrl library for implementing and evaluating robustness methods.They found that optimal adversarial hyperparameters don't transfer and used a large, diverse set of attacker configurations (6-14x more than prior work) for evaluation.
Significant progress has been made in safeguarding deep reinforcement learning (DRL) policies against input perturbations. Developing robust DRL involves three main stages: algorithm design, implementation, and evaluation. In this work, we identify and address a key limitation at each stage. First, we introduce Adversarial Importance Sampling (Advis), a method that uses importance sampling over trajectories from standard training to estimate and optimize verifiable worst-case returns. Advis satisfies three desirable criteria not jointly achieved by prior work: it requires no additional environment interactions, no auxiliary networks, and captures long-term robustness. Second, we introduce advrl, a modular PyTorch library that provides clean, single-file implementations of existing robustness methods and adversarial attacks, facilitating rapid prototyping and enabling reproducible and traceable evaluations. Third, we revisit evaluation under learned adversaries and show that optimal adversarial hyperparameters do not transfer across agents, which can lead to an overestimation of robustness when using a limited set of attacker configurations. Accordingly, we evaluate policies against a large and diverse set of attackers, using 6-14x more configurations than prior work. Finally, we evaluate our approach on continuous control environments, demonstrating its effectiveness relative to existing baselines. The code is available at: https://github.com/AmineAndam04/advrl
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and AI regulation, compliance frameworks, and sovereign compute policy. Key entities and technical topics under continuous tracking include Robust Policy Optimization via Adversarial Importance Sampling, reinforcement learning, robustness, Adversarial Importance Sampling.
Source attribution: curated from arXiv cs.LG (@Amine Andam, Jamal Bentahar, Mustapha Hedabou).