Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning
Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
This research is relevant for anyone working on safety-critical AI systems, as it provides a new method to certify the robustness of reinforcement learning agents against adversarial attacks.
This paper introduces a method to certify the robustness of reinforcement learning agents against adversarial state perturbations.
It extends existing certification techniques to risk-sensitive objectives, providing lower bounds on expected cumulative rewards under such attacks.
The method uses φ-divergence relaxation and dual optimization to formulate the problem as a convex optimization.
Experiments on OpenAI Gym environments show that risk-averse training generally yields policies with higher certified lower bounds, especially under larger perturbation budgets.
Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations
Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under $l_{p}$-norm-bounded state adversarial perturbations ($1\leq p <\infty$). By introducing a $φ$-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter $β$ independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include reinforcement learning, adversarial attacks, risk-sensitive, certification.
Primary source verification: published directly by arXiv: OpenAI with first-party authority.