Research map / Adversarial robustness and security / Certified and empirical robustness
Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that under a consistency assumption of policy the Bellman optimal policy is an optimal robust policy for state-adversarial DQN, shows L-infinity Bellman error is needed for robustness, and trains CAR-DQN on a surrogate of Bellman infinity-error that matches or beats SA-DQN, RADIAL-DQN and WocaR-DQN on four Atari games.
Problem
Standard DRL agents trained toward the Bellman optimal policy lack adversarial state robustness, and prior SA-MDP work suggests an optimal robust policy may not exist, forcing trade-offs between natural and robust performance.
Approach
Introduces a consistency assumption of policy (CAP) under which the Bellman optimal policy is an optimal robust policy, shows that minimizing the Bellman error in L-infinity (rather than L1) is necessary for robustness, and trains DQN with a soft surrogate of the Bellman infinity-error (CAR-DQN).
Key results
- On RoadRunner, CAR-DQN (PGD solver) reaches about 43,286 under PGD attack versus 20,482 for SA-DQN (PGD), roughly 110% higher robust reward.
- Under the L-infinity Bellman error ablation, L1 and L2 DQN collapse to near-zero or negative returns under PGD on Pong, Freeway and RoadRunner, while the L-infinity version keeps returns near natural levels (e.g., RoadRunner 48,230 under PGD).
- CAR-DQN (IBP solver) attains about 0.974-0.986 ACR on the four games, comparable to RADIAL-DQN and WocaR-DQN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with the CAR objective using the PGD solver instead of SA-DQN's PGD robust training | DQN · RoadRunner | PGD reward: 20482 ± 1087 -> 43286 ± 801 | Helped |
| Train with the CAR objective using the PGD solver instead of SA-DQN's PGD robust training | DQN · RoadRunner | natural reward: 33380 ± 611 -> 49700 ± 1015 | Helped |
| Use L^∞ rather than L^1 Bellman error (ablation) | DQN · Freeway | PGD reward: 0.0 ± 0.0 -> 33.2 ± 0.1 | Helped |
| Use L^∞ rather than L^2 Bellman error (ablation) | DQN · Freeway | PGD reward: 21.7 ± 0.3 -> 33.2 ± 0.1 | Helped |
| Use L^∞ rather than L^1 Bellman error (ablation) | DQN · BankHeist | PGD reward: 27.0 ± 2.0 -> 1356.5 ± 1.1 | Helped |
| Use L^∞ rather than L^2 Bellman error (ablation) | DQN · BankHeist | PGD reward: 18.5 ± 1.5 -> 1356.5 ± 1.1 | Helped |
| Use L^∞ rather than L^1 Bellman error (ablation) | DQN · RoadRunner | PGD reward: 0 ± 0 -> 48230 ± 1648 | Helped |
| Use L^∞ rather than L^2 Bellman error (ablation) | DQN · RoadRunner | PGD reward: 0 ± 0 -> 48230 ± 1648 | Helped |
| Set the CAR soft coefficient λ to 0 instead of 1 (ablation) | CAR-DQN · RoadRunner | natural reward: 49500 ± 2106 -> 25160 ± 802 | Hurt |
| Set the CAR soft coefficient λ to ∞ instead of 1 (ablation) | CAR-DQN · RoadRunner | PGD reward: 48230 ± 1648 -> 36760 ± 1874 | Hurt |
| Set the CAR soft coefficient λ to 0 instead of 1 (ablation) | CAR-DQN · Pong | PGD reward: 21.0 ± 0.0 -> 21.0 ± 0.0 | No clear effect |
Surprising findings
- The Bellman optimal policy is itself an optimal robust policy under the consistency assumption, so robustness need not reduce natural performance.
- Standard L1-norm Bellman error DQN collapses to near-zero robust reward on several games even though its natural return is high.
Limitations
- Experiments restricted to discrete action spaces and value-based DQN on four Atari games.
- Theory relies on the consistency assumption, which is supported empirically and theoretically but not guaranteed in general.
- PGD solver yields weaker certified robustness (lower ACR) than the IBP convex relaxation solver.
Experimental setup
Datasets: Atari (Pong, Freeway, BankHeist, RoadRunner)
Benchmarks: Atari Pong Atari Freeway Atari BankHeist Atari RoadRunner
Metrics: episode reward (natural) episode reward under PGD attack episode reward under MinBest attack action certification rate (ACR)
Baselines: SA-DQN RADIAL-DQN WocaR-DQN Standard DQN
Builds on: SA-MDP SA-DQN Double Dueling DQN RADIAL-DQN WocaR-DQN
Closest papers by meaning
- Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.89)
- Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR 2024 · similarity 0.89)
- Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach (NeurIPS 2024 · similarity 0.88)
- Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Confounding Robust Deep Reinforcement Learning: A Causal Approach (NeurIPS 2025 · similarity 0.87)
- Zero-Sum Positional Differential Games as a Framework for Robust Reinforcement Learning: Deep Q-Learning Approach (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.