Research map / Adversarial robustness and security / Certified and empirical robustness
Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PROTECTED, which iteratively discovers a small set of non-dominated victim policies during training and uses adversarial-bandit online adaptation at test time to defend against both worst-case and non-worst-case state-adversarial attacks in MuJoCo.
Problem
Robust RL policies trained against worst-case state-adversarial attacks sacrifice natural performance and performance against weak or non-worst-case attacks; the goal is test-time adaptive defense with regret guarantees.
Approach
Frame test-time defense as regret minimization over a small finite policy class refined during training by iteratively adding non-dominated policies (those maximizing the gap to the mixtures of already-found policies), then run an EXP3-style adversarial bandit over that class at test time.
Key results
- On Walker2d, the method reaches natural reward 6319 ± 31 versus 4156 ± 495 for WocaR-PPO, the strongest baseline.
- With |Π̃|=5, the method outperforms baselines on most attacks in all four MuJoCo environments; on Hopper under PA-AD it scores 2896 vs 2579 for WocaR-PPO.
- The paper reports that only about 3 policies in Π̃ are needed to beat baselines against PA-AD attacks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PROTECTED instead of the strongest listed baseline | Hopper · Hopper natural | episode reward: 3616 ± 99 -> 3652 ± 108 | Helped |
| Use PROTECTED instead of the strongest listed baseline | Walker2d · Walker2d natural | episode reward: 4178 ± 529 -> 6319 ± 31 | Helped |
| Use PROTECTED instead of the strongest listed baseline | Halfcheetah · Halfcheetah natural | episode reward: 6289 ± 342 -> 7095 ± 88 | Helped |
| Use PROTECTED instead of the strongest listed baseline | Ant · Ant natural | episode reward: 5596 ± 225 -> 5769 ± 290 | Helped |
| Use PROTECTED instead of the strongest listed baseline | Hopper · Hopper PA-AD attack | episode reward: 2521 ± 325 -> 2896 ± 723 | Helped |
| Use PROTECTED instead of the strongest listed baseline | Walker2d · Walker2d PA-AD attack | episode reward: 2722 ± 173 -> 5803 ± 857 | Helped |
| Use PROTECTED instead of the strongest listed baseline | Halfcheetah · Halfcheetah SA-RL attack | episode reward: 5365 ± 54 -> 5089 ± 86 | Hurt |
| Use PROTECTED instead of the strongest listed baseline | Ant · Ant PA-AD attack | episode reward: 3164 ± 163 -> 4312 ± 281 | Helped |
| Use policy class sizes within 3 for PROTECTED (ablation) | PROTECTED · MuJoCo PA-AD attacks | episode reward | Helped |
| Adapt online against periodic and probabilistic switching attackers rather than use a single static victim | PROTECTED · Ant dynamic attacks | average cumulative reward | Helped |
Surprising findings
- Only about 3 policies in the refined class are needed to outperform baselines against PA-AD attacks.
- On Ant, the method is worse than WocaR-PPO against some attacks in some columns, and PA-AD attacks on Ant hit ATLA-PPO especially hard.
Limitations
- Training overhead is roughly 2x compared with prior methods.
- Theoretical hardness results mean the policy class may need to be large for some problems; the Lipschitz-type condition that would avoid it is not resolved.
- Evaluation is limited to four MuJoCo tasks with state-adversarial attacks.
Experimental setup
Datasets: MuJoCo Hopper MuJoCo Walker2d MuJoCo Halfcheetah MuJoCo Ant
Benchmarks: MuJoCo Hopper MuJoCo Walker2d MuJoCo Halfcheetah MuJoCo Ant
Metrics: average episode reward robust reward under attack
Baselines: ATLA-PPO PA-ATLA-PPO WocaR-PPO
Builds on: SA-MDP PA-AD ATLA WocaR-RL EXP3 PPO
Closest papers by meaning
- Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL (ICLR 2024 · similarity 0.90)
- Optimal Attack and Defense for Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation (ICML 2025 · similarity 0.89)
- Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error (ICML 2024 · similarity 0.89)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.