Research map / Adversarial robustness and security / Attacks on text and graphs
Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generalizes adversarial policy attacks in multi-agent RL with a partial-control attack budget that regulates stealthiness, and proposes adversarial training with two-timescale separation that provably converges to the most robust victim policy.
Problem
Existing adversarial policy attacks assume full attacker control and produce easily detectable abnormal behavior, and there are no provably efficient defenses against adversarial policies in multi-agent RL.
Approach
Model partial attacker control as a mixture policy (1-eps) benign + eps adversarial, so the attack budget eps bounds value and state-distribution shift; defend via adversarial training with timescale separation (min oracle or two-timescale gradient updates), with convergence guarantees.
Key results
- Constrained attack with eps=0.3 to 0.7 achieves the same winning rate as unconstrained attack with much lower state-distribution shift (Wasserstein-2) in RoboSumo.
- Adversarial training with min oracle in Kuhn Poker gives fastest convergence and lowest exploitability; two-timescale training approximates it and beats single-timescale training.
- In RoboSumo, two-timescale adversarial training yields victims with lower attacker win rate than single-timescale, self-play and fictitious-play baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Constrain the adversarial policy with an attack budget instead of using an unconstrained attack | Robosumo attacks · Robosumo | state distribution shift | Helped |
| Reduce the attack budget (ablation) | Generalized adversarial policy · Robosumo | visual similarity to the system without an attack | Helped |
| Vary the attack budget to trade off stealthiness against attack performance (ablation) | Generalized adversarial policy · Robosumo | attack performance and stealthiness | Mixed |
| Use a min oracle for the victim update rather than single-timescale adversarial training | Adversarial training · Kuhn Poker | victim exploitability | Helped |
| Increase the attacker-to-victim learning-rate ratio using two-timescale training | Adversarial training · Kuhn Poker | victim exploitability | Helped |
| Replace two-timescale training with single-timescale training | Adversarial training · Robosumo | learning stability | Hurt |
| Use two-timescale adversarial training instead of single-timescale training | Victim policy · Robosumo | attacker winning rate against the victim | Helped |
| Use two-timescale adversarial training instead of self-play or fictitious-play | Victim policy · Robosumo | attacker winning rate against the victim | Helped |
Surprising findings
- Single-timescale adversarial training yields unstable learning with large variance and gradient norm in RoboSumo.
Limitations
- Theory assumes direct policy parameterization and bounded mismatch coefficient; stochastic and function-approximation extensions are only claimed.
- Only a single attacker is considered; multiple independent self-interested attackers are left for future work.
- Min oracle is not available in continuous control tasks, so the two-timescale approximation is used there.
Experimental setup
Datasets: Kuhn Poker RoboSumo
Benchmarks: Kuhn Poker RoboSumo
Metrics: winning rate exploitability Wasserstein-2 distance (squared) of state distribution gradient norm victim score (win rate + tie rate/2)
Baselines: unconstrained adversarial policy single timescale adversarial training self-play fictitious-play
Builds on: PPO adversarial policies in multi-agent RL (Gleave et al.) SA-MDP PR-MDP regret policy gradient
Closest papers by meaning
- Optimal Attack and Defense for Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.90)
- Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.89)
- Diffusion Guided Adversarial State Perturbations in Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- PolicyGuard: Towards Test-time and Step-level Adversary Defense for Reinforcement Learning Agent (ICML 2026 · similarity 0.88)
- AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.