Research map / Adversarial robustness and security / Certified and empirical robustness
Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Quantal Adversarial RL (QARL), which uses entropy-regularized zero-sum Markov games (QRE/bounded rationality) and an automatic gamma-distribution curriculum that gradually increases adversary rationality by decreasing its SAC temperature.
Problem
Robust adversarial RL (RARL) requires solving hard saddle-point (Nash equilibrium) optimization problems, which are unstable and prone to suboptimal solutions in high-dimensional continuous control.
Approach
Entropy-regularize the zero-sum Markov game so the solution is a logit quantal response equilibrium whose temperature controls the adversary's rationality; then curriculum-anneal the adversary's temperature from irrational to fully rational, using a constrained KL-based update of a gamma distribution over temperatures that keeps protagonist performance above a threshold.
Key results
- QARL achieves +4.2±0.5% performance and +48.7±1.2% robustness improvement over SAC across 15 MuJoCo problems, versus RARL's -5.7% performance and +10.1% robustness.
- In the quadruped maze, QARL resists wind forces up to four times the training strength, while RARL and other baselines do not (except Force Curriculum sometimes).
- Simple linear and point curricula are compared against the automatic gamma-distribution curriculum in robustness ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QARL instead of SAC | Adversarial RL · 15 MuJoCo control problems | Performance improvement vs SAC (%): +4.2\pm0.5% | Helped |
| Use QARL instead of SAC | Adversarial RL · 15 MuJoCo control problems | Robustness improvement vs SAC (%): +48.7\pm1.2% | Helped |
| Use SAC MixedNE-LD instead of SAC | Adversarial RL · 15 MuJoCo control problems | Performance improvement vs SAC (%): -16.6\pm0.2% | Hurt |
| Use SAC MixedNE-LD instead of SAC | Adversarial RL · 15 MuJoCo control problems | Robustness improvement vs SAC (%): +20.0\pm1.1% | Helped |
| Use RARL instead of SAC | Adversarial RL · 15 MuJoCo control problems | Performance improvement vs SAC (%): -5.7\pm0.3% | Hurt |
| Use RARL instead of SAC | Adversarial RL · 15 MuJoCo control problems | Robustness improvement vs SAC (%): +10.1\pm0.6% | Helped |
| Use Force Curriculum instead of SAC | Adversarial RL · 15 MuJoCo control problems | Performance improvement vs SAC (%): -14.7\pm0.3% | Hurt |
| Use Force Curriculum instead of SAC | Adversarial RL · 15 MuJoCo control problems | Robustness improvement vs SAC (%): -3.3\pm0.6% | Hurt |
| Use CAT instead of SAC | Adversarial RL · 15 MuJoCo control problems | Performance improvement vs SAC (%): -18.0\pm0.2% | Hurt |
| Use CAT instead of SAC | Adversarial RL · 15 MuJoCo control problems | Robustness improvement vs SAC (%): +11.7\pm0.8% | Helped |
| Use an intermediate KL constraint value rather than too-high or too-low values (ablation) | QARL · Selected MuJoCo control problems | Robustness | Helped |
Surprising findings
- Force Curriculum (curriculum over adversary force budget) reduces robustness below SAC (-3.3%) despite a moderate performance gain relative to other baselines.
- Non-curriculum methods (RARL, MixedNE-LD) perform notably poorly on the sparse-reward Cartpole Swingup task.
Limitations
- Curriculum hyperparameters (KL constraint epsilon, performance threshold xi) require per-environment tuning
- Adversary force budgets and action spaces are environment-specific and chosen by hand
- Theoretical QRE connection relies on approximating integrals via SAC, so equilibrium guarantees are only approximate
Experimental setup
Datasets: DeepMind Control Suite (MuJoCo)
Benchmarks: DeepMind Control Suite MuJoCo tasks (15 problems) Quadruped maze navigation
Metrics: average return performance percentage improvement vs SAC robustness percentage improvement vs SAC return under varying mass/friction/force
Baselines: RARL SAC MixedNE-LD Curriculum Adversarial Training (CAT) Force Curriculum SAC
Builds on: RARL Soft Actor-Critic Curriculum Adversarial Training MixedNE-LD
Closest papers by meaning
- Robust and Diverse Multi-Agent Learning via Rational Policy Gradient (NeurIPS 2025 · similarity 0.90)
- Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error (ICML 2024 · similarity 0.89)
- Learning Equilibria in Adversarial Team Markov Games: A Nonconvex-Hidden-Concave Min-Max Optimization Problem (NeurIPS 2024 · similarity 0.89)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policies (ICLR 2024 · similarity 0.88)
- Enhancing Robustness in Deep Reinforcement Learning: A Lyapunov Exponent Approach (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.