Research map / Reinforcement learning / Policy gradient and actor-critic
Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces temporally-coupled perturbations for robust RL and proposes GRAD, a PSRO-based game-theoretic training method that treats robust RL as a partially-observable two-player zero-sum game to defend against both temporally-coupled and standard attacks on MuJoCo control tasks.
Problem
Existing robust RL methods assume adversarial perturbations are independent across time under a static L_p constraint, which is unrealistic for real-world perturbations that are temporally correlated (e.g., wind), leading to overly conservative policies and poor defense against temporally-coupled at
Approach
Define temporally-coupled attack constraints that bound the difference between consecutive perturbations, and train the agent against an adaptive adversary via a PSRO-style double-oracle game-theoretic procedure (GRAD) so the agent approaches an approximate Nash equilibrium against the learned adversary policy set.
Key results
- Under temporally-coupled state attacks, GRAD's average performance is reported as 45% higher than the strongest baseline.
- On Humanoid, GRAD outperforms baselines by 17% under standard action attacks and by 40% under temporally-coupled action attacks.
- Under memorized temporally-coupled attacks, GRAD achieves e.g. 2736 on Humanoid vs 2017 for WocaR-PPO and 1573 for PA-ATLA-PPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with a temporally-coupled PA-AD adversary using GRAD | GRAD · MuJoCo state perturbations | episode reward: 45% higher | Helped |
| Train with a temporally-coupled action adversary using GRAD | GRAD · Humanoid, standard action perturbations | episode reward: 17% | Helped |
| Train with a temporally-coupled action adversary using GRAD | GRAD · Humanoid, temporally-coupled action perturbations | episode reward: 40% | Helped |
| Train with a temporally-coupled mixed attacker using GRAD | GRAD · Humanoid, non-temporally-coupled mixed attacks | episode reward: over 20% | Helped |
| Train with a temporally-coupled mixed attacker using GRAD | GRAD · Multiple MuJoCo environments, temporally-coupled mixed attacks | episode reward: 30% | Helped |
| Increase the temporally-coupled constraint value ε̄ (ablation) | GRAD · State attacks | episode reward | No clear effect |
| Set ε̄ close to zero (ablation) | Temporally-coupled attacker · State attacks | attack performance | Hurt |
| Increase ε̄ so the temporal-coupling constraint weakens (ablation) | WocaR-PPO · Temporally-coupled state attacks | agent episode reward | Helped |
| Use ε̄ = ε/5 or ε̄ = ε/10 rather than the other tested choice (ablation) | GRAD · MuJoCo robust models | final performance: does not differ by more than 5% | No clear effect |
| Train with a short-term memorized temporally-coupled attacker | GRAD · Hopper, Walker2d, HalfCheetah, Ant, and Humanoid | episode reward | Helped |
| Increase the attack budget ε | Robust agents · MuJoCo attack-budget evaluations | episode reward | Hurt |
| Use GRAD rather than the RNAC-PPO baselines | Transition-robust RL · Perturbed MuJoCo environments | episode reward | Mixed |
Surprising findings
- Restricting the adversary to a temporally-coupled search space can make attacks stronger rather than weaker, since the optimal attack may lie within the reduced set.
Limitations
- PSRO-based approach may require many iterations to converge, making training costly.
- Scalability to pixel-based or higher-complexity real-world settings not yet demonstrated.
- Theoretical convergence result relies on finite-horizon MDP with discrete actions, while experiments use continuous control.
Experimental setup
Datasets: MuJoCo Hopper-v2 MuJoCo Walker2d-v2 MuJoCo Halfcheetah-v2 MuJoCo Ant-v2 MuJoCo Humanoid-v2
Benchmarks: MuJoCo Hopper MuJoCo Walker2d MuJoCo Halfcheetah MuJoCo Ant MuJoCo Humanoid
Metrics: average episodic reward standard deviation over 100 episodes
Baselines: PA-ATLA-PPO WocaR-PPO ATLA PR-MDP NR-MDP Mixed-ATLA RNAC-PPO
Builds on: Policy Space Response Oracles Double Oracle PPO ATLA PA-AD
Closest papers by meaning
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.89)
- Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.88)
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertainty (ICML 2024 · similarity 0.88)
- Byzantine Robust Cooperative Multi-Agent Reinforcement Learning as a Bayesian Game (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.