Research map / Reinforcement learning / Policy gradient and actor-critic
Adversarial Diffusion for Robust Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AD-RRL, which uses CVaR-guided adversarial diffusion over trajectories inside a PolyGRAD-style Dyna loop to train A2C policies robust to physics parameter shifts on MuJoCo.
Problem
Robustness of RL policies to modeling errors and dynamics mismatch (Sim2Real gap, compounding errors) in model-based RL, targeting worst-alpha-percentile return.
Approach
Guide a trajectory-level diffusion model with a learned cumulative-reward function so that it samples low-return trajectories in the CVaR worst-alpha tail; the guidance coefficients c_i are chosen so the implied perturbation stays inside the CVaR risk envelope, and the policy is trained on these adversarial synthetic trajectories with A2C.
Key results
- On the nominal training environments, AD-RRL attains the best mean return on four of five MuJoCo tasks (e.g., Cheetah 4126.11 ± 246.96 vs PolyGRAD 3879.16 ± 626.40; Walker 4357.80 ± 187.37 vs 3489.48 ± 456.70).
- Only on Hopper does PolyGRAD score higher (3346.99 vs AD-RRL 3280.23), within overlapping confidence intervals.
- Across varied physics parameters, AD-RRL is reported to consistently outperform DR-PPO, M2TD3, and PolyGRAD, though the paper's claims rest on plots.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AD-RRL instead of PolyGRAD | MuJoCo control policies · Hopper nominal | return: 3280.23 \pm 13.83 vs 3346.99 \pm 52.39 | Hurt |
| Use AD-RRL instead of PolyGRAD | MuJoCo control policies · Cheetah nominal | return: 4126.11 \boldsymbol\pm 246.96 vs 3879.16 \pm 626.40 | Helped |
| Use AD-RRL instead of PolyGRAD | MuJoCo control policies · Walker nominal | return: 4357.80 \pm 187.37 vs 3489.48 \pm 456.70 | Helped |
| Use AD-RRL instead of PolyGRAD | MuJoCo control policies · Reacher nominal | return: -3.97 \pm 0.13 vs -4.48 \pm 0.13 | Helped |
| Use AD-RRL instead of PolyGRAD | MuJoCo control policies · InvertedPendulum nominal | return: 1000.00 \pm 0.00 vs 1000.00 \pm 0.00 | No clear effect |
| Train with AD-RRL rather than DR-PPO or M2TD3 | MuJoCo policies under test-time physics changes · MuJoCo varied-physics environments | cumulative return | Helped |
| Use AD-RRL instead of PolyGRAD | MuJoCo policies under perturbed physics · MuJoCo varied-physics environments | return | Helped |
| Use AD-RRL instead of PolyGRAD | MuJoCo policies under nominal physics · MuJoCo nominal environments | return | Mixed |
| Train AD-RRL for 1.5M samples rather than model-free baselines for 3M samples | AD-RRL versus model-free baselines · Proposed MuJoCo tasks | return | No clear effect |
Surprising findings
- PPO and TRPO appear surprisingly stable under parameter perturbations, matching or nearly matching the robust baselines.
Limitations
- Performance degrades for extreme physics parameter changes, hypothesized because generated adversarial trajectories stay plausible under the policy and dynamics.
- Theoretical analysis assumes one-dimensional, bounded trajectories; multidimensional extension is only sketched in the appendix.
- Adds computational overhead from diffusion training and guided sampling.
Experimental setup
Datasets: MuJoCo Hopper MuJoCo HalfCheetah MuJoCo Walker MuJoCo Reacher MuJoCo InvertedPendulum
Benchmarks: MuJoCo Hopper MuJoCo HalfCheetah MuJoCo Walker MuJoCo Reacher MuJoCo InvertedPendulum
Metrics: episode return under perturbed physics parameters final episode return on nominal environment (mean ± standard
Baselines: DR-PPO M2TD3 CVaR-PPO (CPPO) PolyGRAD TRPO PPO
Builds on: PolyGRAD Diffuser CVaR-PPO Dyna classifier guidance
Closest papers by meaning
- Robust Reinforcement Learning via Adversarial training with Langevin Dynamics (NeurIPS 2020 · similarity 0.91)
- Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning (ICML 2021 · similarity 0.91)
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.91)
- Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption (ICML 2024 · similarity 0.90)
- Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.89)
- Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.