Research map / Reinforcement learning / Imitation learning
Diffusion-Reward Adversarial Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRAIL, a generative adversarial imitation learning method whose discriminator is a conditional diffusion model turned into a binary classifier via diffusion losses, providing bounded diffusion rewards for PPO policy learning.
Problem
GAIL training is brittle and unstable, and its discriminator-derived rewards can overfit to expert demonstrations and generalize poorly to unseen states.
Approach
Build the GAIL discriminator from a conditional diffusion model conditioned on real/fake labels: the sigmoid of the difference between the expert-conditioned and agent-conditioned diffusion losses gives a bounded "realness" probability, trained with BCE and used as a reward via log D - log(1-D), with only one denoising step needed.
Key results
- On the main tasks, DRAIL reaches 96.83% success on Maze, 95.78% on FetchPush, and 5381 return on Walker, matching or beating the best baseline in most tasks.
- At 2.00x noise on FetchPush, DRAIL achieves 99.09% success versus 79.20% for Diffusion Policy, the best baseline there.
- With a single expert trajectory on Walker, DRAIL retains a return of 4960.10, while DiffAIL drops to 3057.35.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DRAIL instead of DiffAIL | Adversarial imitation learning · Maze | success rate: 94.06%\pm1.67% -> 96.83%\pm0.49% | Helped |
| Use DRAIL instead of DiffAIL | Adversarial imitation learning · FetchPush | success rate: 93.66%\pm3.15% -> 95.78%\pm0.80% | Helped |
| Use DRAIL instead of DiffAIL | Adversarial imitation learning · HandRotate | success rate: 97.49%\pm0.48% -> 93.88%\pm3.23% | Hurt |
| Use DRAIL instead of WAIL | Adversarial imitation learning · AntReach | success rate: 33.84%\pm2.63% -> 39.02%\pm1.40% | Helped |
| Use DRAIL instead of BC | Imitation learning · Walker | return: 5431.19\pm267.40 -> 5381.15\pm115.25 | Hurt |
| Use DRAIL instead of DiffAIL | Adversarial imitation learning · CarRacing | return: 689.62\pm96.72 -> 746.35\pm31.28 | Helped |
| Use DRAIL instead of Diffusion Policy | Imitation learning · FetchPush 2.00× noise | success rate: 79.20%\pm3.76% -> 99.09%\pm0.47% | Helped |
| Use DRAIL instead of Diffusion Policy | Imitation learning · FetchPick 2.00× noise | success rate: 76.64% -> 87.22% | Helped |
| Use DRAIL instead of BC | Imitation learning · FetchPush, 2000 expert state-action pairs | success rate: 80.20%\pm14.70% -> 86.44%\pm13.44% | Helped |
| Use DRAIL instead of DiffAIL | Adversarial imitation learning · FetchPush, 2000 expert state-action pairs | success rate: 46.61%\pm37.37% -> 86.44%\pm13.44% | Helped |
| Use DRAIL instead of BC | Imitation learning · Walker, 1 expert trajectory | return: 3055.27\pm1834.75 -> 4960.10\pm164.57 | Helped |
Surprising findings
- DiffAIL fails on FetchPick at 2.00x noise (0% success), and GAIL is highly unstable across seeds on several tasks.
- BC outperforms all AIL baselines on the Walker locomotion task in the main results.
Limitations
- Results on image-based CarRacing use a small expert dataset (671 transitions) and the paper gives no compute details.
- Reward smoothness and stability are evaluated mainly on a 1-D toy environment and on simulated tasks; real-world robotics is left to future work.
Experimental setup
Datasets: maze2d-medium-v2 expert demonstrations FetchPush expert demonstrations HandRotate expert demonstrations AntReach expert demonstrations Walker PPO expert trajectories CarRacing PPO expert demonstrations
Benchmarks: Maze FetchPush HandRotate AntReach Walker CarRacing FetchPick
Metrics: success rate return
Baselines: Behavioral Cloning Diffusion Policy GAIL GAIL-GP WAIL DiffAIL
Builds on: GAIL DiffAIL Diffusion Classifier DDPM Diffusion Policy
Closest papers by meaning
- Diffusion Imitation from Observation (NeurIPS 2024 · similarity 0.91)
- DiffAIL: Diffusion Adversarial Imitation Learning (AAAI 2024 · similarity 0.91)
- Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective (AAAI 2024 · similarity 0.90)
- C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory (NeurIPS 2024 · similarity 0.89)
- Diffusing States and Matching Scores: A New Framework for Imitation Learning (ICLR 2025 · similarity 0.89)
- Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.