Research map / Reinforcement learning / Imitation learning
DiffAIL: Diffusion Adversarial Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiffAIL, which replaces the binary discriminator in adversarial imitation learning with an unconditional diffusion-loss-based discriminator over state-action (or state, next-state) pairs, with SAC as forward RL.
Problem
Adversarial imitation learning uses simple binary discriminators that do not capture the expert distribution well, so they fail to identify expert-level state-action pairs generated by the policy and generalize poorly.
Approach
Model state-action pairs with an unconditional diffusion model and use the exponentiated negative diffusion (noise-prediction) loss as the discriminator output, with the surrogate reward averaged over a few diffusion timesteps; the diffusion model is used only as a distribution-matching loss, not for sampling.
Key results
- With 1 expert trajectory, DiffAIL reaches best return of 5362.25 on HalfCheetah and 5142.60 on Ant, above expert returns of 4463 and 4228, and outperforms GAIL, ValueDICE and CFIL in the table.
- On held-out expert state-action pairs (4 training trajectories), DiffAIL classifies 92.8%-98.8% as expert versus 81.7%-90.5% for GAIL across the four tasks.
- Pearson correlation between surrogate reward and true return for DiffAIL is 97.1%-99.6% across tasks, versus GAIL values including -89.0% on one task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the binary GAIL discriminator with a diffusion-loss discriminator | GAIL · Hopper, 1 trajectory, state-action | average episodic return: 3234.61 ± 266.62 -> 3382.03 ± 142.86 | Helped |
| Replace the binary GAIL discriminator with a diffusion-loss discriminator | GAIL · HalfCheetah, 1 trajectory, state-action | average episodic return: 4976.73 ± 60.41 -> 5362.25 ± 96.92 | Helped |
| Replace the binary GAIL discriminator with a diffusion-loss discriminator | GAIL · Ant, 1 trajectory, state-action | average episodic return: 4530.43 ± 133.30 -> 5142.60 ± 90.05 | Helped |
| Replace the binary GAIL discriminator with a diffusion-loss discriminator | GAIL · Walker2d, 1 trajectory, state-action | average episodic return: 5039.77 ± 307.17 -> 6292.28 ± 97.65 | Helped |
| Replace the binary GAIL discriminator with a diffusion-loss discriminator | GAIL · Hopper, held-out expert trajectories | expert discrimination rate: 83.3% ± 1.2% -> 92.8% ± 2.5% | Helped |
| Replace the binary GAIL discriminator with a diffusion-loss discriminator | GAIL · HalfCheetah, held-out expert trajectories | expert discrimination rate: 83.8% ± 0.9% -> 94.5% ± 0.6% | Helped |
| Replace the binary GAIL discriminator with a diffusion-loss discriminator | GAIL · Walker2d, held-out expert trajectories | expert discrimination rate: 90.5% ± 0.8% -> 96.1% ± 0.6% | Helped |
| Replace the binary GAIL discriminator with a diffusion-loss discriminator | GAIL · Ant, held-out expert trajectories | expert discrimination rate: 81.7% ± 1.6% -> 98.8% ± 0.1% | Helped |
| Use diffusion surrogate rewards instead of GAIL surrogate rewards | GAIL · Hopper, HalfCheetah, Walker2d, Ant | Pearson correlation with actual return: [ 97.0%, 54.1%, -89.0%, 92.1%] -> [ 97.1%, 98.8%, 99.1%, 99.6%] | Mixed |
| Increase diffusion discriminator steps from 2 to 10 (ablation) | DiffAIL · MuJoCo tasks | training stability and convergence speed | Helped |
| Increase diffusion discriminator steps from 10 to 20 (ablation) | DiffAIL · MuJoCo tasks | training time: 6.2h -> 8.0h; 6.4 -> 8.2h; 7.5h -> 9.3h; 9.5h -> 11.4h | Hurt |
Surprising findings
- GAIL's surrogate return correlates negatively (-89.0%) with true return on one task, while DiffAIL stays above 97%.
- Only about 10 diffusion steps were sufficient in MuJoCo, unlike image diffusion needing thousands of steps.
Limitations
- Computing the diffusion noise loss requires multi-step noise sampling, increasing computational cost versus binary discriminator.
- Stability issues: in the single-trajectory setting the policy does not maintain optimality in later training stages.
- Less time-efficient than ValueDICE's optimized implementation.
Experimental setup
Datasets: MuJoCo expert trajectories (Hopper, HalfCheetah, Walker2d, Ant)
Benchmarks: MuJoCo Hopper MuJoCo HalfCheetah MuJoCo Walker2d MuJoCo Ant
Metrics: average episodic return best return during training discriminator classification rate on held-out expert traject Pearson correlation between surrogate return and actual retu training time
Baselines: Behavior cloning GAIL ValueDICE CFIL OPOLO
Builds on: GAIL DAC CFIL DDPM Diffusion-QL
Closest papers by meaning
- Diffusion Imitation from Observation (NeurIPS 2024 · similarity 0.91)
- Diffusion-Reward Adversarial Imitation Learning (NeurIPS 2024 · similarity 0.91)
- Diffusing States and Matching Scores: A New Framework for Imitation Learning (ICLR 2025 · similarity 0.90)
- Adversarial Imitation Learning via Boosting (ICLR 2024 · similarity 0.88)
- Imitation Learning from Purified Demonstrations (ICML 2024 · similarity 0.87)
- Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.