Research map / Reinforcement learning / Imitation learning
Diffusion Imitation from Observation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIFO, an adversarial learning-from-observation method that uses a conditional diffusion model as the discriminator, turning its denoising losses into "realness" rewards for RL policy training.
Problem
Learning from observation (state-only demonstrations, no action labels) with adversarial imitation is brittle and sensitive to hyperparameters, and existing LfO methods perform poorly with high-dimensional or highly variable expert trajectories.
Approach
Train a diffusion model conditioned on the current state and an expert/agent label to predict the next state; reformulate its expert and agent denoising losses as a binary discriminator whose score (via a single denoising step) gives GAIL-style rewards to the policy.
Key results
- In AntMaze with 50 expert trajectories DIFO drops modestly from about 80% to 70% success, while the best baseline WAIfO falls to about 20%.
- With only 25 expert trajectories in AntMaze, all baselines are near zero success while DIFO retains about 20%.
- DIFO matches or outperforms the best baseline across all eight tasks, and converges faster in AntMaze, AdroitDoor, and CarRacing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reduce expert demonstrations from 200 to 50 trajectories | DIFO on AntMaze · AntMaze | success rate: 80% -> 70% | Hurt |
| Use DIFO instead of WAILfO with 50 expert trajectories | AntMaze imitation · AntMaze (50 expert trajectories) | success rate: 20% to 70% | Helped |
| Reduce expert demonstrations to 25 trajectories | DIFO on AntMaze · AntMaze | success rate: around 20% | Hurt |
| Replace the full method with DIFO-NA, using a pretrained conditional diffusion model and denoising reward without adversarial online-data training (ablation) | DIFO · Most evaluated tasks | learning performance | Hurt |
| Remove conditioning on the current state and denoise both states (ablation) | DIFO · Multiple tasks | learning performance | Mixed |
| Remove the expert-data MSE loss (ablation) | DIFO · PointMaze and Walker | performance | Hurt |
| Remove the BCE classification loss (ablation) | DIFO · PointMaze and Walker | learning performance | Hurt |
| Train the MSE loss using agent data (ablation) | DIFO · AdroitDoor | success rate: 0% | Hurt |
| Vary the number of denoising-step samples used to compute rewards (ablation) | DIFO · PointMaze and Walker | performance | No clear effect |
| Vary the ratio of the MSE and BCE loss weights (ablation) | DIFO · PointMaze and Walker | performance | No clear effect |
Surprising findings
- Optimizing the diffusion denoising MSE on agent data leads to unstable training and a 0% success rate on AdroitDoor, so the method uses expert data only for that term.
- Optimal Transport only learns in AdroitDoor, Walker, and CloseDrawer, where trajectory variety is limited.
Limitations
- Uses state-only transitions (s, s') as rewards, which may be suboptimal in tasks where actions have delayed effects.
- Assumes agent and expert share the same state space, limiting cross-embodiment transfer.
- Relies heavily on expert demonstrations; sub-optimal demonstrations may degrade performance.
Experimental setup
Datasets: D4RL PointMaze expert data Minari AntMaze expert data D4RL AdroitDoor demonstrations D4RL FrankaKitchen (OpenMicrowave) demonstrations Gymnasium environments with self-trained SAC/PPO experts Meta-World CloseDrawer scripted expert
Benchmarks: PointMaze AntMaze FetchPush AdroitDoor Walker OpenMicrowave CarRacing CloseDrawer Sine (toy)
Metrics: success rate episode return standard deviation over seeds sample efficiency vs number of demonstrations
Baselines: BC BCO GAIfO WAIfO AIRLfO DePO IQ-Learn OT DIFO-NA DIFO-Uncond
Builds on: GAIL DDPM GAIfO AIRL Wasserstein adversarial imitation
Closest papers by meaning
- Diffusing States and Matching Scores: A New Framework for Imitation Learning (ICLR 2025 · similarity 0.91)
- DiffAIL: Diffusion Adversarial Imitation Learning (AAAI 2024 · similarity 0.91)
- Diffusion-Reward Adversarial Imitation Learning (NeurIPS 2024 · similarity 0.91)
- Imitation from Observations with Trajectory-Level Generative Embeddings (NeurIPS 2026 · similarity 0.91)
- Diffusion Model-Augmented Behavioral Cloning (ICML 2024 · similarity 0.90)
- Latent Diffusion Planning for Imitation Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.