Research map / Image generation / GANs
Diffusing States and Matching Scores: A New Framework for Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMILING, an imitation learning framework that replaces adversarial discriminator training with regression-based score matching on diffused expert and learner state distributions, with theoretical instance-dependent bounds.
Problem
Adversarial imitation learning (GAN-style discriminators) is unstable to train, and offline diffusion-based behavioral cloning suffers from compounding errors; the paper asks how to lift diffusion score matching into inverse RL using only expert states.
Approach
Define the Diffusion Score Divergence, a sequential Fisher-like divergence comparing diffusion score functions of expert and learner state distributions; minimize it via a min-max game where the learner's cost is the difference of squared noise-prediction errors of an expert score network and a learner score network, optimized by FTL score matching plus RL (SAC or DreamerV3).
Key results
- SMILING matches expert performance and outperforms all baselines on five of six state-only tasks across five seeds (3M steps).
- DiffAIL collapsed numerically at the start of training on all four humanoid tasks, so its results were not reported.
- On humanoid-walk, SMILING approaches expert performance with 5K expert states while DAC requires 25K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use score-matching imitation learning instead of the tested baselines | SMILING · Continuous control, state-only demonstrations (six tasks) | normalized return | Helped |
| use SMILING instead of DAC | SMILING and DAC · cheetah-run | normalized return | Mixed |
| use SMILING instead of BC | SMILING and BC · Continuous control, state-only demonstrations | normalized return | Helped |
| use SMILING instead of DAC | SMILING and DAC · humanoid-walk | final performance | Helped |
| increase expert demonstrations from 5K to 25K | DAC · humanoid-walk | final performance | Helped |
| remove activation functions, making score functions and discriminators purely linear (ablation) | SMILING and DAC · cheetah-run | performance | Mixed |
| train on state-action demonstrations instead of state-only demonstrations | SMILING · Continuous control tasks | normalized return | No clear effect |
Surprising findings
- DiffAIL diverged (weights to infinity) on all four humanoid tasks, preventing comparison.
- Even with linear score functions, SMILING still learns well on cheetah-run while DAC degrades notably.
Limitations
- Discretization error and Lipschitz/second-moment constants enter the bounds when discretization is considered; theory assumes score function class realizability.
- Experiments limited to continuous control in simulation with state-only or state-action demonstrations.
- Baseline comparison to BC uses state-action data, so not directly comparable.
Experimental setup
Datasets: DeepMind Control Suite tasks HumanoidBench tasks
Benchmarks: DeepMind Control Suite (ball-in-cup-catch, cheetah-run) HumanoidBench (humanoid-walk, humanoid-sit, humanoid-crawl, humanoid-pole)
Metrics: normalized return (expert=1, random=0) final performance vs number of expert demonstrations
Baselines: DAC IQ-Learn DiffAIL Behavior Cloning BCO
Builds on: GAIL DAC SQIL DAgger DDPM
Closest papers by meaning
- Diffusion Imitation from Observation (NeurIPS 2024 · similarity 0.91)
- DPAIL: Training Diffusion Policy for Adversarial Imitation Learning without Policy Optimization (NeurIPS 2025 · similarity 0.91)
- DiffAIL: Diffusion Adversarial Imitation Learning (AAAI 2024 · similarity 0.90)
- Diffusion-Reward Adversarial Imitation Learning (NeurIPS 2024 · similarity 0.89)
- Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective (AAAI 2024 · similarity 0.89)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.