Research map / Image generation / Autoregressive and latent generation
Latent Diffusion Planning for Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Latent Diffusion Planning (LDP), a modular imitation learning method that trains a VAE latent space, a diffusion planner that forecasts latent state sequences (can use action-free data), and a diffusion inverse dynamics model (can use suboptimal data).
Problem
Imitation learning policies that directly model expert actions cannot use the abundant action-free or suboptimal data, and video-planning alternatives are computationally expensive and slow for closed-loop control in visual manipulation.
Approach
Separate planning from action prediction: a diffusion planner forecasts dense sequences of VAE latent states (trainable on action-free demos) and a diffusion inverse dynamics model extracts actions from latent pairs (trainable on suboptimal data), enabling fast closed-loop planning over a compact latent space.
Key results
- LDP + Action-Free + Subopt reaches 0.95 average success across four simulated tasks, versus 0.51 for DP and 0.59 for DP-VPT (Table with action-free data).
- LDP outperforms DP in the real-world Franka Lift task (73.3 vs 69.6 success), and adding action-free data reaches 79.3.
- UniPi-OL and UniPi-CL achieve only 0.12 and 0.17 average success, far below LDP variants, indicating video planning struggles in the low-demo regime.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add action-free data to the planner | LDP · Lift | success rate: 0.69 \Std{0.03} -> 0.67 \Std{0.01} | No clear effect |
| add action-free data to the planner | LDP · Can | success rate: 0.70 \Std{0.02} -> 0.78 \Std{0.04} | Helped |
| add action-free data to the planner | LDP · Square | success rate: 0.46 \Std{0.00} -> 0.47 \Std{0.03} | No clear effect |
| add action-free data to the planner | LDP · ALOHA Cube | success rate: 0.64 \Std{0.04} -> 0.70 \Std{0.02} | Helped |
| add suboptimal data to the inverse dynamics model | LDP · Lift | success rate: 0.69 \Std{0.03} -> 0.84 \Std{0.06} | Helped |
| add action-free and suboptimal data | LDP · simulated tasks | success rate: 0.65 -> 0.95 | Helped |
| add action-free data | LDP · Franka Lift | success rate: 73.3 $\pm$ 7 -> 79.3 $\pm$ 3 | Helped |
| add suboptimal data | LDP · Franka Lift | success rate: 73.3 $\pm$ 7 -> 74.8 $\pm$ 4 | Helped |
| replace dense latent-state forecasting with subgoals 4 steps apart (ablation) | LDP · Lift | success rate: 0.69 \Std{0.03} -> 0.53 \Std{0.03} | Hurt |
| replace dense latent-state forecasting with subgoals 4 steps apart (ablation) | LDP · Can | success rate: 0.70 \Std{0.02} -> 0.62 \Std{0.04} | Hurt |
| replace dense latent-state forecasting with subgoals 4 steps apart (ablation) | LDP · Square | success rate: 0.46 \Std{0.00} -> 0.31 \Std{0.01} | Hurt |
| replace learned VAE embeddings with frozen DINOv2 embeddings (ablation) | LDP · Lift | success rate: 1.00 \Std{0.00} -> 0.00 \Std{0.00} | Hurt |
Surprising findings
- Planning directly over frozen DINOv2 embeddings yields 0% success, while a VAE latent space is crucial.
- Pretrained UniPi checkpoints (AVDC-THOR) do not substantially improve UniPi performance.
Limitations
- VAE latent space may not capture the most useful features for control.
- Diffusing over states incurs more computational overhead than diffusing actions.
- Did not explore recent diffusion improvements, which may matter in large-data regimes.
Experimental setup
Datasets: Robomimic ALOHA Sim Transfer Cube suboptimal failed trajectories from under-trained BC agent real-world Franka Lift teleoperated demos
Benchmarks: Robomimic Lift Robomimic Can Robomimic Square ALOHA Sim Transfer Cube Real-world Franka Lift
Metrics: success rate
Baselines: Diffusion Policy RC-DP DP+Repr DP PT + FT DP-VPT UniPi-OL UniPi-CL
Builds on: Diffusion Policy UniPi IDQL beta-VAE Decision Diffuser
Closest papers by meaning
- Diffusion Model-Augmented Behavioral Cloning (ICML 2024 · similarity 0.90)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.89)
- Diffusion Imitation from Observation (NeurIPS 2024 · similarity 0.89)
- Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution (NeurIPS 2025 · similarity 0.89)
- ET-SEED: EFFICIENT TRAJECTORY-LEVEL SE(3) EQUIVARIANT DIFFUSION POLICY (ICLR 2025 · similarity 0.89)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.