Research map / Image generation / Autoregressive and latent generation
SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SkillDiffuser, a hierarchical planner that learns discrete VQ skill abstractions from images and language and uses them to condition a classifier-free state diffusion model plus an inverse dynamics MLP.
Problem
Diffusion planners struggle to generate coherent trajectories for long-horizon, multi-step instructions, and prior skill-conditioned diffusers rely on predefined skill libraries that do not scale or learn end-to-end.
Approach
Learn a discrete skill set end-to-end via a horizon-based skill predictor with vector quantization over visual and language embeddings, then condition a classifier-free state diffusion U-Net on the learned skill embeddings, decoding actions with a task-specific inverse dynamics model.
Key results
- On LOReL Sawyer, SkillDiffuser achieves 43% average task success vs. 40% for LISA and 13% for Lang DT.
- On LOReL multi-step compositional tasks, SkillDiffuser reaches 25.21% success vs. 18.18% for LOReL planner and 20.89% for LISA.
- On Meta-World MT10 ablation, SkillDiffuser attains 23.3% vs. 16.7% for language-conditioned diffuser and 13.8% for LISA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SkillDiffuser instead of LISA | LOReL Sawyer task-wise evaluation · LOReL Sawyer, task-wise | success rate: 40% -> 43 ± 1.1% | Helped |
| use SkillDiffuser instead of LISA | LOReL Sawyer rephrasal evaluation · LOReL Sawyer, average rephrasals | success rate: 30.14 -> 39.71 | Helped |
| use SkillDiffuser instead of LISA | LOReL multi-step composition tasks · LOReL multi-step composition tasks | success rate: 20.89 ±0.6 -> 25.21 ±2.7 | Helped |
| add a discrete skill set to language-conditioned diffusion planning (ablation) | SkillDiffuser on Meta-World MT10 · Meta-World MT10 | performance: 16.7% -> 23.3% | Helped |
| add language conditioning and skill sets instead of using Flat R3M (ablation) | Meta-World MT10 planner · Meta-World MT10 | performance: 13.3% -> 23.3% | Helped |
| use a diffusion model instead of LISA’s planning model (ablation) | Meta-World MT10 planner · Meta-World MT10 | performance: 13.8% -> 23.3% | Helped |
| add skill abstraction compared with the non-hierarchical Lang-DT baseline (ablation) | SkillDiffuser · LOReL multi-step composition tasks | success rate: 13.33 ±1.3 -> 25.21 ±2.7 | Helped |
Surprising findings
- Flat R3M without language or skill conditioning succeeds only on simple tasks such as drawer-close and reach.
Limitations
- Inverse dynamics model must be retrained per task's kinematics when transferring to new tasks.
- Absolute success rates remain low on several Meta-World tasks (e.g., 10% on move white mug down).
Experimental setup
Datasets: LOReL Sawyer Meta-World MT10
Benchmarks: LOReL Sawyer Meta-World MT10
Metrics: success rate
Baselines: Random policy LCBC LCRL Lang DT LISA Flat R3M LOReL planner Lang Diffuser
Builds on: Diffuser Decision Diffuser LISA VQ-VAE classifier-free diffusion guidance
Closest papers by meaning
- Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning (ICML 2025 · similarity 0.88)
- Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts (CVPR 2024 · similarity 0.88)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.87)
- LLM-based Skill Diffusion for Zero-shot Policy Adaptation (NeurIPS 2024 · similarity 0.87)
- Generative Trajectory Stitching through Diffusion Composition (NeurIPS 2025 · similarity 0.87)
- Language Guided Skill Discovery (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.