Research map / Reinforcement learning / Offline RL
Habitizing Diffusion Planning for Efficient and Effective Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Habi, which distills slow diffusion planners into a lightweight habitual policy via a variational (ELBO-style) prior-posterior framework with a critic, achieving 800+ Hz CPU decision frequency on D4RL while matching planner performance.
Problem
Diffusion planners achieve strong offline RL performance but their inference is too slow (often 0.1 to over 1 second per decision) for real-time decision making.
Approach
Habitization: a prior encoder (habitual policy, state only) and a posterior encoder (goal-directed, state plus planner action) are aligned by KL divergence in a latent space trained with an ELBO-like objective, plus a critic that selects among sampled candidate actions at inference, so the habitual prior can run without planning.
Key results
- HI achieves an average performance of 89.2 on MuJoCo, 82.3 on Kitchen, 80.3 on AntMaze and 164.5 on Maze2D, comparable to or above its DV planner teacher.
- HI runs at 1329.7 Hz on a laptop CPU for MuJoCo and 908.3 Hz for AntMaze, versus 7.5 Hz (DV) and 0.7 Hz (DV) on the same CPU.
- Without the critic (HI w/o Critic) and with direct distillation, performance drops by 2.0% and 4.9% on MuJoCo respectively, and by 18.5% and 29.4% on AntMaze.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| habitize the diffusion planner into HI | D4RL MuJoCo locomotion · MuJoCo locomotion | performance: 90.5 -> 89.2 | Mixed |
| habitize the diffusion planner into HI | D4RL Kitchen · Kitchen | performance: 83.8 -> 82.3 | Hurt |
| habitize the diffusion planner into HI | D4RL AntMaze · AntMaze | performance: 83.2 -> 80.3 | Hurt |
| habitize the diffusion planner into HI | D4RL Maze2D · Maze2D | performance: 163.6 -> 164.5 | Helped |
| replace the diffusion planner with HI | D4RL MuJoCo on CPU · MuJoCo | action frequency (CPU, Hz): 197.2 -> 1329.7 | Helped |
| replace the diffusion planner with HI | D4RL Kitchen on CPU · Kitchen | action frequency (CPU, Hz): 146.0 -> 385.7 | Helped |
| replace the diffusion planner with HI | D4RL AntMaze on CPU · AntMaze | action frequency (CPU, Hz): 198.6 -> 908.3 | Helped |
| replace the diffusion planner with HI | D4RL Maze2D on CPU · Maze2D | action frequency (CPU, Hz): 215.8 -> 1532.6 | Helped |
| add critic-based selection over multiple sampled candidates (ablation) | HI · MuJoCo locomotion | performance: 87.5 -> 89.2 | Helped |
| add critic-based selection over multiple sampled candidates (ablation) | HI · Kitchen | performance: 80.4 -> 82.3 | Helped |
| add critic-based selection over multiple sampled candidates (ablation) | HI · AntMaze | performance: 65.4 -> 80.3 | Helped |
| add critic-based selection over multiple sampled candidates (ablation) | HI · Maze2D | performance: 161.3 -> 164.5 | Mixed |
Surprising findings
- HI outperforms its diffusion planner teacher (DV) on several tasks, suggesting habitization may mitigate planning errors.
- Using a single candidate (N=1) already gives competitive results on several environments.
Limitations
- Evaluated only on state-based offline RL tasks; vision-based and online RL not studied.
- Performance drops on some tasks compared with the teacher (e.g., Hopper Medium-Expert 92.4 vs 111.8 for DD), and HI is not uniformly better.
- Relies on a pretrained diffusion planner and its Q-function as teacher, so its quality is bounded by the teacher.
Experimental setup
Datasets: D4RL
Benchmarks: D4RL MuJoCo D4RL AntMaze D4RL Franka Kitchen D4RL Maze2D
Metrics: normalized return decision frequency (Hz)
Baselines: BC SRPO IDQL DQL Diffuser AdaptDiffuser Decision Diffuser Diffusion Veteran DiffuserLite DTQL
Builds on: Diffuser Decision Diffuser Diffusion Veteran CleanDiffuser Bayesian behavior framework
Closest papers by meaning
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- What Makes a Good Diffusion Planner for Decision Making? (ICLR 2025 · similarity 0.89)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.88)
- Simple Hierarchical Planning with Diffusion (ICLR 2024 · similarity 0.87)
- Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner (ICML 2025 · similarity 0.87)
- Learning to Reach Goals via Diffusion (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.