Research map / Reinforcement learning / Offline RL
Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIHD, a hierarchical diffusion planner whose layers are built adaptively from structural-entropy community partitioning of offline states, with structural information gain as conditioning and a structural entropy regularizer for exploration.
Problem
Existing hierarchical diffusion methods for offline RL use a fixed two-layer hierarchy with a single predefined temporal scale, which limits adaptability in long-horizon, sparse-reward tasks and leads to overreliance on offline data.
Approach
Construct a multi-layer diffusion hierarchy by minimizing k-dimensional structural entropy over a kNN state similarity graph, segment trajectories per community layer, condition each diffuser on the structural information gain of its community, and add a structural entropy regularizer computed from diffusion-estimated transition probabilities to encourage exploration of underrepresented states.
Key results
- SIHD achieves the highest reward on all Gym-MuJoCo Medium-Expert, Medium, and Medium-Replay datasets, e.g., 117.7 on Hopper-Expert versus 115.3 for HD.
- On Maze2D single-task U-Maze, SIHD reaches 144.6 vs 128.4 for HD, with average improvements of 8.3% across single-task Maze2D, 7.4% multi-task Maze2D, and 4.4% AntMaze as reported.
- Compared with Diffuser, SIHD reduces training time by 82.5% and planning time by 54.8% on Maze2D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SIHD instead of the strongest listed baseline | Gym-MuJoCo offline RL · Gym-MuJoCo Medium-Expert, Medium, and Medium-Replay datasets | average reward: 1.6%; 3.8%; 3.9% | Helped |
| Use SIHD instead of the strongest listed baseline | SIHD · Single-task Maze2D | average reward: 8.3% | Helped |
| Use SIHD instead of the strongest listed baseline | SIHD · Multi-task Maze2D | average reward: 7.4% | Helped |
| Use SIHD instead of the strongest listed baseline | SIHD · AntMaze | average reward: 4.4% | Helped |
| Replace the adaptive multi-scale hierarchy with a rigid two-layer single-scale diffuser (ablation) | SIHD · Hopper, Walker2D, Maze2D, and AntMaze | average reward | Hurt |
| Remove classifier-based guidance (ablation) | SIHD · Hopper, Walker2D, Maze2D, and AntMaze | average reward | Hurt |
| Remove the structural entropy regularizer (ablation) | SIHD · Hopper, Walker2D, Maze2D, and AntMaze | average reward | Hurt |
| Increase maximum tree height | SIHD · AntMaze | average reward | Mixed |
| Increase regularization weight η | SIHD · HalfCheetah Expert | average reward | Hurt |
| Increase regularization weight η | SIHD · HalfCheetah Medium | average reward | Helped |
| Use SIHD instead of flat Diffuser | Maze2D planning · Maze2D U-Maze, Medium, and Large | planning time: 54.8% | Helped |
Limitations
- Structural entropy-guided community partitioning introduces significant computational overhead on large-scale datasets.
- Subgoal constraints are implemented via a simple ending-state replacement strategy.
- Evaluated only on D4RL locomotion and navigation tasks, not more complex offline RL environments.
Experimental setup
Datasets: D4RL HalfCheetah D4RL Hopper D4RL Walker2D D4RL Maze2D D4RL AntMaze
Benchmarks: D4RL Gym-MuJoCo D4RL Maze2D D4RL AntMaze
Metrics: average normalized reward standard deviation over five seeds wall-clock training and planning time
Baselines: CQL IQL Decision Transformer MoReL Trajectory Transformer Diffuser HDMI Hierarchical Diffuser MPPI IRIS HiGoC
Builds on: Diffuser Hierarchical Diffuser HDMI HCSE Decision Transformer
Closest papers by meaning
- Simple Hierarchical Planning with Diffusion (ICLR 2024 · similarity 0.90)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.89)
- State-Covering Trajectory Stitching for Diffusion Planners (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.