Research map / Reinforcement learning / Visual and representation RL
Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HIDI, a hierarchical RL method where the high-level subgoal policy is a conditional diffusion model regularized by a sparse Gaussian Process prior, with a subgoal selection rule that mixes GP predictive means and diffusion samples.
Problem
Off-policy hierarchical RL suffers from non-stationarity because the low-level policy changes, making it hard for the high-level policy to generate effective, achievable subgoals; existing subgoal generators are sample-inefficient and lack uncertainty quantification.
Approach
Model the state-conditioned subgoal distribution with a conditional diffusion policy trained with diffusion, TD3-style deterministic policy gradient and a sparse GP negative log-likelihood regularizer, then select subgoals with probability epsilon from the GP predictive mean and otherwise from the diffusion policy.</n
Key results
- HIDI consistently surpasses HLPS, SAGA, HIGL, HRAC and HIRO in learning stability, sample efficiency and asymptotic success rate across tasks, averaged over 10 trials.
- Diffusional subgoals give about 15% gain over the baseline; GP regularization adds roughly 15% sample-efficiency and 16% performance gains.
- Subgoal selection adds about 7% and 8% on two challenging tasks; diffusion steps N=5 and selection probability epsilon=0.1 chosen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HIDI's conditional diffusion model, GP prior, and subgoal selection | HIDI · Continuous control benchmarks | performance | Helped |
| Add conditional diffusion subgoal generation (ablation) | Baseline · Continuous control tasks | performance: ~15% | Helped |
| Add GP prior regularization (ablation) | HIDI-B · Continuous control tasks | performance: ~16% | Helped |
| Add GP prior regularization (ablation) | HIDI-B · Continuous control tasks | sample efficiency: ~15% | Helped |
| Add GP-mean subgoal selection (ablation) | HIDI-A · Two challenging tasks | performance: approximately 7% and 8% | Helped |
| Add GP-mean subgoal selection (ablation) | HIDI-A · Continuous control tasks | sample efficiency | Helped |
| Increase diffusion steps from 3 to 7 (ablation) | HIDI · Continuous control tasks | high-level policy expressiveness: from 3 to 7 | Helped |
| Set diffusion steps to N=5 (ablation) | HIDI · All tasks | performance: N=5 | Helped |
| Use large subgoal selection probability ε=0.25 (ablation) | HIDI · Continuous control tasks | performance stability: 0.25 | Hurt |
| Use small subgoal selection probability ε=0.05 (ablation) | HIDI · Continuous control tasks | benefit from subgoal selection: 0.05 | Hurt |
| Increase RL-objective scaling factor η (ablation) | HIDI · Early training (0∼10^{6}) | performance: 0∼10^{6} | Helped |
| Increase RL-objective scaling factor η (ablation) | HIDI · Later training (4.2×10^{6}∼5×10^{6}) | performance: 4.2×10^{6}∼5×10^{6} | No clear effect |
Limitations
- Theoretical guarantees rely on strong assumptions such as a near-optimal diffusion policy and Lipschitz Q-functions.
- Diffusion sampling adds inference cost from multiple denoising steps per subgoal.
- Sensitivity to the subgoal selection probability: large epsilon can cause instability.
Experimental setup
Datasets: MuJoCo simulated environments (Reacher, Pusher, Point Maze, Ant Maze U/W, Ant Fall, Ant FourRooms)
Benchmarks: Reacher Pusher Point Maze Ant Maze (U-shape) Ant Maze (W-shape) Ant Fall Ant FourRooms
Metrics: success rate sample efficiency distance between generated and reached subgoals
Baselines: HLPS SAGA HIGL HRAC HIRO
Builds on: HIRO HIGL HRAC TD3 Denoising diffusion probabilistic models
Closest papers by meaning
- Probabilistic Subgoal Representations for Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.93)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Hierarchical Reinforcement Learning with Targeted Causal Interventions (ICML 2025 · similarity 0.88)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.