Research map / Reinforcement learning / Visual and representation RL
Probabilistic Subgoal Representations for Hierarchical Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HLPS, a goal-conditioned hierarchical RL method that learns probabilistic (Gaussian process) subgoal representations with a learnable Matérn kernel and a softplus ratio-based objective, plus an efficient state-space GP (Kalman filter) for online subgoal inference.
Problem
Deterministic subgoal representations in goal-conditioned HRL struggle with stochasticity and novel/unexplored states, hurting exploration, stationarity, and performance.
Approach
Model the state-to-subgoal mapping as a GP posterior with a learnable Matérn kernel over states, trained jointly with the hierarchical policies via a softplus ratio objective, and perform online subgoal inference with a state-space (Kalman filter) GP whose cost is constant per step.
Key results
- In stochastic Ant Fall with sparse reward, HLPS reaches 0.79±0.01 success vs 0.54±0.02 for LESSON and 0.29±0.05 for HESS.
- On 7-DOF Pusher with sparse reward, HLPS achieves 0.54±0.11 vs 0.21±0.10 for HESS and 0.01±0.01 for LESSON.
- In deterministic environments HLPS scores 0.96 on Ant Maze and 0.74 on Ant Fall, above HESS (0.91, 0.56) and LESSON (0.89, 0.54).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use probabilistic GP subgoal representations and the proposed learning objective instead of deterministic subgoal learning | HLPS · Ant Fall and other stochastic MuJoCo tasks | success rate | Helped |
| Add the probabilistic subgoal formulation to the baseline without it (ablation) | HLPS-BL-A · Ablation environments | asymptotic performance | Helped |
| Add the proposed learning objective to the probabilistic-subgoal variant (ablation) | HLPS-BL-B · Ablation environments | asymptotic performance | Helped |
| Increase the batch-estimation state time window size T (ablation) | HLPS · Ablation environments, early training | performance | Helped |
| Increase the batch-estimation state time window size T (ablation) | HLPS · Ablation environments, later training | performance | No clear effect |
| Transfer the subgoal representation and low-level policy from Ant Fall | HLPS on Ant Push · Ant Fall → Ant Push | sample efficiency | Helped |
| Transfer the subgoal representation and low-level policy from Ant Fall (Image) | HLPS on Ant Push (Image) · Ant Fall (Image) → Ant Push (Image) | performance | Helped |
| Use probabilistic rather than deterministic subgoal representations | HLPS compared with HESS and LESSON · Increasing Gaussian noise (σ values 0, 0.1, 0.15) | success rate degradation | Helped |
Surprising findings
- Flat TD3 achieves essentially zero success on all Ant tasks, showing hierarchy is required in these settings.
- HRAC, with a pre-defined subgoal space, fails completely on Ant Push and Ant Push with image observations.
Limitations
- Batch GP inference scales cubically with the number of states, requiring the online approximation or small time windows (T=3).
- Stochasticity is simulated only as Gaussian noise on agent (x,y) position; evaluation is restricted to MuJoCo navigation/manipulation tasks.
Experimental setup
Datasets: Ant Maze Ant Push Ant Fall Ant FourRooms 7-DOF Reacher 7-DOF Pusher
Benchmarks: Ant Maze Ant Push Ant Fall Ant FourRooms 7-DOF Reacher 7-DOF Pusher
Metrics: success rate final performance averaged over 10 seeds with standard error
Baselines: LESSON HESS HRAC TD3 HIRO Oracle
Builds on: SAC HIRO LESSON HESS HRAC
Closest papers by meaning
- Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals (ICML 2025 · similarity 0.93)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Hierarchical Reinforcement Learning with Targeted Causal Interventions (ICML 2025 · similarity 0.88)
- Improving Unsupervised Hierarchical Representation with Reinforcement Learning (CVPR 2024 · similarity 0.88)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.88)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.