Research map / Reinforcement learning / Visual and representation RL
Flattening Hierarchies with Policy Bootstrapping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAW, a flat offline goal-conditioned RL policy extraction objective that bootstraps on dataset in-trajectory subgoal states with advantage-weighted importance sampling, avoiding any generative subgoal model.
Problem
Offline goal-conditioned RL struggles on long-horizon goal-reaching because sparse rewards and discounting obscure action advantages for distant goals; hierarchical methods work but need complex generative subgoal models that scale poorly to high-dimensional spaces.
Approach
Derive a flat policy objective from hierarchical inference: regress the flat goal-conditioned policy toward a subgoal-conditioned subpolicy, with subgoals sampled from dataset future states and weighted by exp(subgoal advantage) via Bayes' rule, so no generative subgoal model is needed. Combined with a one-step AWR term and GCIVL value learning.
Key results
- SAW achieves 35% success on humanoidmaze-giant-navigate versus 12% for the previous best (HIQL), the only method with nontrivial success there.
- SAW reaches 73% on antmaze-giant-navigate, the best across methods in the table.
- On cube-single-play with tuned value hyperparameters (tau=0.9, beta=0.3), SAW reaches 72% versus 32% at tau=0.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAW instead of HIQL | offline GCRL · humanoidmaze-giant-navigate-v0 | success rate (%): 12 ± 4 -> 35 ± 4 | Helped |
| Use SAW instead of HIQL | offline GCRL · antmaze-giant-navigate-v0 | success rate (%): 65 ± 5 -> 73 ± 4 | Helped |
| Use SAW instead of HIQL | offline GCRL · visual manipulation tasks | success rate (%) | No clear effect |
| Use GCWAE flat policy extraction instead of HIQL hierarchy | antmaze-large-navigate · antmaze-large-navigate | success rate (%): 90% -> 75% | Hurt |
| Train SAW target subpolicy on HIQL subgoal representations instead of observations (ablation) | SAW · giant maze environments | success rate | Hurt |
| Set value-learning settings to τ = 0.9 and β = 0.3 instead of τ=0.7 (ablation) | SAW · cube-single-play-v0 | success rate (%): 32% ± 4% -> 72% ± 5 | Helped |
Surprising findings
- Subgoal representations from HIQL's value function harm SAW performance relative to training directly on observations in large state spaces.
- Expectile-based value methods, including SAW, are highly sensitive to value learning hyperparameters on cube-single.
Limitations
- Assumes subgoals sampled from k-step future dataset states, introducing bias relative to the learned policy's own occupancy.
- Performance degrades on highly suboptimal explore datasets.
- Pixel-based giant tasks show a significant drop and value training diverged for some visual tasks.
Experimental setup
Datasets: OGBench pointmaze OGBench antmaze OGBench humanoidmaze OGBench cube OGBench scene OGBench visual-antmaze OGBench visual-cube OGBench visual-scene
Benchmarks: OGBench
Metrics: average binary success rate (%)
Baselines: GCBC GCIVL GCIQL QRL CRL HIQL RIS
Builds on: HIQL RIS GCIVL Advantage-Weighted Regression OGBench
Closest papers by meaning
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.91)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.90)
- Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Score Models for Offline Goal-Conditioned Reinforcement Learning (ICLR 2024 · similarity 0.89)
- OGBench: Benchmarking Offline Goal-Conditioned RL (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.