Research map / Reinforcement learning / Visual and representation RL
Subgoal generation for hierarchical RL
Generate adjacency-constrained or probabilistic subgoals so high-level policies guide low-level control.
35 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Visual and representation RL, keyword cluster “goal · conditioned · hierarchical”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
7 papers in 2023–24 and 14 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 1.2% to 2.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Data-Efficient Hierarchical Reinforcement Learning (NeurIPS 2018 · 248 citations)
- 2019 · Language as an Abstraction for Hierarchical Deep Reinforcement Learning (NeurIPS 2019 · 97 citations)
- 2020 · Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning (NeurIPS 2020 · 31 citations)
- 2021 · Hierarchical Reinforcement Learning by Discovering Intrinsic Options (ICLR 2021 · 24 citations)
- 2022 · Deep Hierarchical Planning from Pixels (NeurIPS 2022 · 14 citations)
- 2023 · State-Conditioned Adversarial Subgoal Generation (AAAI 2023 · 7 citations)
- 2024 · Handling Long and Richly Constrained Tasks through Constrained Hierarchical Reinforcement Learning (AAAI 2024 · 2 citations)
- 2025 · State-Covering Trajectory Stitching for Diffusion Planners (NeurIPS 2025 · 1 citation)
- 2026 · Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL (ICML 2026 · 0 citations)
Earliest papers here
- Data-Efficient Hierarchical Reinforcement Learning (NeurIPS 2018 · 248 citations)
- Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (ICML 2018 · 79 citations)
- Language as an Abstraction for Hierarchical Deep Reinforcement Learning (NeurIPS 2019 · 97 citations)
What has been tried
25 papers here report 260 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 10%, hurt 20%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Guide stitching with learned temporal distance-preserving embeddings rather than raw state-space distances | AntMaze Stitch · success rate | 45 ± 1 -> 93 ± 1; 7 ± 2 -> 87 ± 2 | Helped | State-Covering Trajectory Stitching for Diffusion Planners (NeurIPS 2025 · 1 citation) |
| Add conditional diffusion subgoal generation | Continuous control tasks · performance | ~15% | Helped | Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals (ICML 2025 · 0 citations) |
| add TE filtering to graph construction | antmaze-giant-navigate · normalized return | 63.4 ± 3.7 -> 77.6 ± 2.9; +14.2 | Helped | Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning (ICML 2025 · 0 citations) |
| Use more NF blocks in NF-BC | Push-T · normalized reward | 0.77 to 0.87 across tested block counts; 48 blocks: 0.87, 54 blocks: 0.85 | Mixed | Normalizing Flows are Capable Models for Continuous Control (NeurIPS 2025 · 0 citations) |
| Scale depth beyond 4 layers | JaxGCRL tasks · success | No clear effect | 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities (NeurIPS 2025 · 0 citations) | |
| Increase action sequence length from 2 to 16 | most continuous-control tasks · FAS | Mixed | Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · 0 citations) | |
| Add the probabilistic subgoal formulation to the baseline without it | Ablation environments · asymptotic performance | Helped | Probabilistic Subgoal Representations for Hierarchical Reinforcement Learning (ICML 2024 · 0 citations) | |
| Add primitive-informed value regularization | Pick and place and push · success rate | Helped | PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 8 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- OGBench · 2 of 8 papers
Baselines
- QRL · 4 of 8 papers
- GCBC · 3 of 8 papers
- GCIQL · 3 of 8 papers
- CRL · 3 of 8 papers
- HIQL · 2 of 8 papers
- SAC · 2 of 8 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 21 papers; from OpenAlex and the papers' affiliations).
- Aalto University · 3 papers
- Tampere University · 2 papers
- Princeton University · 2 papers
Use it
Download BibTeX of 34 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Goal-conditioned policy learning (66 papers)
- Graph search and compositional goals (55 papers)
Elsewhere in the topic
- In-context and adaptive RL (202 papers)
- Scalable RL systems and environments (179 papers)
- Probabilistic and autonomous RL (177 papers)
- RL for combinatorial routing (158 papers)
- Learned state and action representations (148 papers)
- Dexterous manipulation transfer (141 papers)
- RL with tree search for discovery (121 papers)
- Inferring decision-making from behavior (93 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).