Research map / Reinforcement learning / Visual and representation RL
Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Graph-Assisted Stitching (GAS), which replaces high-level policy learning in offline HRL with shortest-path subgoal selection over a graph of TDR-space state clusters filtered by a Temporal Efficiency metric.
Problem
Offline hierarchical RL relies on learned high-level policies for subgoal generation, which degrade on long-horizon tasks and fail to stitch state transitions across trajectories.
Approach
Build a graph whose nodes are clusters of states in a Temporal Distance Representation space, keep only high-Temporal-Efficiency states, run Dijkstra to pick subgoals, and train a TD-aware subgoal-conditioned low-level policy (DDPG+BC) with no explicit high-level policy.
Key results
- Achieves 88.3 on antmaze-giant-stitch versus 1.0 for the prior best (HIQL).
- Achieves 94.2 on antmaze-large-explore versus 2.9 for HIQL and 2.4 for HHILP.
- Visual antmaze-giant-stitch score of 55.8, the only method with meaningful performance on that task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TE filtering to graph construction (ablation) | GAS · antmaze-giant-navigate | normalized return: 63.4 ± 3.7 -> 77.6 ± 2.9; +14.2 | Helped |
| add TE filtering to graph construction (ablation) | GAS · antmaze-giant-stitch | normalized return: 75.3 ± 5.7 -> 88.3 ± 3.6; +13.0 | Helped |
| add TE filtering to graph construction (ablation) | GAS · antmaze-large-explore | normalized return: 75.4 ± 4.3 -> 94.2 ± 3.0; +18.8 | Helped |
| add TE filtering to graph construction (ablation) | GAS · scene-play | normalized return: 63.5 ± 5.5 -> 73.6 ± 8.0; +10.1 | Helped |
| use fixed temporal-distance subgoal sampling instead of random-direction sampling (ablation) | GAS low-level policy training · antmaze-giant-navigate | normalized return: 23.3 ± 4.0 -> 77.6 ± 2.9 | Helped |
| use fixed temporal-distance subgoal sampling instead of step-based sampling (ablation) | GAS low-level policy training · scene-play | normalized return: 17.6 ± 4.7 -> 73.6 ± 8.0 | Helped |
| set H_TD to 8 rather than 4 (ablation) | GAS · antmaze-giant-stitch | normalized return: 80.7 ± 3.1 -> 88.3 ± 3.6 | Helped |
| set H_TD to 4 rather than 8 (ablation) | GAS · antmaze-large-explore | normalized return: 94.2 ± 3.0 -> 98.2 ± 1.3 | Helped |
| use graph node selection with uniform temporal-distance separation instead of FPS or K-Means++ (ablation) | GAS · all evaluated tasks | normalized return | Helped |
| use a smaller BC coefficient (ablation) | GAS low-level policy training · antmaze-large-explore | normalized return | Helped |
| use sufficiently high TDR expectile (0.95 or above) (ablation) | GAS TDR · most environments | normalized return | Helped |
| increase TDR latent dimension within 16 to 48 (ablation) | GAS TDR · across tasks | normalized return | No clear effect |
Surprising findings
- Fixed-step subgoal sampling and random direction sampling both underperform the TD-aware sampling, with random direction sampling scoring 5.3 on antmaze-giant-stitch.
- Using all dataset states for graph construction yields lower returns than TE-filtered graphs, even though TE keeps only 2-8% of states.
Limitations
- Performance on visual state environments is lower than on numeric states, attributed to limited high-dimensional representation learning.
- Hyperparameters such as H_TD and the TE threshold vary per environment and must be tuned.
Experimental setup
Datasets: OGBench antmaze navigate/stitch/explore OGBench scene-play D4RL kitchen-partial visual-antmaze and visual-scene variants
Benchmarks: OGBench D4RL kitchen
Metrics: normalized return
Baselines: GCBC GCIQL QRL CRL HGCBC HHILP HIQL
Builds on: HIQL HILP IQL DDPG+BC OGBench
Closest papers by meaning
- DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching (ICML 2024 · similarity 0.90)
- State-Covering Trajectory Stitching for Diffusion Planners (NeurIPS 2025 · similarity 0.89)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.89)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.89)
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning (NeurIPS 2026 · similarity 0.88)
- Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View. (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.