Research map / Reinforcement learning / Visual and representation RL
Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OTA, an option-aware temporally abstracted value learning scheme that shortens the effective horizon of HIQL's high-level value function, improving offline goal-conditioned RL on OGBench maze and visual manipulation tasks.
Problem
Offline goal-conditioned RL struggles on long-horizon tasks because HIQL's high-level policy receives incorrect advantage signs from a noisy goal-conditioned value function when state-goal distances are large.
Approach
Modify the IQL-style TD target so that reward and bootstrapped value are computed at the state reached after an n-step option (fixed timeout or goal), with the discount kept fixed per option, which contracts the effective horizon and yields more order-consistent high-level advantages.
Key results
- OTA raises HumanoidMaze-giant-navigate success from 12% (HIQL) to 92%, and HumanoidMaze-giant-stitch from 3% to 79%.
- On AntMaze-large-explore, OTA achieves 75% success versus 4% for HIQL.
- Order consistency ratio on AntMaze-large-explore rises from 0.75 (HIQL) to 0.97 (OTA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use option-aware temporally abstracted value learning instead of HIQL value learning | Hierarchical offline GCRL · HumanoidMaze-giant-navigate | success rate: 12 ${\scriptstyle \pm 4}$ -> 92 ${\scriptstyle \pm 0}$ | Helped |
| use option-aware temporally abstracted value learning instead of HIQL value learning | Hierarchical offline GCRL · Visual-cube-double | success rate: 59 ${\scriptstyle \pm 3}$ -> 65 ${\scriptstyle \pm 2}$ | Helped |
| use option-aware temporally abstracted value learning instead of HIQL value learning | Hierarchical offline GCRL · Visual-scene | success rate: 50 ${\scriptstyle \pm 1}$ -> 54 ${\scriptstyle \pm 2}$ | Helped |
| use n-step TD with the original γ instead of 1-step TD (ablation) | HIQL · AntMaze-large-explore (ALE) | success rate: 4 ${\scriptstyle \pm 5}$ -> 0 ${\scriptstyle \pm 0}$ | Hurt |
| use n-step TD with the original γ instead of 1-step TD (ablation) | HIQL · HumanoidMaze-large-stitch (HLS) | success rate: 12 ${\scriptstyle \pm 4}$ -> 50 ${\scriptstyle \pm 4}$ | Helped |
| use n-step TD with the original γ instead of 1-step TD (ablation) | HIQL · HumanoidMaze-giant-stitch (HGS) | success rate: 28 ${\scriptstyle \pm 3}$ -> 2 ${\scriptstyle \pm 2}$ | Hurt |
| use n-step TD with the original γ instead of 1-step TD (ablation) | HIQL · AntMaze-giant-stitch (AGS) | success rate: 2 ${\scriptstyle \pm 2}$ -> 0 ${\scriptstyle \pm 0}$ | Hurt |
| increase the discount factor to γ^(1/n) while retaining 1-step TD (ablation) | HIQL · AntMaze-large-explore (ALE) | success rate: 4 ${\scriptstyle \pm 5}$ -> 3 ${\scriptstyle \pm 3}$ | No clear effect |
| increase the discount factor to γ^(1/n) while retaining 1-step TD (ablation) | HIQL · HumanoidMaze-giant-stitch (HGS) | success rate: 28 ${\scriptstyle \pm 3}$ -> 2 ${\scriptstyle \pm 1}$ | Hurt |
| increase the abstraction factor n (ablation) | OTA value learning · Long-horizon trajectories | order consistency ratio | Mixed |
| provide oracle short-horizon subgoals to the HIQL low-level policy (ablation) | HIQL on challenging OGBench maze tasks · Eight challenging maze tasks | success rate | Helped |
Surprising findings
- Simply using n-step TD learning or increasing the discount factor (gamma^(1/n)) in HIQL does not match OTA's gains, indicating temporal abstraction via options matters beyond horizon shortening by discounting.
- Non-hierarchical methods such as GCBC, GCIVL, GCIQL, and CRL largely fail on the hardest maze tasks where HIQL also degrades.
Limitations
- Introduces an additional hyperparameter n (abstraction factor) plus per-task tuning of k, beta, and n.
- Does not yield order-consistent value estimates for all state-goal pairs.
- Long-horizon evaluation (trajectories over 1000 steps) is restricted to maze datasets.
Experimental setup
Datasets: OGBench AntMaze datasets OGBench HumanoidMaze datasets OGBench PointMaze datasets OGBench Visual-cube datasets OGBench Visual-scene datasets
Benchmarks: OGBench
Metrics: success rate order consistency ratio
Baselines: GCBC GCIVL GCIQL QRL CRL HIQL
Builds on: HIQL IQL OGBench Hindsight Experience Replay advantage-weighted regression
Closest papers by meaning
- Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.91)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.90)
- Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.89)
- Conservative Offline Goal-Conditioned Implicit V-Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.