Research map / Reinforcement learning / Visual and representation RL
Rethinking Decision Transformer via Hierarchical Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Autotuned Decision Transformer (ADT), a hierarchical RL framework in which a high-level policy generates prompts (values or sub-goals) that condition a transformer low-level policy, jointly optimized for offline RL stitching.
Problem
Decision Transformer relies on recalling dataset trajectories conditioned on hand-picked return-to-go and cannot stitch suboptimal trajectories together for offline RL.
Approach
Recast DT as a two-level hierarchy (high-level prompt policy, low-level prompt-conditioned action policy) and jointly optimize both: the high-level policy outputs in-sample optimal values (V-ADT) or IQL/HIQL-learned sub-goals (G-ADT) as prompts, and the low-level transformer is trained with advantage-weighted regression and concatenated prompt-state tokens.
Key results
- V-ADT averages 65.0 on D4RL (MuJoCo, AntMaze, Kitchen) versus 43.8 for DT and 68.3 for IQL in the reported table.
- V-ADT reaches 52.4 AntMaze average versus 16.0 for DT, and 78.7 MuJoCo average versus 65.3 for DT.
- G-ADT averages 72.6 across AntMaze and Kitchen, above HIQL (77.1 is HIQL average in its table, so G-ADT is higher than WT 66.0 and RvS-R/G 53.7 but below HIQL's reported 77.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use V-ADT instead of DT | Transformer-based offline RL · AntMaze average | normalized score: 16.0 ± 2.1 -> 52.4 ± 2.9 | Helped |
| Use V-ADT instead of DT | Transformer-based offline RL · Kitchen average | normalized score: 34.6 ± 9.2 -> 49.3 ± 3.1 | Helped |
| Use V-ADT instead of TD3+BC, CQL, or IQL | Offline RL on MuJoCo · MuJoCo average | normalized score: 75.3 ± 4.9; 77.6 ± 3.4; 76.9 ± 5.8 -> 78.7 ± 2.8 | Helped |
| Use G-ADT instead of WT | Goal-conditioned transformer policy · AntMaze average | normalized score: 68.4 ± 4.9 -> 78.1 ± 2.5 | Helped |
| Use G-ADT instead of WT | Goal-conditioned transformer policy · Kitchen average | normalized score: 61.3 ± 3.4 -> 61.6 ± 3.4 | No clear effect |
| Use G-ADT instead of HIQL | Goal-conditioned offline RL · AntMaze average | normalized score: 86.2 ± 5.4 -> 78.1 ± 2.5 | Hurt |
| Use V-ADT instead of IQL | Offline RL · Kitchen average | normalized score: 53.3 -> 49.3 ± 3.1 | Hurt |
| Add value prompts (ablation) | V-ADT · Hard diverse datasets | evaluation performance | Helped |
| Add value prompts (ablation) | V-ADT · Play datasets | evaluation performance | No clear effect |
| Tune DT's target return manually (ablation) | DT · Four Walker2d datasets | evaluation performance | No clear effect |
| Use advantage-weighted low-level policy optimization instead of the DT sequence prediction loss (ablation) | ADT · Offline RL tasks | evaluation performance | Helped |
| Concatenate prompts and states before tokenization (ablation) | ADT · Offline RL tasks | evaluation performance | Helped |
Surprising findings
- Manually tuning the return-to-go target of DT does not close the gap to V-ADT, suggesting learned prompts matter more than prompt engineering.
- Using the value prompt with V-ADT helps mainly on diverse datasets, with little benefit on play datasets.
Limitations
- Results of DT and baselines partly taken from prior work or re-run with CORL, with some kitchen baselines lacking standard deviations.
- Authors note uncertainty about benefits of transformers in RL for these benchmarks, which may not be advantageous with limited data.
Experimental setup
Datasets: D4RL halfcheetah v2 D4RL hopper v2 D4RL walker2d v2 D4RL antmaze v2 D4RL kitchen v0
Benchmarks: D4RL MuJoCo D4RL AntMaze D4RL Franka Kitchen
Metrics: normalized return
Baselines: TD3+BC CQL IQL DT QLDT HIQL RvS Waypoint Transformer
Builds on: Decision Transformer Implicit Q-learning Hierarchical Implicit Q-learning Advantage-weighted regression Q-learning Decision Transformer
Closest papers by meaning
- In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought (ICML 2024 · similarity 0.90)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.90)
- Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making (ICLR 2024 · similarity 0.90)
- Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling (NeurIPS 2024 · similarity 0.90)
- Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining (ICLR 2024 · similarity 0.89)
- In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.