Research map / Reinforcement learning / Visual and representation RL

In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought

ICML 2024 · Reinforcement learning › Visual and representation RL · approach: Probabilistic and autonomous RL · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes In-Context Decision Transformer (IDT), which performs in-context trial-and-error over high-level decisions (latent vectors every c steps) instead of low-level actions, cutting context length and online evaluation time on D4RL and Grid World.

Problem

In-context RL with across-episodic contexts becomes computationally expensive for long-horizon tasks (around 1000 steps) because self-attention cost grows quadratically with the length of multi-episode action-level contexts.

Approach

Reconstructs the across-episodic context to consist of high-level latent decisions generated every c steps, each controlling c low-level actions via a hierarchical trio of transformer modules (Making Decisions, Decisions to Go, Reviewing Decisions), trained end-to-end, so the context is shorter than one episode.

Key results

• On D4RL, IDT reaches a total average normalized score of 87.90 versus 85.21 for AT and 84.83 for TD3+BC.
• Online evaluation is about 36x faster than baselines on D4RL and 27x faster on Large Grid World, per the paper's claims.
• On Large Darkroom with a 2000-step context, IDT's testing time is 5.27 minutes versus 67.22 for AT and 45.08 for AD.

Measured effects reported in the paper

Surprising findings

• Neither AT nor AD shows clear self-improvement on Large Darkroom Hard, a sparse-reward task, while IDT does.

Limitations

• Baseline AT training on D4RL was infeasible due to memory, so AT numbers are taken from the original paper.
• Hyperparameter sensitivity to c differs between dense D4RL and sparse Grid World tasks.
• In-context self-improvement emerges only when context size is large enough (around 4 episodes).

Experimental setup

Datasets: D4RL HalfCheetah D4RL Hopper D4RL Walker Grid World (Darkroom, Darkroom Hard, Darkroom Dynamic, Dark Key-to-Door) Large Grid World variants

Benchmarks: D4RL Grid World (Darkroom variants) Large Grid World variants

Metrics: normalized return testing time training time

Baselines: Agentic Transformer Algorithm Distillation TD3+BC TD3 Decision Transformer BC-10%

Builds on: Decision Transformer Algorithm Distillation Agentic Transformer Chain of Hindsight GPT

Closest papers by meaning

• In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · similarity 0.91)
• In-context Exploration-Exploitation for Reinforcement Learning (ICLR 2024 · similarity 0.90)
• Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.90)
• XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.90)
• Generalization to New Sequential Decision Making Tasks with In-Context Learning (ICML 2024 · similarity 0.89)
• Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning (ICLR 2025 · similarity 0.89)