Research map / Reinforcement learning / Offline RL
In-Context Reinforcement Learning From Suboptimal Historical Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Decision Importance Transformer (DIT), which pretrains an in-context autoregressive transformer for RL from suboptimal historical trajectories by weighting actions with exponentiated in-context advantage estimates, removing the need for optimal action labels.
Problem
In-context RL pretraining (e.g., DPT, AD) requires optimal action labels or full learning histories, which are hard to obtain; the paper targets pretraining transformers for ICRL using only suboptimal historical trajectories.
Approach
Weight the supervised pretraining loss of a causal transformer policy by exp(advantage/eta), where advantages are estimated per trajectory by in-context Q and V transformers trained with Monte Carlo regression and Bellman regularization, so that good actions in suboptimal data are emphasized and the policy improves over behavior policies.
Key results
- On linear bandits, DIT outperforms UCB and Thompson Sampling in the online setting and significantly outperforms TS and LCB in offline testing with random behavior contexts.
- On Dark Room, DIT outperforms AD and matches DPT (which uses optimal action labels) in most offline and online scenarios.
- On Meta-World and Half-Cheetah, DIT outperforms PDT and BC in all testing scenarios and consistently outperforms AD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DIT’s exponential advantage-weighted pretraining instead of unweighted behavior cloning (ablation) | DIT · Miniworld | episode cumulative return | Helped |
| Use DIT instead of Prompt-DT | In-context continuous-control models · Meta-World and Half-Cheetah | episode cumulative return | Helped |
| Use DIT instead of Behavior Cloning | In-context continuous-control models · Meta-World and Half-Cheetah | episode cumulative return | Helped |
| Use DIT instead of Algorithm Distillation | In-context continuous-control models · Meta-World and Half-Cheetah | episode cumulative return | Helped |
| Use DIT instead of Prompt-DT | In-context policy · Dark Room | episode cumulative return | Mixed |
| Use DIT instead of DPT | In-context policy · Dark Room, offline evaluation with expert and random trajectories | episode cumulative return | No clear effect |
| Use DIT instead of Thompson Sampling and Lower Confidence Bound | Offline bandit policy · Linear Bandits, random-policy context | suboptimality | Helped |
| Use DIT instead of UCB and Thompson Sampling | Online bandit policy · Linear Bandits | cumulative regret | Helped |
| Use DIT instead of DPT | Offline bandit policy · Linear Bandits | suboptimality | Hurt |
| Use expert-policy context instead of randomly generated behavior-policy context | DIT · Linear Bandits, offline evaluation | suboptimality | Helped |
| Restrict DPT query states to observed pretraining states (ablation) | DPT compared with DIT · Miniworld | episode cumulative return | Mixed |
Surprising findings
- DPT struggles in the online setting on Miniworld while DIT gradually improves its return.
- Without any optimal action labels, DIT is slightly outperformed by DPT on bandits but matches it in many MDP scenarios.
Limitations
- Requires behavior policies with reasonable rewards; cannot infer near-optimal actions from purely random trajectories.
- Comparisons to AD and DPT are acknowledged as unfair because they use extra information during pretraining.
- Experiments use small GPT-2 models (6 layers, embedding 256) due to limited compute.
Experimental setup
Datasets: Dark Room (synthetic, suboptimal mixture of optimal and random actions) Miniworld (synthetic) Linear bandit synthetic tasks Meta-World historical trajectories from SAC Half-Cheetah historical trajectories from SAC
Benchmarks: Linear bandits Dark Room Miniworld Meta-World reach-v2 Half-Cheetah velocity
Metrics: cumulative regret suboptimality episode cumulative return
Baselines: Empirical Mean UCB LCB Thompson Sampling SAC Algorithm Distillation DPT Prompt-DT Behavior Cloning
Builds on: Decision-Pretrained Transformer Algorithm Distillation Advantage-weighted regression Prompt-DT Decision Transformer
Closest papers by meaning
- Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining (ICLR 2024 · similarity 0.92)
- Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement (NeurIPS 2026 · similarity 0.91)
- In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought (ICML 2024 · similarity 0.91)
- From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- When should we prefer Decision Transformers for Offline Reinforcement Learning? (ICLR 2024 · similarity 0.89)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.