Research map / Reinforcement learning / Visual and representation RL
Emergence of In-Context Reinforcement Learning from Noise Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AD^ε, which builds in-context RL learning histories by injecting an annealed epsilon-greedy noise schedule into a policy, removing the need for many RL agents or optimal policies.
Problem
In-context RL (Algorithm Distillation, DPT) needs costly learning histories from thousands of RL agents or optimal-action data, which is hard to obtain for complex environments.
Approach
Generate learning histories by running any policy with an epsilon-greedy noise schedule decaying from 1 to 0 (or to a nonzero floor for suboptimal policies), so successive trajectories show gradual improvement, then distill with a causal transformer via supervised next-token prediction.
Key results
- In Dark Room, AD^ε with suboptimal data improves return from 6.36 to 14.08 (+120%) over the best available data policy.
- In Key-to-Door, return rises from 1.0 to 1.74 (+74%), and in Watermaze from 0.52 to 0.92 (+76%).
- Noise-generated histories enable in-context generalization comparable to the original AD on Dark Room and Key-to-Door.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AD$^\varepsilon$ trained on noise-induced data from suboptimal policies instead of the best policy in the data | AD$^\varepsilon$ · Dark Room | cumulative reward: 6.36 → 14.08; +120% | Helped |
| Use AD$^\varepsilon$ trained on noise-induced data from suboptimal policies instead of the best policy in the data | AD$^\varepsilon$ · Dark Key-to-Door | cumulative reward: 1.0 → 1.74; +74% | Helped |
| Use AD$^\varepsilon$ trained on noise-induced data from suboptimal policies instead of the best policy in the data | AD$^\varepsilon$ · Watermaze | cumulative reward: 0.52 → 0.92; +76% | Helped |
| Generate learning histories using a suboptimal behavior-cloning policy rather than an oracle policy (ablation) | AD$^\varepsilon$ · Dark Room and Key-to-Door | cumulative reward | No clear effect |
| Set starting noise to 0.5 instead of 1 (ablation) | AD$^\varepsilon$ · Data generation | cumulative reward | No clear effect |
| Use a faster noise-decay schedule (ablation) | AD$^\varepsilon$ · Dark Key-to-Door | cumulative reward: 7 \times 10^{-5} (200 episodes) | Mixed |
| Use a noise-decay rate of 7 × 10⁻⁶ (ablation) | AD$^\varepsilon$ · Dark Key-to-Door | cumulative reward: 7 \times 10^{-6} (2000 episodes) | Helped |
| Use slower noise-decay schedules (ablation) | AD$^\varepsilon$ · Dark Key-to-Door | cumulative reward: 3.5 \times 10^{-6}; 1.4 \times 10^{-6} (4k and 10k histories, respectively) | Hurt |
| Decrease the final policy performance available in the data (ablation) | AD$^\varepsilon$ · Dark Key-to-Door | cumulative reward: 50%, 30% and 15% | Hurt |
| Train from a severely suboptimal policy with max_perf = 0.3 (ablation) | AD$^\varepsilon$ · Dark Key-to-Door | cumulative reward: 0.3 | Helped |
| Use a dataset with insufficient examples of the door-finding subgoal (ablation) | AD$^\varepsilon$ · Dark Key-to-Door | cumulative reward: 0.6 | Hurt |
| Replace the original AD learning-history collection strategy with noise-generated histories | AD$^\varepsilon$ · Dark Room and Key-to-Door | in-context learning performance | No clear effect |
Surprising findings
- In-context agents can exceed the best suboptimal policy in the data by about 2x despite never seeing optimal actions.
- Performance peaks at an intermediate epsilon decay rate; too fast or too slow decay hurts in-context learning.
Limitations
- Suboptimal data must still contain sufficient examples of every subgoal, otherwise in-context learning fails (e.g., finding the door).
- Evaluated only on small grid-world and one 3D environment; limited environment diversity.
- Learning pace (epsilon decay rate) requires tuning and has an ideal intermediate value.
Experimental setup
Datasets: Dark Room Dark Key-to-Door Watermaze
Benchmarks: Dark Room Dark Key-to-Door Watermaze
Metrics: cumulative return over in-context episodes
Baselines: Algorithm Distillation Expert Distillation Behavior Cloning Decision Pretrained Transformer
Builds on: Algorithm Distillation Decision Pretrained Transformer CORL
Closest papers by meaning
- Distilling Reinforcement Learning Algorithms for In-Context Model-Based Planning (ICLR 2025 · similarity 0.88)
- XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.88)
- Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds (NeurIPS 2025 · similarity 0.88)
- Towards Provable Emergence of In-Context Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- In-Context Reinforcement Learning From Suboptimal Historical Data (ICML 2025 · similarity 0.87)
- Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.