Research map / Reinforcement learning / RL from feedback
Imitation Dominates Reinforcement: Direct In-Context RL Is Closer to ICL Than RL
Minchan Kwon, Seunghee Koh, Sunghyun Baek, Minsung Bae, Junmo Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Investigates whether direct in-context reinforcement learning (ICRL) in LLMs is genuine inference-time RL or better understood as in-context learning, arguing imitation dominates the reward-driven signal.
Problem
Whether in-context RL, where an LLM accumulates trajectory-reward pairs in its context at inference time, actually learns from reward as RL does or merely functions as in-context imitation learning.
Approach
Analyzes direct ICRL, in which the model directly uses past trajectory-reward pairs in context, and argues that its behavior is closer to ICL (imitation of context) than to RL, with imitation dominating the reward-based improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| flip the reward field of all trajectories (ablation) | direct ICRL (Qwen-3.5-9B) · HumanEval+, MBPP+, ScienceWorld | episode reward | No clear effect |
| randomize the reward field (ablation) | direct ICRL (Qwen-3.5-9B) · HumanEval+, MBPP+, ScienceWorld | episode reward | No clear effect |
| remove the reward field (trajectory only) (ablation) | direct ICRL (Qwen-3.5-9B) · HumanEval+, MBPP+, ScienceWorld | episode reward | No clear effect |
| use true rewards instead of random/flipped with memory held fixed (ablation) | six LLMs · HumanEval+/MBPP+ | share of tasks improved: 14 of 360 reward-sensitive | No clear effect |
| show trajectories in context (no reward) (ablation) | Qwen-3.5-9B · HumanEval+/MBPP+ | memory replay rate: 16.3 -> 34.3 | Improved |
| attach true reward to trajectories (ablation) | Qwen-3.5-9B · HumanEval+/MBPP+ | Delta LOO for lowest-reward target: 0.416 -> 0.360 | Mixed |
| place high-reward target last in memory (ablation) | Qwen-3.5-9B · HumanEval+/MBPP+ | target replay rate: 3.0 -> 13.2 | Improved |
| set all memory rewards to zero (All Failure) (ablation) | Qwen-3.5-9B · HumanEval+/MBPP+ | memory replay rate: 30.5 -> 27.6 | No clear effect |
| add an explicit instruction not to repeat shown trajectories (ablation) | Qwen-3.5-9B · HumanEval+/MBPP+ | memory replay rate: 30.5 -> 30.2 | No clear effect |
| replace raw trajectories with Reflexion summaries or cross-task trajectories (ablation) | Qwen-3.5-9B · HumanEval+/MBPP+ | memory replay rate: 30.5 -> 11.4 (Reflexion) | Hurt |
| use ICRL history instead of random sampling memory (ablation) | Qwen-3.5-9B · MBPP+ | mean reward: 0.250 -> 0.245 | No clear effect |
| shuffle or cross over trajectory contents (ablation) | Qwen-3.5-9B · MBPP+ | mean reward: 0.245 -> 0.243 | No clear effect |
Surprising findings
- Direct ICRL behaves more like imitation of in-context trajectories than like reward-driven reinforcement, per the paper's framing.
Closest papers by meaning
- Reward Is Enough: LLMs Are In-Context Reinforcement Learners (ICLR 2026 · similarity 0.90)
- Towards Provable Emergence of In-Context Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Context Value Informed In-Context Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.88)
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.88)
- TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
