AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Imitation Dominates Reinforcement: Direct In-Context RL Is Closer to ICL Than RL

Minchan Kwon, Seunghee Koh, Sunghyun Baek, Minsung Bae, Junmo Kim

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Investigates whether direct in-context reinforcement learning (ICRL) in LLMs is genuine inference-time RL or better understood as in-context learning, arguing imitation dominates the reward-driven signal.

Problem

Whether in-context RL, where an LLM accumulates trajectory-reward pairs in its context at inference time, actually learns from reward as RL does or merely functions as in-context imitation learning.

Approach

Analyzes direct ICRL, in which the model directly uses past trajectory-reward pairs in context, and argues that its behavior is closer to ICL (imitation of context) than to RL, with imitation dominating the reward-based improvement.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
flip the reward field of all trajectories (ablation)direct ICRL (Qwen-3.5-9B) · HumanEval+, MBPP+, ScienceWorldepisode rewardNo clear effect
randomize the reward field (ablation)direct ICRL (Qwen-3.5-9B) · HumanEval+, MBPP+, ScienceWorldepisode rewardNo clear effect
remove the reward field (trajectory only) (ablation)direct ICRL (Qwen-3.5-9B) · HumanEval+, MBPP+, ScienceWorldepisode rewardNo clear effect
use true rewards instead of random/flipped with memory held fixed (ablation)six LLMs · HumanEval+/MBPP+share of tasks improved: 14 of 360 reward-sensitiveNo clear effect
show trajectories in context (no reward) (ablation)Qwen-3.5-9B · HumanEval+/MBPP+memory replay rate: 16.3 -> 34.3Improved
attach true reward to trajectories (ablation)Qwen-3.5-9B · HumanEval+/MBPP+Delta LOO for lowest-reward target: 0.416 -> 0.360Mixed
place high-reward target last in memory (ablation)Qwen-3.5-9B · HumanEval+/MBPP+target replay rate: 3.0 -> 13.2Improved
set all memory rewards to zero (All Failure) (ablation)Qwen-3.5-9B · HumanEval+/MBPP+memory replay rate: 30.5 -> 27.6No clear effect
add an explicit instruction not to repeat shown trajectories (ablation)Qwen-3.5-9B · HumanEval+/MBPP+memory replay rate: 30.5 -> 30.2No clear effect
replace raw trajectories with Reflexion summaries or cross-task trajectories (ablation)Qwen-3.5-9B · HumanEval+/MBPP+memory replay rate: 30.5 -> 11.4 (Reflexion)Hurt
use ICRL history instead of random sampling memory (ablation)Qwen-3.5-9B · MBPP+mean reward: 0.250 -> 0.245No clear effect
shuffle or cross over trajectory contents (ablation)Qwen-3.5-9B · MBPP+mean reward: 0.245 -> 0.243No clear effect

Surprising findings

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy