Research map / Reinforcement learning / Offline RL

Context Value Informed In-Context Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Diffusion models for offline decision making

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CV-ICRL, which adds a learned context-value head whose predictions are written back into the in-context history to enable TD-style, lower-variance test-time credit assignment for distilled in-context RL policies.

Problem

In-context reinforcement learning (ICRL) test-time adaptation is brittle and unstable under limited rollouts because return-based (Monte-Carlo) credit signals are high-variance.

Approach

Equip a causal Transformer policy with a critic head that predicts a context value conditioned on state and interaction context, write the predicted value back into the context as a value token, and use the resulting one-step TD target rt + γvt+1 for in-context updates; supervised distillation from PPO checkpoints with checkpoint-level evaluated returns as value targets.

Key results

• On Minigrid across 6 task types, CV-ICRL reports higher AER than AD, AD-ϵ and IDT on most tasks and lower Instability Frequency than AD on five of six task types (e.g., SimpleCrossingS11N5 IF 13.2% vs 15.8% for AD).
• CV-ICRL achieves the highest mean AER on all four held-out Minigrid task types in cross-task-type generalization, with Reinformer competitive except on LavaCrossingS9N3.
• Ablations show critic-only (no write-back), context-agnostic value token, and AD-DT variants all underperform full CV-ICRL on Minigrid.

Surprising findings

• Using PPO's own critic as the value target improves AER on the two SimpleCrossing task types, while checkpoint-return supervision performs better on the other four.
• AD-DT (return-to-go conditioning) is competitive on LavaCrossingS9N2 but weaker on SimpleCrossing tasks because sparse terminal rewards give near-constant RTG before termination.

Limitations

• The policy-gradient view is a working interpretation, not a formal equivalence.
• Cross-task-type generalization is mainly within Minigrid with only small-scale MetaWorld evidence.
• More refined context-value estimation may further improve performance.

Experimental setup

Datasets: PPO learning-progress trajectories on Dark Room PPO learning-progress trajectories on Minigrid PPO learning-progress trajectories on Procgen (Bigfish, Starpilot, Miner)

Benchmarks: Dark Room Minigrid Procgen MetaWorld

Metrics: Average Episode Return Last Episode Return Instability Frequency

Baselines: AD AD-ϵ IDT Reinformer IC-CQL DPT AD-DT PPO

Builds on: Algorithm Distillation Decision Transformer Inverse Decision Transformer PPO Reinformer

Closest papers by meaning

• Behavior-agnostic Task Inference for Robust Offline In-context Reinforcement Learning (ICML 2025 · similarity 0.89)
• Towards Provable Emergence of In-Context Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• Vintix: Action Model via In-Context Reinforcement Learning (ICML 2025 · similarity 0.88)
• Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement (NeurIPS 2026 · similarity 0.87)
• Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.87)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.87)