Research map / Reinforcement learning / RL from feedback
Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RICL, which turns an LLM's log-probability shift after retrospective in-context updates into advantage estimates, and RICOL, an online advantage-weighted-regression loop that trains LLM policies on BabyAI with far fewer environment steps than PPO.
Problem
Temporal credit assignment from sparse rewards for LLM agents in multi-turn settings usually requires learned value functions, which are sample-inefficient and do not generalize across tasks.
Approach
Estimate the advantage function of an LLM policy from the log-probability difference between the original policy and a version updated in context with reflector feedback computed on the same trajectory (retrospective ICL), then use this advantage in a KL-regularized AWR-style update of the policy.
Key results
- On 1D Key-Door, RICL reaches low advantage-estimation error with about 10 trajectories, whereas Monte Carlo needs about 1000 trajectories for similar accuracy (~100x sample efficiency).
- RICL outperforms standard ICL by 7.2% in expert-action accuracy on BabyAI goto across 1000 trajectories.
- RICOL reaches comparable convergent success to PPO (10M) with roughly 50x fewer environment steps and outperforms PPO (3B) by about 10x in sample efficiency, while beating GPT-4o mini on all four BabyAI scenarios.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RICL rather than Monte Carlo for advantage estimation | Advantage estimation · 1D Key-Door | sample efficiency: 100 ×; around 10 samples vs. about 1000 samples | Helped |
| apply retrospective updates to states in the trajectory that produced the feedback, rather than updates on a different trajectory | LLaMA-3.1-8B-Instruct policy · BabyAI goto | expert-action accuracy: 7.2% | Helped |
| use RICOL instead of PPO (10M) | Online policy learning · BabyAI scenarios | environment steps to comparable performance: nearly 50 times less data | Helped |
| use RICOL instead of PPO (3B) | Online policy learning · BabyAI scenarios | sample efficiency: 10× | Helped |
| use RICOL instead of GPT-4o mini | Online policy learning · BabyAI scenarios | task success rate | Helped |
| use state-specific temporal credit assignment instead of trajectory-level rewards | RICOL versus RWR · BabyAI goto | task success rate | Mixed |
| use state-specific temporal credit assignment instead of trajectory-level rewards | RICOL versus RWR · BabyAI sparse-reward tasks | task success rate | Helped |
| use RICOL with hand-crafted verbal feedback at 70% accuracy | RICOL · BabyAI goto | task success rate: 70% | Helped |
Surprising findings
- RICOL remains effective when feedback accuracy drops to 70%, and random (50%) feedback is the tested floor rather than a failure point.
- An LLM prompted to directly identify critical states from trajectories identifies only the pick-up-key state and misses earlier critical states that RICL finds.
Limitations
- RICOL requires a discrete, enumerable action space because the partition function is computed by enumerating actions.
- Evaluated only on a 1D toy environment and four BabyAI scenarios, not on token-level MDPs or reasoning tasks.
- Requires a reflector LLM (GPT-4o mini) and extra per-action queries (|A| queries per state), adding cost.
Experimental setup
Datasets: BabyAI goto BabyAI pickup BabyAI pick_up_seq_go_to BabyAI open 1D Key-Door (synthetic)
Benchmarks: BabyAI goto BabyAI pickup BabyAI pick_up_seq_go_to BabyAI open 1D Key-Door
Metrics: task success rate environment time steps KL divergence to ground-truth policy accuracy of expert action prediction training time
Baselines: GPT-4o mini Reflexion PPO (3B) PPO (10M) RWR Monte Carlo advantage estimation Retroformer
Builds on: Reflexion advantage-weighted regression PPO BabyAI BALROG
Closest papers by meaning
- Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025 · similarity 0.87)
- S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF (NeurIPS 2026 · similarity 0.86)
- Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use (NeurIPS 2026 · similarity 0.86)
- Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning (ICML 2024 · similarity 0.85)
- VinePPO: Refining Credit Assignment in RL Training of LLMs (ICML 2025 · similarity 0.85)
- STAS: Spatial-Temporal Return Decomposition for Solving Sparse Rewards Problems in Multi-agent Reinforcement Learning (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.