Research map / Multi-agent systems and games / LLM agents
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Retroformer, which trains a small local retrospective LLM (LongChat-7B) with PPO on reward differences between consecutive trials to write reflections that improve a frozen actor LLM's prompt.
Problem
Frozen LLM language agents generate poor verbal self-reflections because they are not optimized for credit assignment from environment rewards, and existing reflective agents (Reflexion) do not use gradient-based learning from rewards.
Approach
Treat the frozen actor LLM as part of the environment and train a plug-in retrospective LM with policy gradient (offline PPO with a learned reward model), where the rating of a reflection is the change in episode return in the next trial.
Key results
- On HotPotQA with GPT-3 actor, Retroformer (LoRA r=4) reaches 54% success after 4 trials vs 50% for Reflexion and 34% for ReAct.
- On AlfWorld with GPT-3 actor, Retroformer reaches 100% success after 4 retries vs 84.33% for Reflexion.
- On WebShop, improvements are small: 36% vs 35% for Reflexion with GPT-3 actor after 4 retries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the retrospective model with policy gradients (Retroformer vs. Reflexion) | GPT-3 actor · HotPotQA, N=4 | success rate: 50% -> 54% | Helped |
| Fine-tune the retrospective model with policy gradients (Retroformer vs. Reflexion) | GPT-3 actor · AlfWorld, N=4 | success rate: 84.33% -> 100% | Helped |
| Fine-tune the retrospective model with policy gradients (Retroformer vs. Reflexion) | GPT-3 actor · WebShop, N=4 | success rate: 35% -> 36% | Helped |
| Fine-tune the retrospective model with policy gradients (Retroformer vs. Reflexion) | GPT-4 actor · HotPotQA, N=4 | success rate: 52% -> 54% | Helped |
| Fine-tune the retrospective model with policy gradients (Retroformer vs. Reflexion) | GPT-4 actor · AlfWorld, N=4 | success rate: 85.07% -> 100% | Helped |
| Fine-tune the retrospective model with policy gradients (Retroformer vs. Reflexion) | GPT-4 actor · WebShop, N=4 | success rate: 44% -> 46% | Helped |
| Increase retrospective-model LoRA rank from 1 to 4 (ablation) | Retroformer with GPT-3 actor · HotPotQA, N=1 | success rate: 45% -> 48% | Helped |
| Increase retrospective-model LoRA rank from 1 to 4 (ablation) | Retroformer with GPT-3 actor · AlfWorld, N=1 | success rate: 93.28% -> 97.76% | Helped |
| Increase retrospective-model LoRA rank from 1 to 4 (ablation) | Retroformer with GPT-3 actor · WebShop, N=1 | success rate: 36% -> 34% | Hurt |
| Increase retrospective-model LoRA rank from 1 to 4 (ablation) | Retroformer with GPT-4 actor · HotPotQA, N=1 | success rate: 48% -> 51% | Helped |
| Increase retrospective-model LoRA rank from 1 to 4 (ablation) | Retroformer with GPT-4 actor · AlfWorld, N=1 | success rate: 95.62% -> 97.76% | Mixed |
| Increase retrospective-model LoRA rank from 1 to 4 (ablation) | Retroformer with GPT-4 actor · WebShop, N=1 | success rate: 43% -> 43% | No clear effect |
Surprising findings
- The reinforced retrospective model spontaneously restructured its outputs into separate Reflection and New plan sections without being trained or prompted for this format.
- Verbal-feedback approaches appear not optimal for WebShop, with gains limited to about 1-4 points.
Limitations
- WebShop gains are marginal and the authors note verbal feedback may not suit that environment.
- Actor LLM is frozen and evaluated only with GPT-3 and GPT-4 at temperature zero; the reward rating assumes returns vary mostly due to reflections.
- Evaluation uses only 100 HotPotQA, 134 AlfWorld and 100 WebShop validation tasks.
Experimental setup
Datasets: HotPotQA AlfWorld WebShop ShareGPT
Benchmarks: HotPotQA AlfWorld WebShop
Metrics: success rate
Baselines: ReAct Reflexion Soft Actor-Critic
Builds on: Reflexion ReAct PPO RLHF LoRA
Closest papers by meaning
- AgentRefine: Enhancing Agent Generalization through Refinement Tuning (ICLR 2025 · similarity 0.89)
- Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space (NeurIPS 2026 · similarity 0.89)
- Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization (ICML 2025 · similarity 0.88)
- Self-Challenging Language Model Agents (NeurIPS 2025 · similarity 0.88)
- Recursive Introspection: Teaching Language Model Agents How to Self-Improve (NeurIPS 2024 · similarity 0.88)
- AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.