Research map / Large language models / Agents, code and math
The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Critique-Guided Improvement (CGI), a two-player actor-critic framework where a fine-tuned Llama-3-8B critic gives natural-language critiques of candidate actions and the actor is iteratively refined via SFT to use them, evaluated on WebShop, ScienceWorld and TextCraft.
Problem
LLM agents get limited guidance from scalar reward signals and struggle to use or implement natural-language feedback; self-refinement is unreliable and fine-tuned actors often ignore critiques.
Approach
A separately trained critic (fine-tuned on GPT-4o expert critiques using gold trajectories) scores and suggests revisions for M candidate actions at each step; the actor is then iteratively fine-tuned on correct trajectories and critique-action pairs so it learns to follow critiques, with general data mixed in to preserve generalization.
Key results
- Critic model (Llama-3-8B) with Llama-3-8B actor reaches 61.44 average vs 32.28 for GPT-4o critic and 12.65 no-critique baseline.
- CGI after three action-refinement iterations achieves 74.20 average across three environments, ScienceWorld 78.43, versus 58.21 for iterative SFT.
- Llama-3-70B actor with the 8B critic reaches 65.88 average vs 38.65 with GPT-4o critic.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a trained verbal critic instead of no critique | Llama-3-8B-Instruct actor · WebShop | average final score: 13.49 -> 56.80 | Helped |
| Use a trained verbal critic instead of no critique | Llama-3-8B-Instruct actor · ScienceWorld | average final score: 14.48 -> 68.51 | Helped |
| Use a trained verbal critic instead of no critique | Llama-3-8B-Instruct actor · TextCraft | success rate: 10.00 -> 59.00 | Helped |
| Use CGI action refinement rather than iterative SFT | Llama-3-8B-Instruct · ScienceWorld | average final score: 41.42 -> 78.43 | Helped |
| Use CGI action refinement rather than iterative SFT | Llama-3-8B-Instruct · TextCraft | success rate: 55.00 -> 68.00 | Helped |
| Use CGI action refinement rather than iterative SFT | Llama-3-8B-Instruct · WebShop | average final score: 78.21 -> 76.17 | Hurt |
| Add critique-action pairs to actor training (ablation) | CGI first-iteration actor · Average across three environments | average score: 50.37 -> 68.50 | Helped |
| Add correct trajectories to actor training (ablation) | CGI first-iteration actor · Average across three environments | average score | Helped |
| Add general conversational data to actor training (ablation) | CGI first-iteration actor · Average across three environments | average score | Helped |
| Train the critic with expert trajectories rather than GPT-4o-generated trajectories (ablation) | CGI · Average across three environments | average score: 50.71 -> 61.44 | Helped |
| Update the critic after actor refinement (ablation) | CGI · Average across three environments | average score: 61.44 -> 67.02 | Helped |
| Increase candidate actions from one to more candidates | CGI inference · Average across three environments | average score: 56.89% to 61.72% | Helped |
Surprising findings
- Fine-tuned actors become less responsive to critiques: critique guidance lowers WebShop score for fine-tuned Llama-3-8B (76.12 to 74.68).
- Self-critique by the untrained actor degrades performance on WebShop (1.50 vs 13.49 no-critique).
Limitations
- Inference is roughly 4x slower than no-critique baseline and sampling M candidate actions per step adds cost.
- Performance plateaus at M=7 candidates and additional refinement iterations (4th) give no gain.
- Critic training relies on GPT-4o expert critiques built from gold expert trajectories, though authors report reduced but still strong results with GPT-4o-generated trajectories.
Experimental setup
Datasets: WebShop ScienceWorld TextCraft ShareGPT AgentGym expert trajectories
Benchmarks: WebShop ScienceWorld TextCraft
Metrics: average final score (WebShop, ScienceWorld) success rate (TextCraft) revision ratio inference time
Baselines: DGAP Explicit RM Self-Critique GPT-4o critic Iterative SFT Reflexion DeepSeek-Chat Claude-3-Haiku Claude-3-Sonnet GPT-3.5-Turbo GPT-4o Llama-3-70B-Instruct
Builds on: AgentGym AgentTuning Reflexion Iterative SFT (ReST-style expert iteration)
Closest papers by meaning
- Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space (NeurIPS 2026 · similarity 0.90)
- Selective Critique for Cost-Aware LLM Agents in Long-Horizon Decision Making (NeurIPS 2026 · similarity 0.90)
- Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.90)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.90)
- AgentRefine: Enhancing Agent Generalization through Refinement Tuning (ICLR 2025 · similarity 0.89)
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.