Research map / Multi-agent systems and games / LLM agents
AgentRefine: Enhancing Agent Generalization through Refinement Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AgentRefine, which synthesizes diverse agent environments and multi-turn trajectories with error-refine steps (via GPT-4o), then fine-tunes LLaMA-3 and Mistral with a loss that masks erroneous turns to improve held-out agent generalization.
Problem
Agent-tuned open LLMs perform well on held-in agent environments but generalize poorly to held-out tasks, often repeating the same erroneous actions despite environment feedback.
Approach
Synthesize diverse persona-driven agent environments and trajectories that contain error turns followed by environment-feedback-based refinement, then train with a loss that masks erroneous turns so the model learns to self-correct step-by-step rather than memorize observation-action pairs.
Key results
- AgentRefine (LLaMA-3-8B) reaches 14.4% SciWorld success vs 1.1% for Agent-FLAN, and leads Agent-FLAN by 13.3 points on SciWorld held-out success per the text.
- Under perturbations of Alfworld action descriptions, AgentRefine averages 48.5% success with std 3.73, while AgentGym and Agent-FLAN drop to ~36-37% average with std near 20.
- Best-of-N with 10 samples raises AgentRefine Alfworld success to 93.3% and averages over 25% gains over greedy decoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AgentRefine instead of Agent-FLAN | LLaMA-3-8B agent tuning · SciWorld | Success: 1.1 -> 14.4 | Helped |
| Use AgentRefine instead of Agent-FLAN | LLaMA-3-8B agent tuning · BabyAI | Success: 25.0 -> 37.5 | Helped |
| Use AgentRefine instead of Agent-FLAN | LLaMA-3-8B agent tuning · PDDL | Success: 8.3 -> 16.6 | Helped |
| Use AgentRefine instead of Agent-FLAN | LLaMA-3-8B agent tuning · Jericho | Progress: 10.1 -> 32.3 | Helped |
| Use AgentRefine instead of Agent-FLAN | LLaMA-3-8B agent tuning · Alfworld | Success: 67.2 -> 44.8 | Hurt |
| Add loss on refinement trajectory tokens (ablation) | AgentRefine · SciWorld | Success: 4.4 -> 7.7 | Helped |
| Add refinement trajectories to training data (ablation) | AgentRefine · SciWorld | Success: 5.5 -> 7.7 | Helped |
| Mask loss on erroneous turn tokens (ablation) | AgentRefine · SciWorld | Success: 3.3 -> 7.7 | Helped |
| Add verification to synthesized training data (ablation) | AgentRefine · Alfworld | Success: 25.4 -> 48.5 | Helped |
| Increase AgentRefine training data from 4k to 64k | AgentRefine · agent tasks | performance | Helped |
| Use DeepSeek-v2.5 instead of GPT-4o for synthesis | AgentRefine · BabyAI | Success: 25.0 -> 36.6 | Helped |
| Use GPT-4o rather than DeepSeek-v2.5 for synthesis | AgentRefine · SciWorld | Success: 2.2 -> 11.1 | Helped |
Surprising findings
- Training on erroneous turns (not masking them) causes a severe drop, e.g., nearly 75% on SciWorld, contrary to prior findings.
- Held-in tuned models (Agent-FLAN, AgentGym) lose much of their Alfworld success under minor action-description perturbations, suggesting memorization.
Limitations
- AgentRefine underperforms held-in-trained baselines on their in-distribution Alfworld/BabyAI tasks.
- Relies on GPT-4o for synthesis; open-source DeepSeek-V2.5 synthesis yields lower quality data.
- Verification and refinement judged by GPT-4 with only moderate human agreement (82% on wrong turns).
Experimental setup
Datasets: Persona-Hub AgentBoard test tasks (SciWorld, Alfworld, BabyAI, PDDL, Jericho) HotpotQA
Benchmarks: AgentBoard (Alfworld, BabyAI, SciWorld, PDDL, Jericho) HotpotQA MT-bench
Metrics: success rate progress rate exact match F1 standard deviation under perturbation
Baselines: GPT-4o GPT-4o-mini LLaMA-3-8B-Instruct LLaMA-3-70B-Instruct Mistral-7B-Instruct-v0.3 Agent-FLAN AgentGym AgentGen
Builds on: Agent-FLAN AgentTuning AgentGym AgentGen ReAct Self-Refine
Closest papers by meaning
- Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.91)
- MoRAgent: Parameter Efficient Agent Tuning with Mixture-of-Roles (ICML 2025 · similarity 0.89)
- The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement (NeurIPS 2025 · similarity 0.89)
- FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.89)
- Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use (ICML 2026 · similarity 0.89)
- Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.