Research map / Multi-agent systems and games / LLM agents
Does Reinforcement Fine-Tuning Improve Generalization of LLM Agents? An Empirical Study
Zhiheng Xi, Xin Guo, Jiaqi Liu, Jiazheng Zhang, Yutao Fan, Zhihao Zhang, Shichun Liu, Mingxu Chai, Xiaowei Shi, Yitao Zhai, Xunliang Cai, Tao Gui and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of whether reinforcement fine-tuning (RFT) of LLM agents generalizes to unseen environments that differ in background knowledge, observation spaces, and action spaces.
Problem
Most RFT evaluations for multi-turn LLM agents are in-domain (same environment or same tasks), leaving out-of-domain generalization to unseen environments with different knowledge, observations, and actions unexplored.
Approach
Systematically evaluates RFT-trained agents on unseen environments that vary background knowledge, observation space, and action space, rather than only in-domain test tasks; the abstract does not specify the method details.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with easy tasks, test on hard tasks | RFT Qwen2.5-7B-Instruct · WebShop (hard) | avg@8: 17.4 -> 77.5 | Improved |
| train on hard tasks instead of easy tasks | RFT Qwen2.5-3B-Instruct · AlfWorld | avg@8: +8.7 points | Improved |
| easy-to-hard curriculum instead of single difficulty level | RFT Qwen2.5-3B-Instruct · BabyAI | avg@8: +2.0 and +3.3 points | Improved |
| easy-to-hard curriculum instead of mixed training on all tasks | RFT Qwen2.5-3B-Instruct · WebShop | avg@8: 87.9 -> 88.7 | Improved |
| easy-to-hard curriculum instead of mixed training on all tasks | RFT Qwen2.5-7B-Instruct · WebShop | avg@8: 86.5 -> 83.0 | Hurt |
| RFT on easy BabyAI tasks reduces interaction turns | RFT Qwen2.5-3B-Instruct · BabyAI | average interaction turns: 10.76 -> 4.19 | Improved |
| train on SearchQA, evaluate on WebShop (unseen) | RFT Qwen2.5-7B-Instruct · WebShop | avg@8: +18.48 points | Improved |
| train on BabyAI, evaluate on WebShop (unseen) | RFT Qwen2.5-7B-Instruct · WebShop | avg@8: 28.59 -> 10.25 | Hurt |
| RFT on single environment evaluated on unseen environments (average) | RFT Qwen2.5-3B-Instruct and 7B · Held-out environments | avg@8: +3.32 and +3.44 points | Improved |
| sequentially train on TextCraft after WebShop | RFT Qwen2.5-7B-Instruct · TextCraft (downstream) | avg@8: 80.88 -> 82.50 | Improved |
| sequentially train on TextCraft after WebShop, check upstream forgetting | RFT Qwen2.5-7B-Instruct · WebShop (upstream) | avg@8: 86.5 -> 86.32 | No clear effect |
| train BabyAI after TextCraft (TC+BA vs TC) | RFT Qwen2.5-7B-Instruct · WebShop (unseen) | avg@8: 38.30 -> 23.0 | Hurt |
Closest papers by meaning
- Don't Just Fine-tune the Agent, Tune the Environment (ICLR 2026 · similarity 0.90)
- BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning (ICLR 2026 · similarity 0.90)
- AgentRefine: Enhancing Agent Generalization through Refinement Tuning (ICLR 2025 · similarity 0.89)
- Understanding the Effects of RLHF on LLM Generalisation and Diversity (ICLR 2024 · similarity 0.89)
- Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use (ICML 2026 · similarity 0.89)
- Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
