Research map / Multi-agent systems and games / LLM agents

Don't Just Fine-tune the Agent, Tune the Environment

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Environment Tuning, an RL paradigm that trains tool-use agents from 400 BFCL V3 instances via a four-stage curriculum, actionable environment feedback, and a progress reward, improving in-distribution and OOD generalization.

Problem

Training multi-turn tool-use LLM agents is hard under extreme data scarcity: SFT on synthetic trajectories overfits and generalizes poorly, while direct RL suffers from cold-start and training instability over long interaction chains.

Approach

Train agents directly from problem instances without expert trajectories, using a structured four-stage curriculum (syntax, basic with augmented feedback, advanced, then alignment with evaluation environment), LLM-generated actionable hints in tool error feedback, and a turn-level progress reward instead of sparse binary success.

Key results

• Qwen2.5-7B-Instruct improves from 7.00% to 36.92% average on BFCL V3 multi-turn with Environment Tuning, versus 18.00% for ToolRL.
• watt-tool-8B rises from 35.74% to 54.34% on BFCL V3 multi-turn with Environment Tuning.
• On ACEBench Agent, ToolACE-2-Llama-3.1-8B improves from 8.34% to 15.00% while ToolRL drops to 6.65%.

Surprising findings

• Direct single-stage GRPO on Qwen2.5-7B-Instruct collapsed within 70 steps, yielding only about a 10% improvement.
• ToolRL degraded ToolACE-2-Llama-3.1-8B on BFCL V3 (37.99% to 33.75%), showing direct RL can hurt SFT-tuned models.

Limitations

• Training and in-distribution evaluation confined to BFCL V3 closed-domain API calls.
• Augmentation pipeline relies on LLM generator and judge and requires few-shot seed examples per domain.
• Scores on OOD benchmarks remain low in absolute terms (e.g., web search and memory tasks).

Experimental setup

Datasets: BFCL V3 multi-turn

Benchmarks: BFCL V3 multi-turn BFCL V4 web search BFCL V4 memory tau2-bench ACEBench Agent

Metrics: accuracy (success rate)

Baselines: ToolRL xLAM-2-8b-fc-r BitAgent-8B Arch-Agent-7B Claude Sonnet 4 GPT-4o Gemini 2.5 Pro o3

Builds on: GRPO ToolRL DAPO ProRL

Closest papers by meaning

• Test-Time Adaptation for LLM Agents via Environment Interaction (ICLR 2026 · similarity 0.90)
• Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.89)
• Training a Generally Curious Agent (ICML 2025 · similarity 0.89)
• Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use (ICML 2026 · similarity 0.89)
• FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.88)
• $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.88)