Research map / Multi-agent systems and games / LLM agents
Offline Training of Language Model Agents with Functions as Learnable Weights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AgentOptimizer, which trains LLM agents by iteratively adding, revising, and removing Python functions (treated as learnable parameters) using an LLM optimizer, with roll-back and early-stop, without modifying LLM weights.
Problem
Improving LLM agents usually requires manually curating tools or fine-tuning the LLM, which is costly and impossible for proprietary black-box models.
Approach
Treat the agent's function set as learnable parameters and use an LLM (AgentOptimizer) to progressively edit those functions based on execution history over a training set, with roll-back and early-stop to avoid performance degradation; the LLM weights are never updated.
Key results
- On MATH, GPT-4+ agent test accuracy rose in 11 of 14 data-type splits with agent training (e.g., Counting & Probability 72.5% to 76.3%).
- On TabMWP, ReAct agent test accuracy went from 59.0% to 70.0%; GPT-4+ agent from 51.0% to 56.0%.
- On GAIA, GPT-4+ agent train accuracy rose from 10% to 30%, with test 16% to 23%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train GPT-4+ Agent by updating its functions | GPT-4+ Agent · GAIA test | accuracy: 16.0% -> 23.0% | Helped |
| Train ReAct Agent by updating its functions | ReAct Agent · GAIA test | accuracy: 12.0% -> 18.0% | Helped |
| Train GPT-4+ Agent by updating its functions | GPT-4+ Agent · TabMWP test | accuracy: 51.0% -> 56.0% | Helped |
| Train ReAct Agent by updating its functions | ReAct Agent · TabMWP test | accuracy: 59.0% -> 70.0% | Helped |
| Train GPT-4+ and ReAct agents with generated functions | GPT-4+ Agent and ReAct Agent · MATH | accuracy: GPT-4+ Agent: 60.0% -> 66.25%; ReAct Agent: 60.0% -> 77.5% | Mixed |
| Add roll-back and early-stop to agent training (ablation) | GPT-4+ Agent · MATH Number Theory, Intermediate Algebra, and Counting and probability | accuracy: 63.8% -> 67.5%; 36.3% -> 38.8%; 72.5% -> 76.3% | Helped |
| Use progressive function updates instead of one-step function generation (ablation) | GPT-4+ Agent · MATH Number Theory, Intermediate Algebra, and Counting and probability | accuracy: 60.0% -> 67.5%; 28.8% -> 38.8%; 70.0% -> 76.3% | Helped |
| Train agents backed by Code-Llama-34B | Code-Llama-34B agent · MATH | accuracy: 7.5% -> 11.3% | Helped |
| Train agents backed by Mixtral-8x7B | Mixtral-8x7B agent · MATH | accuracy: 23.8% -> 28.8% | Helped |
| Train agents backed by GPT-3.5-turbo-1106 | GPT-3.5-turbo-1106 agent · MATH | accuracy: 25.0% -> 28.8% | Helped |
| Use batch training with larger training sets | GPT-4+ Agent · MATH Intermediate Algebra | accuracy: 7.8% | Mixed |
Surprising findings
- Training without progressive function updates performed worse than the untrained GPT-4+ agent on Intermediate Algebra (28.8% vs 30.0%).
- Batch training with larger training sets did not reliably improve test performance; batch size 20 dropped test accuracy by 7.8%.
Limitations
- Very small training sets (10-20 examples) and test subsets due to OpenAI API cost, so results are noisy.
- Theoretical generalization bound relies on strong assumptions (same distribution, exact training-loss minimizer).
- Training set size is limited by the LLM optimizer's context window, hindering scaling.
Experimental setup
Datasets: MATH TabMWP GAIA
Benchmarks: MATH TabMWP GAIA
Metrics: accuracy correct rate
Baselines: GPT-4+ agent without training ReAct agent without training CREATOR CRAFT Tool-maker
Builds on: ReAct AutoGen CREATOR CRAFT
Closest papers by meaning
- How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.91)
- AgentRefine: Enhancing Agent Generalization through Refinement Tuning (ICLR 2025 · similarity 0.91)
- Self-Challenging Language Model Agents (NeurIPS 2025 · similarity 0.91)
- ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering (ICML 2026 · similarity 0.90)
- FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.90)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.