Research map / Multi-agent systems and games / LLM agents
TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TeamTR, a stage-wise trust-region fine-tuning method for shared-context multi-LLM teams that resamples rollouts after each agent update and bounds per-agent token-level KL, with theory claiming stale-occupancy penalties scale quadratically in team size versus linearly.
Problem
Sequential fine-tuning of shared-context multi-agent LLM teams suffers from occupancy shift: updating one agent changes the context distribution for others, and reusing cached rollouts causes compounding mismatch and coordination regressions.
Approach
Resample trajectories under the partially updated team after each component update, enforce per-agent token-decomposed (reverse) KL trust regions, and derive per-update and per-stage improvement lower bounds with an O(n) rather than O(n^2) occupancy-shift penalty.
Key results
- TeamTR (3x Qwen3-8B homogeneous) reaches 88.1% on AIME24 vs 71.1% for naive sequential training and 68.7% for debate.
- On MATH-500 with n=8 Qwen3-1.7B agents, TeamTR reaches 87.9% vs 58.7% for naive sequential training.
- Plug-and-play swap with Stage-0 alignment yields +27% on AIME24 over direct swap after replacement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add intermediate-occupancy resampling plus trust region (full TeamTR) vs naive sequential (ablation) | 3x Qwen3-8B sequential team · AIME24 | avg accuracy (%): 88.1 vs 71.1 | Helped |
| resample every 2 updates instead of every update (ablation) | TeamTR · AIME24 | avg accuracy (%): -8.8 points vs full TeamTR | Hurt |
| remove trust region (delta to infinity) (ablation) | TeamTR · AIME24 | avg accuracy (%): 68.3 vs 88.1 | Hurt |
| KL penalty only with adaptive beta instead of full trust region enforcement (ablation) | TeamTR · AIME24 | avg accuracy (%): -3.2 points vs full TeamTR | Hurt |
| fixed KL penalty coefficient instead of adaptive trust region (ablation) | TeamTR · AIME24 | avg accuracy (%): 80.7 vs 88.1 | Hurt |
| TeamTR training (resampling + trust region) vs naive sequential fine-tuning on heterogeneous 8B+14B+32B team | sequential team of 3 agents · AIME24 | avg accuracy (%): 92.5 vs 77.8 | Helped |
| TeamTR vs single-agent GRPO/DAPO fine-tuning on Qwen3-8B | Qwen3-8B team · AIME24 | avg accuracy (%): 88.1 vs 41.3 (best single-agent DAPO) | Helped |
| TeamTR vs multi-agent debate baselines | 3x Qwen3-8B team · AIME24 | avg accuracy (%): 88.1 vs 71.1 (Debate+Judge) | Helped |
| TeamTR training vs naive sequential in n=8 scaling | homogeneous Qwen3-1.7B team · MATH-500 (n=8 agents) | accuracy (%): 87.9 vs 58.7 | Helped |
| stale-occupancy gap grows with team size under naive sequential updates | naive sequential training · MATH-500 scaling | fitted exponent of stale surrogate gap: exponent 1.94 (quadratic-like) vs 1.07 for TeamTR | No clear effect |
| TeamTR heterogeneous (1.7B+8B+14B) vs homogeneous (3x8B) | TeamTR team composition · AIME24 / ZebraLogic / MATH-500 / AutoLogi | accuracy (%): +1.6 AIME24, +1.6 ZebraLogic, -1.2 MATH-500, -1.8 AutoLogi | Mixed |
| stage-0 aligned swap of a 1.5B agent for Qwen3-8B vs direct swap | plug-and-play replacement in Qwen2.5 team · AIME24 | final accuracy: +27% post-swap final performance; direct swap causes -18% drop | Helped |
Surprising findings
- Multi-agent debate and role-play teams of 3x Qwen3-8B underperform single-agent-trained baselines on several benchmarks in the reported table, and naive sequential fine-tuning degrades sharply as team size grows.
- Direct swap of a component causes an immediate 18% drop on AIME24 before recovery.
Limitations
- Formal guarantees are restricted to a single-active-agent turn-taking protocol and do not cover concurrent or branching interaction.
- Empirical certificate is a conservative training diagnostic whose tightness depends on a proxy for the estimation error term and sampled token-KL monitoring.
Experimental setup
Datasets: DeepScaleR DAPO ARBench training set PlanBench training set
Benchmarks: AIME 2024 AIME 2025 MATH-500 ZebraLogic AutoLogi ARBench PlanBench tau-bench
Metrics: pass@K avg@K accuracy KL out-of-region rate stale-occupancy gap
Baselines: GPT-4o-mini o4-mini Qwen3-30B-A3B QwQ-32B Qwen3-32B thinking DeepSeek-R1 Distill 70B GRPO DAPO Debate Debate + Judge Role-play Chain-of-Thought
Builds on: HATRPO HAPPO GRPO PPO DAPO
Closest papers by meaning
- Multi-Agent Teams Hold Experts Back (ICML 2026 · similarity 0.89)
- AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systems (NeurIPS 2025 · similarity 0.89)
- Improving the Efficiency of Language Agent Teams with Adaptive Task Graphs (NeurIPS 2026 · similarity 0.89)
- Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.88)
- LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination (ICML 2025 · similarity 0.88)
- SafeSieve: From Heuristics to Experience in Progressive Pruning for LLM-based Multi-Agent Communication (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.