Research map / Large language models / Reasoning and chain of thought
Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finetunes a society of LLM generation and critic agents on debate-generated data (each on its own data subset) so that diversity is preserved, enabling multiple rounds of self-improvement on reasoning tasks.
Problem
Single-agent self-improvement via synthetic data plateaus after a few rounds because diversity of reasoning collapses, limiting gains beyond the base model's performance.
Approach
Instead of finetuning one model, finetune a multiagent set of models from the same base, with separate generation and critic agents each trained on independent subsets of debate-generated data (majority-vote-consistent first-round responses for generators; corrected and maintained trajectories for critics), so specialization and diversity are maintained across iterations.
Key results
- On MATH with Phi-3, multiagent finetuning raises accuracy from 58.8% (one iteration) to 66.0% after five iterations; Mistral goes from 22.5% to 28.2%.
- With a single iteration, the method reaches 85.60% on GSM and 60.60% on MATH for GPT-3.5, versus 83.30% and 55.73% for the best baseline (Debate) in those columns.
- LLaMA-3 accuracy on GSM rises from 75.00% (Base) to 88.60% with the method, versus 78.40% for Debate.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multiagent finetuning instead of the strongest listed baseline | GPT-3.5 · Arithmetic | accuracy: 98.40 -> 99.62 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | GPT-3.5 · GSM | accuracy: 83.70 -> 85.60 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | GPT-3.5 · MATH | accuracy: 55.73 -> 60.60 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | Phi-3 · Arithmetic | accuracy: 96.20 -> 99.40 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | Phi-3 · GSM | accuracy: 85.80 -> 88.60 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | Phi-3 · MATH | accuracy: 53.40 -> 58.80 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | Mistral · Arithmetic | accuracy: 19.60 -> 22.60 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | Mistral · GSM | accuracy: 52.60 -> 58.40 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | Mistral · MATH | accuracy: 18.91 -> 22.50 | Helped |
| use multiagent finetuning instead of the strongest listed baseline | LLaMA-3 · GSM | accuracy: 78.40 -> 88.60 | Helped |
| add debate summarization (ablation) | Multiagent FT, GPT-3.5 · GSM | accuracy: 82.20 -> 85.60 | Helped |
| add critic agents (ablation) | Multiagent FT, GPT-3.5 · MATH | accuracy: 50.80 -> 60.60 | Helped |
Surprising findings
- Single-agent finetuning saturates after one iteration and then drops, while the multiagent approach keeps improving.
- Simply giving agents unique ID prompts gives results similar to plain debate, suggesting that finetuning, not prompting, creates the specialization.
Limitations
- Multiagent finetuning is substantially more expensive at training and inference, requiring multiple model copies.
- Experiments use only 500 training questions and modest model sizes; results on larger scales are not shown.
- Some reported improvements are within overlapping standard errors, and the gains over baselines on many settings are modest.
Experimental setup
Datasets: Arithmetic (generated) GSM8K MATH MMLU
Benchmarks: Arithmetic (generated) GSM8K MATH MMLU
Metrics: accuracy standard error consensus-based diversity embedding dissimilarity negative log-likelihood KL divergence
Baselines: Base Majority Debate STaR Majority FT Single-agent FT Unique ID
Builds on: multiagent debate STaR self-consistency
Closest papers by meaning
- Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.89)
- FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.89)
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.89)
- ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvement (ICLR 2025 · similarity 0.89)
- Recursive Introspection: Teaching Language Model Agents How to Self-Improve (NeurIPS 2024 · similarity 0.88)
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.