Research map / Multi-agent systems and games / LLM agents
MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains Qwen2.5-Coder-32B-Instruct (MAS-GPT) via SFT on a consistency-oriented dataset of ~11k query-to-executable-MAS code pairs so it generates a query-specific multi-agent system in one inference.
Problem
Existing LLM-based multi-agent systems are either manually designed and query-agnostic, or require many LLM calls to adaptively design an MAS per query, which is costly and inadaptable.
Approach
Reframe MAS construction as a generative language task: an LLM takes a query and outputs an executable Python MAS (forward function) in one inference. Training data is built with inter-consistency (cluster similar queries and assign the same top MAS) and intra-consistency (LLM-refined MAS plus reasoning, accepted only if not worse) selection and refinement.
Key results
- With Llama-3-70B-Instruct as MAS driver, MAS-GPT averages 65.47 across 8 benchmarks, the best among 10 baselines (second best DyLAN 60.54).
- With o1-preview and DeepSeek-R1 drivers, MAS-GPT improves over the single LLM on AIME-2024 by 13.34% and 10.0% respectively.
- On MATH, MAS-GPT scores 77.59 vs ChatDev 62.07 in the appendix comparison; ablation shows removing inter-consistency selection drops MATH by 8.39 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MAS-GPT instead of Self-Consistency | LLM-based MAS · MATH | accuracy: 61.59 -> 68.65 | Helped |
| use MAS-GPT instead of Self-Consistency | LLM-based MAS · GSM8K | accuracy: 94.99 -> 93.39 | Hurt |
| use MAS-GPT instead of Self-Consistency | LLM-based MAS · GSM-Hard | accuracy: 47.20 -> 62.40 | Helped |
| use MAS-GPT instead of the strongest baseline | LLM-based MAS · HumanEval | accuracy: 79.01 -> 80.25 | Helped |
| use MAS-GPT instead of Chain-of-Thought | LLM-based MAS · HumanEval+ | accuracy: 77.02 -> 78.88 | Helped |
| use MAS-GPT instead of DyLAN | LLM-based MAS · MMLU | accuracy: 78.18 -> 78.38 | Helped |
| use MAS-GPT instead of AgentVerse | LLM-based MAS · GPQA | accuracy: 40.19 -> 37.62 | Hurt |
| use MAS-GPT instead of Self-Consistency | LLM-based MAS · SciBench | accuracy: 20.00 -> 24.21 | Helped |
| use inter-consistency-oriented selection rather than random pair selection (ablation) | MAS-GPT dataset construction · MATH | accuracy: 60.26 -> 68.65 | Helped |
| add MAS adjustment during intra-consistency-oriented refinement (ablation) | MAS-GPT dataset construction · MATH | accuracy: 66.23 -> 68.65 | Helped |
| add a reasoning process during intra-consistency-oriented refinement (ablation) | MAS-GPT dataset construction · MATH | accuracy: 64.90 -> 68.65 | Helped |
Surprising findings
- The untrained base model fails to generate an effective MAS zero-shot, so MAS generation needs specific training.
- MAS-GPT beats the math-specialized AFlow on MATH by 3.53%, while AFlow falls below a single LLM on out-of-domain benchmarks.
Limitations
- Training data is built using a fixed pool of about 40 base MAS designs and LLM-generated refinements, limiting diversity.
- Cost analysis treats MAS-GPT inference as 0.5 calls by assuming model size ratio, a rough approximation.
- Evaluated on a limited set of benchmarks with single runs and drivers; generalization to very different tasks is only shown via case studies.
Experimental setup
Datasets: MATH GSM8K MBPP MMLU SciQ GSM-Hard AQUA-RAT
Benchmarks: MATH GSM8K GSM-Hard HumanEval HumanEval+ MMLU GPQA SciBench AIME-2024
Metrics: accuracy
Baselines: Single Chain-of-Thought Self-Consistency LLM-Debate Self-Refine Quality-Diversity SPP AgentVerse GPTSwarm DyLAN
Builds on: AFlow ADAS GPTSwarm DyLAN AgentVerse
Closest papers by meaning
- How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.89)
- MAS$^2$: Self-Generative, Self-Configuring, Self-Rectifying Multi-Agent Systems (ICLR 2026 · similarity 0.89)
- Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.89)
- MASPO: Joint Prompt Optimization for LLM-based Multi-Agent Systems (ICML 2026 · similarity 0.89)
- Agent Primitives: Reuseable Latent Building Blocks for Multi-Agent Systems (ICML 2026 · similarity 0.89)
- Large Language Models Miss the Multi-agent Mark (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.