Research map / Large language models / Agents, code and math

Mixture-of-Agents Enhances Large Language Model Capabilities

ICLR 2025 · Large language models › Agents, code and math · approach: Long-horizon LLM agents · 26 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Mixture-of-Agents, a layered prompt-only architecture where each LLM agent refines its answer using the outputs of the previous layer's agents, reaching 65.1% LC win rate on AlpacaEval 2.0 with open-source models.

Problem

Single LLMs have limits in size, training data, and skill specialization, and scaling them is costly; the question is how to harness the collective strengths of multiple LLMs to produce better responses.

Approach

Exploits "collaborativeness" of LLMs: a model produces better responses when shown other models' outputs, even weaker ones. MoA stacks layers of multiple proposer LLMs, each layer's agents see all prior-layer outputs via an aggregate-and-synthesize prompt, requiring no fine-tuning; it is framed as a model-level analogue of Mixture-of-Experts.

Key results

• MoA using only open-source models reaches 65.1% LC win rate on AlpacaEval 2.0 vs 57.5% for GPT-4 Omni (05/13).
• MoA-Lite (2 layers, Qwen1.5-72B aggregator) reaches 59.3% LC win rate, still above GPT-4 Omni.
• On MT-Bench, MoA w/ GPT-4o averages 9.40 and MoA averages 9.25, both ranking at or above GPT-4 Turbo (9.31).

Measured effects reported in the paper

Surprising findings

• Models generate better responses when given outputs from other models, even when those outputs are of lower quality than their own.
• Using multiple different LLMs as proposers consistently beats a single model sampled multiple times at temperature 0.7 for the same number of proposals.

Limitations

• Iterative layered aggregation delays the first token, giving high time to first token (TTFT).
• Improvements on MT-Bench are relatively incremental since top models already score above 9/10.
• MoA is more verbose and less concise on FLASK.

Experimental setup

Datasets: AlpacaEval 2.0 instructions MT-Bench FLASK MATH

Benchmarks: AlpacaEval 2.0 MT-Bench FLASK MATH

Metrics: length-controlled win rate win rate MT-Bench score FLASK skill scores accuracy Spearman correlation

Baselines: GPT-4 Omni (05/13) GPT-4 Turbo (04/09) GPT-4 Preview (11/06) WizardLM 8x22B Qwen1.5 110B Chat LLM ranker

Builds on: Mixture-of-Experts Chain-of-Thought PairRanker FrugalGPT

Closest papers by meaning

• Improving Model Alignment Through Collective Intelligence of Open-Source Models (ICML 2025 · similarity 0.91)
• Chain of Agents: Large Language Models Collaborating on Long-Context Tasks (NeurIPS 2024 · similarity 0.89)
• $\texttt{Model-GLUE}$: Democratized LLM Scaling for A Large Model Zoo in the Wild (NeurIPS 2024 · similarity 0.89)
• Knowledge Fusion of Large Language Models (ICLR 2024 · similarity 0.89)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment (ICLR 2025 · similarity 0.88)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.88)