Research map / Multi-agent systems and games / LLM agents

MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a Qwen-2.5-7B orchestrator with GRPO to emit a complete multi-agent system in one function-calling step, and introduces MASBench, a five-axis controlled benchmark for when MAS beats single-agent systems.

Problem

Automatic multi-agent system (MAS) design relies on sequential, code-level orchestration that scales poorly, and it is unclear when MAS actually beat single-agent systems (SAS).

Approach

Formulate MAS orchestration as function-calling RL in which sub-agents are black-box callable functions and the orchestrator emits the entire MAS (via create_agent and create_flow) in a single decision step, trained with GRPO on a rule-based final-answer reward; a controlled five-axis benchmark (Depth, Horizon, Breadth, Parallel, Robustness) is used to analyze MAS vs. SAS.

Key results

• On public benchmarks, MAS-Orchestra reaches 66.25 on AIME24 and 61.25 on AIME25 (avg@8), above the best listed baseline AFlow at 62.50 and 53.33.
• On BrowseComp+, MAS-Orchestra scores 11.00 versus 8.56 for DeepResearchAgent and 1.38 for ToolOrchestra; HotpotQA 49.00 vs 46.44 for DeepResearchAgent.
• MAS outperform SAS on the Robustness axis, where SAS collapses to near-zero accuracy under adversarial notes, while MAS gains diminish on Depth/Horizon/Breadth/Parallel when the sub-agent is stronger (GPT-OSS-120B).

Surprising findings

• Instruction-tuned LLM orchestrators outperform reasoning LLM orchestrators, which tend to solve tasks themselves and delegate to only one simple sub-agent.
• MAS gains diminish when sub-agents are stronger, so MAS benefit depends on the edge of sub-agent competence rather than holding universally.

Limitations

• Low DoM is restricted to at most one sub-agent, and the benchmark uses synthetic iGSM-derived math tasks, limiting generality.
• Robustness gains require explicit adversarial data in training; without it robustness stays near zero.
• Reasoning-model orchestrators underperform and high reasoning effort causes context-length overflow.

Experimental setup

Datasets: MASBench (iGSM-derived) RULER NIAH DeepScaleR HotpotQA BrowseComp+ AIME24 AIME25 GPQA

Benchmarks: AIME24 AIME25 GPQA HotpotQA BrowseComp+ MASBench

Metrics: accuracy (avg@8)

Baselines: CoTAgent SCAgent DebateAgent ReflexionAgent DeepResearchAgent AFlow MaAS MAS-Zero MAS-GPT ToolOrchestra GPT-5 Claude-Sonnet-4.5

Builds on: GRPO iGSM RULER MAS-GPT ToolOrchestra

Closest papers by meaning

• Reward Modeling for Multi-Agent Orchestration (NeurIPS 2026 · similarity 0.91)
• Learning Latency-Aware Orchestration for Multi-Agent Systems (NeurIPS 2026 · similarity 0.91)
• MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems (ICML 2026 · similarity 0.90)
• The Illusion of Multi-Agent Advantage (NeurIPS 2026 · similarity 0.90)
• MAS$^2$: Self-Generative, Self-Configuring, Self-Rectifying Multi-Agent Systems (ICLR 2026 · similarity 0.89)
• Recognize Your Orchestrator: An Entropy Dynamics Perspective for LLM Multi-Agent Systems (ICML 2026 · similarity 0.88)