Research map / Multi-agent systems and games / LLM agents

Reward Modeling for Multi-Agent Orchestration

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Orch-RM, a self-supervised Bradley-Terry reward model trained on win-lose orchestration pairs mined from MAS-Orchestra training artifacts, used for best-of-N selection of orchestrations and for reward-guided orchestrator training.

Problem

Training LLM orchestrators for multi-agent systems is limited by scarce orchestration-quality supervision and the high cost of executing full sub-agent rollouts for feedback.

Approach

Construct win-lose pairs from intermediate artifacts (specialized-over-base and correct-over-incorrect orchestrations) and train a Bradley-Terry reward model that scores orchestrations before any sub-agent execution, then use it for BoN test-time selection and as the GRPO reward for orchestrator training.

Key results

• On AIME 24&25, Orch-RM best-of-8 accuracy reaches 68.33% vs 63.33% for majority vote/MAS-Orchestra and 66.67% for the best GPT judge, at 2.38M verification tokens.
• On BrowseComp+, Orch-RM reaches 14.00% best-of-8 vs 12.50% for GPT-5-mini judge, while using 8.26M tokens vs 142.80M for the trajectory-level judge.
• In continued training, Orch-RM improves AIME majority-vote accuracy to 68.33% (from 63.33%) and BrowseComp+ to 11.00% (from 9.50%), with up to ~10x fewer training tokens than GRPO on AIME.

Surprising findings

• Gains are small or absent on HotpotQA and GPQA, where orchestration diversity is limited and orchestration-level verification is less discriminative.
• Naively mixing heterogeneous domains into one reward model added noise, motivating separate domain-specific reward models.

Limitations

• Performance constrained by diversity and quality of available orchestrator models and sampled trajectories.
• Domain-specific reward models only; no unified cross-domain RM.
• Limited BrowseComp+ training data due to high rollout cost; gains small on HotpotQA and GPQA.

Experimental setup

Datasets: DeepScaleR AIME 24&25 BrowseComp+ HotpotQA GPQA MAS-Orchestra training logs

Benchmarks: AIME 24&25 BrowseComp+ HotpotQA GPQA

Metrics: accuracy majority-vote accuracy verification tokens (M) training tokens average normalized advantage

Baselines: MAS-Orchestra GRPO RFT DPO LLM-as-a-Judge Skywork-Reward models log P majority vote GPT-5-mini GPT-5.2 GPT-4.1-mini GPT-5.4-mini

Builds on: MAS-Orchestra Skywork-Reward-LLaMA-3.1-8B GRPO DPO Bradley-Terry model

Closest papers by meaning

• MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks (ICML 2026 · similarity 0.91)
• Learning Latency-Aware Orchestration for Multi-Agent Systems (NeurIPS 2026 · similarity 0.91)
• Multi-Agent Collaboration via Evolving Orchestration (NeurIPS 2025 · similarity 0.90)
• Learning to Orchestrate Agents in Natural Language with the Conductor (ICLR 2026 · similarity 0.89)
• MAS$^2$: Self-Generative, Self-Configuring, Self-Rectifying Multi-Agent Systems (ICLR 2026 · similarity 0.88)
• LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination (ICML 2025 · similarity 0.87)