Research map / Multi-agent systems and games / LLM agents

Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MPDF, a multi-agent LLM deliberation framework where agents learn a decentralized meta-cognitive policy over Persist/Refine/Concede actions, trained with SoftRankPO, a rank-based Gaussian-quantile advantage KL-regularized policy optimization.

Problem

Multi-agent LLM collaboration relies on fixed collaboration protocols and treats agents as passive executors that cannot adapt deliberation to internal confidence or uncertainty; policy gradient training of such policies is unstable under reward-scale and variance heterogeneity.

Approach

Agents select high-level meta-cognitive acts (Persist, Refine, Concede) from a learned policy over a state built from decision schema, reasoning profile, and introspective confidence; SoftRankPO converts group rewards into rank-based normal-quantile advantages so training is invariant to affine reward scale.

Key results

• MPDF with SoftRankPO reaches 55.74% average accuracy across six benchmarks, versus 51.99% for LLM-Debate and 51.60% for AgentPrune in the main table.
• SoftRankPO outperforms GRPO (54.40%) and PPO (50.85%) when applied on top of the SFT policy.
• MPDF uses 47.1k average tokens versus 67.2k for LLM-Debate, a reported 29.96% reduction.

Surprising findings

• Vanilla PPO underperforms the SFT-only variant (50.85% vs 52.09%), which the authors attribute to unstable credit assignment.
• PERSIST rate rises from 19.1% to 78.8% after SoftRankPO training, suggesting agents learn to retain confident outputs rather than revise.

Limitations

• Offline corpus is generated from the SFT policy and held fixed during RL, so the RL stage is not fully online.
• Hyperparameter inconsistency: the text says the default tau is 0.5 in one place and tau=0.8 is best, and the paper reports that tau is sensitive to tuning.

Experimental setup

Datasets: GSM8K MATH AIME AMC MMLU HumanEval

Benchmarks: GSM8K MATH AIME AMC MMLU HumanEval

Metrics: solve rate accuracy pass@1 average accuracy token count

Baselines: Vanilla CoT SC PHP LLM-Debate DyLAN AgentPrune AFlow GPTSwarm

Builds on: PPO GRPO LLM-Debate AFlow AgentPrune

Closest papers by meaning

• Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.92)
• ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.89)
• Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs (ICLR 2026 · similarity 0.88)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.88)
• Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.88)