Research map / Multi-agent systems and games / LLM agents
A Diagnostic Study of Multi-Agent LLMs for Real-World Debates
Priya Pitre, Gaurav Srivastava, Lu Zhang, Le Wang, Naren Ramakrishnan, Xuan Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a diagnostic framework that evaluates multi-agent LLM debates on process-level failures (e.g., sycophancy, domination, premature convergence) in addition to outcome-based metrics.
Problem
Multi-agent LLM debates are used in policy, politics, and city planning where ground truth is unavailable, but existing evaluations rely on outcome proxies (consensus, majority vote, LLM-as-judge) that can miss failures such as sycophancy, domination, and premature convergence.
Approach
A diagnostic framework that evaluates both debate outcomes and the debate process itself, targeting failure modes that outcome-based proxies miss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use process-level diagnostics (all process metrics avg) instead of consensus | consensus signal · City Planning human preference | pairwise agreement: 0.57 -> 0.75 | Improved |
| use all process metrics instead of LLM-as-Judge | LLM-as-Judge · Politics human preference | pairwise agreement: 0.60 -> 0.74 | Improved |
| use average process metrics instead of consensus | consensus signal · MMLU-Pro correctness | Spearman rho: 0.19 -> 0.56 | Improved |
| use average process metrics instead of LLM-as-Judge | LLM-as-Judge · MMLU-Pro correctness | Spearman rho: 0.38 -> 0.56 | Improved |
| use all process metrics instead of token count | token count · City Planning human preference | pairwise agreement: 0.52 -> 0.75 | Improved |
| high-process debates versus low-process debates when initial majority is wrong | multi-agent debate · MMLU-Pro | correct answer recovery: 40% -> 74% | Improved |
| heterogeneous vs homogeneous agents: engagement | homogeneous debate · Politics, process score | engagement: 0.23 -> 0.24 | Mixed |
| heterogeneous vs homogeneous agents: stability | homogeneous debate · Politics, process score | stability: 0.94 -> 0.78 | Hurt |
| increase group size from 3 to 7 agents | multi-agent debate · Politics, process score | stability: 0.84 -> 0.79 | Hurt |
| increase group size from 3 to 7 agents | multi-agent debate · Politics, process score | welfare: 0.51 -> 0.26 | Hurt |
| increase group size from 3 to 7 agents | multi-agent debate · Politics, process score | inverse asymmetry: 0.66 -> 0.84 | Improved |
| use larger model (GPT-4o instead of Qwen-2.5-14b) | multi-agent debate · City Planning human preference | all-process pairwise agreement: 0.72 -> 0.75 | Improved |
Closest papers by meaning
- Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · similarity 0.91)
- When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.90)
- Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024 · similarity 0.90)
- The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimization (ICML 2026 · similarity 0.89)
- Multi-Agent Debate for LLM Judges with Adaptive Stability Detection (NeurIPS 2025 · similarity 0.89)
- Key Decision-Makers in Multi-Agent Debates: Who Holds the Power? (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
