AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

A Diagnostic Study of Multi-Agent LLMs for Real-World Debates

Priya Pitre, Gaurav Srivastava, Lu Zhang, Le Wang, Naren Ramakrishnan, Xuan Wang

ICML 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a diagnostic framework that evaluates multi-agent LLM debates on process-level failures (e.g., sycophancy, domination, premature convergence) in addition to outcome-based metrics.

Problem

Multi-agent LLM debates are used in policy, politics, and city planning where ground truth is unavailable, but existing evaluations rely on outcome proxies (consensus, majority vote, LLM-as-judge) that can miss failures such as sycophancy, domination, and premature convergence.

Approach

A diagnostic framework that evaluates both debate outcomes and the debate process itself, targeting failure modes that outcome-based proxies miss.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use process-level diagnostics (all process metrics avg) instead of consensusconsensus signal · City Planning human preferencepairwise agreement: 0.57 -> 0.75Improved
use all process metrics instead of LLM-as-JudgeLLM-as-Judge · Politics human preferencepairwise agreement: 0.60 -> 0.74Improved
use average process metrics instead of consensusconsensus signal · MMLU-Pro correctnessSpearman rho: 0.19 -> 0.56Improved
use average process metrics instead of LLM-as-JudgeLLM-as-Judge · MMLU-Pro correctnessSpearman rho: 0.38 -> 0.56Improved
use all process metrics instead of token counttoken count · City Planning human preferencepairwise agreement: 0.52 -> 0.75Improved
high-process debates versus low-process debates when initial majority is wrongmulti-agent debate · MMLU-Procorrect answer recovery: 40% -> 74%Improved
heterogeneous vs homogeneous agents: engagementhomogeneous debate · Politics, process scoreengagement: 0.23 -> 0.24Mixed
heterogeneous vs homogeneous agents: stabilityhomogeneous debate · Politics, process scorestability: 0.94 -> 0.78Hurt
increase group size from 3 to 7 agentsmulti-agent debate · Politics, process scorestability: 0.84 -> 0.79Hurt
increase group size from 3 to 7 agentsmulti-agent debate · Politics, process scorewelfare: 0.51 -> 0.26Hurt
increase group size from 3 to 7 agentsmulti-agent debate · Politics, process scoreinverse asymmetry: 0.66 -> 0.84Improved
use larger model (GPT-4o instead of Qwen-2.5-14b)multi-agent debate · City Planning human preferenceall-process pairwise agreement: 0.72 -> 0.75Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy