Research map / Multi-agent systems and games / LLM agents
Multi-LLM Debate: Framework, Principals, and Interventions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a latent-concept theoretical framework for multi-LLM debate, proves echo-chamber and shared-misconception effects, and introduces three interventions (diversity pruning, quality pruning, misconception refutation) that improve accuracy on BoolQ, MMLU, TruthfulQA and MathQ.
Problem
Multi-agent LLM debate can become static or converge to majority or shared-misconception answers, and its underlying mechanics are not well understood theoretically.
Approach
Views debate as Bayesian inference over latent concepts (in-context-learning style), so that other agents' responses act as a skew reweighting the next-round generation; from this, derives tyranny-of-majority and shared-misconception results and designs pruning and refutation interventions to counter them.
Key results
- Combined interventions with 6 GPT-3.5 agents raise MMLU accuracy from 0.74 (SoM) to 0.79 and BoolQ from 0.84 to 0.85.
- On MathQ with 6 Llama-3 agents, accuracy rises from 0.33 (SoM) to 0.48 (Ours).
- Mixed-family debate (3x GPT-3.5 + 3x Llama-3) gives TruthfulQA 0.62 vs 0.56 for SoM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine quality pruning, diversity pruning, and misconception refutation | SoM debate with 6× GPT-3.5 · BoolQ | accuracy: .84±.012 -> .85±.012 | Helped |
| combine quality pruning, diversity pruning, and misconception refutation | SoM debate with 6× GPT-3.5 · MMLU | accuracy: .74±.016 -> .79±.014 | Helped |
| combine quality pruning, diversity pruning, and misconception refutation | SoM debate with 6× GPT-3.5 · TruthfulQA | accuracy: .63±.032 -> .69±.030 | Helped |
| combine quality pruning, diversity pruning, and misconception refutation | SoM debate with 6× GPT-3.5 · MathQ | accuracy: .88±.016 -> .93±.01 | Helped |
| add misconception refutation (ablation) | combined debate interventions, 6× GPT-3.5 · BoolQ | accuracy: .83±.013 -> .85±.012 | Helped |
| add diversity pruning (ablation) | combined debate interventions, 6× GPT-3.5 · BoolQ | accuracy: .84±.012 -> .85±.012 | Helped |
| add quality pruning (ablation) | combined debate interventions, 6× GPT-3.5 · BoolQ | accuracy: .84±.012 -> .85±.012 | Helped |
| use misconception refutation alone rather than the combined interventions (ablation) | 6× GPT-3.5 debate · MMLU | accuracy: .79±.014 -> .75±.015 | Hurt |
| use diversity pruning alone rather than the combined interventions (ablation) | 6× GPT-3.5 debate · MMLU | accuracy: .79±.014 -> .72±.016 | Hurt |
| use quality pruning alone rather than the combined interventions (ablation) | 6× GPT-3.5 debate · MMLU | accuracy: .79±.014 -> .73±.016 | Hurt |
| use misconception refutation alone rather than the combined interventions (ablation) | 6× GPT-3.5 debate · TruthfulQA | accuracy: .69±.030 -> .65±.032 | Hurt |
| use diversity pruning alone rather than the combined interventions (ablation) | 6× GPT-3.5 debate · TruthfulQA | accuracy: .69±.030 -> .58±.034 | Hurt |
Surprising findings
- Applying individual interventions in isolation can perform worse than single-model accuracy, though the combination performs best.
- Diversity and quality pruning are less effective on MathQ because sentence embeddings of arithmetic expressions are similar regardless of correctness.
Limitations
- Theory relies on a latent concept space that is not directly accessible, so sentence embeddings are used as a proxy.
- Pruning and diversity interventions are less meaningful for arithmetic tasks where embeddings are similar regardless of correctness.
- Interventions increase inference cost, especially misconception refutation.
Experimental setup
Datasets: BoolQ MMLU (high-school subset, 3,406 questions) TruthfulQA MathQ (3,000 arithmetic questions)
Benchmarks: BoolQ MMLU TruthfulQA MathQ
Metrics: accuracy probability of echoing majority answer accuracy improvement vs response similarity
Baselines: Single model Society of Minds debate (SoM) Ablations: misconception refutation only, diversity pruning only, quality prunin
Builds on: Society of Minds debate (Du et al. 2023) in-context learning as Bayesian inference (Xie et al. 2021) latent concept framework (Jiang 2023)
Closest papers by meaning
- Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024 · similarity 0.93)
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? (NeurIPS 2025 · similarity 0.93)
- Debating with More Persuasive LLMs Leads to More Truthful Answers (ICML 2024 · similarity 0.91)
- When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.91)
- Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate (NeurIPS 2026 · similarity 0.90)
- Multi-Agent Debate for LLM Judges with Adaptive Stability Detection (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.