Research map / Multi-agent systems and games / LLM agents
Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Disentangles multi-agent debate (MAD) into majority voting and inter-agent debate, showing voting explains most gains, and models debate as a martingale over belief trajectories, with biased-update interventions improving debate.
Problem
Unclear whether the gains of multi-agent debate (MAD) for LLMs come from inter-agent communication or simply from aggregating multiple agents' answers via majority voting.
Approach
Compares MAD against majority voting with the same agents (T=0), gives a Dirichlet-Compound-Multinomial Bayesian model showing debate induces a martingale over expected correctness, and proposes correction-biased update rules (MAD-Conformist, MAD-Follower, MAD-oracle) to improve debate.
Key results
- Majority voting with 5 agents averages 0.7691 accuracy on Qwen2.5-7B-Instruct vs. 0.7377 for the best decentralized MAD setting (T=2), with voting best on most benchmarks.
- On Llama3.1-8B-Instruct, majority voting averages 0.7242 versus at most 0.6990 for any MAD configuration.
- MAD-oracle (locking correct answers) raises Decentralized MAD (T=5) average accuracy from 0.7084 to 0.8289 on Qwen2.5-7B-Instruct, while MAD-Conformist and MAD-Follower reach about 0.75-0.76.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use majority voting on initial agent responses instead of iterative debate | Qwen2.5-7B-Instruct multi-agent system · Arithmetics and GSM8K | accuracy | Helped |
| Use majority voting on initial agent responses instead of iterative debate | Qwen2.5-7B-Instruct multi-agent system · MMLU (Pro.Med.), MMLU (Form.Log.), HellaSwag, CommonsenseQA, and HH-RLHF | accuracy | Mixed |
| Use majority voting on initial agent responses instead of iterative debate | Llama3.1-8B-Instruct multi-agent system · Arithmetics, GSM8K, MMLU (Pro.Med.), MMLU (Form.Log.), HellaSwag, CommonsenseQA, and HH-RLHF | accuracy | Mixed |
| Add oracle locking so an agent retains a correct answer in later rounds (ablation) | Decentralized MAD (T=5) · MMLU (Form.Log.) | accuracy: 0.5000 to 0.6825 | Helped |
| Add majority-conformist updates that retain an agent's response when it matches the previous-round majority (ablation) | MAD-vanilla · Qwen2.5-7B-Instruct; reported benchmarks across decentralized and sparse MAD rounds | accuracy | Helped |
| Add 30% probability of adopting the previous-round majority response (ablation) | MAD-vanilla · Qwen2.5-7B-Instruct; reported benchmarks across decentralized and sparse MAD rounds | accuracy | Helped |
| Increase the number of participating debate agents from N=1 toward N=5 (ablation) | Qwen2.5 multi-agent debate · Benchmarks in the agent-count ablation | accuracy | Helped |
| Increase the number of debate rounds (ablation) | Qwen2.5-7B-Instruct decentralized MAD · CNN/DailyMail | ROUGE-1 | Mixed |
| Increase the number of debate rounds (ablation) | Qwen2.5-7B-Instruct decentralized MAD · CNN/DailyMail | ROUGE-L | No clear effect |
Surprising findings
- Majority voting without any debate performs on par with or better than multi-agent debate in most settings.
- Adding more debate rounds often lowers MAD accuracy relative to fewer rounds.
Limitations
- Theoretical model assumes homogeneous, fully connected agents and an idealized Dirichlet-multinomial belief process.
- Interventions require approximations (majority vote as proxy); oracle variant needs ground truth and is not practical.
- Open-ended evaluation is limited to 30 summarization samples with best-agent reporting.
Experimental setup
Datasets: Arithmetics GSM8K MMLU Professional Medicine MMLU Formal Logic HellaSwag CommonsenseQA HH-RLHF CNN/DailyMail
Benchmarks: Arithmetics GSM8K MMLU Professional Medicine MMLU Formal Logic HellaSwag CommonsenseQA HH-RLHF CNN/DailyMail
Metrics: accuracy ROUGE-1 ROUGE-L
Baselines: Decentralized MAD Sparse MAD Centralized MAD Majority Voting single-agent baseline MAD-vanilla
Builds on: multi-agent debate Polya urn scheme sparse MAD centralized MAD
Closest papers by meaning
- Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024 · similarity 0.93)
- Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · similarity 0.93)
- When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.91)
- Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate (ICML 2026 · similarity 0.91)
- Multi-Agent Debate with Memory Masking (ICLR 2026 · similarity 0.91)
- iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference (AAAI 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.