Research map / Multi-agent systems and games / LLM agents
Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks multi-agent debate (MAD) and other prompting strategies with GPT-3.5 on seven medical and reasoning QA datasets, finding MAD not reliably better but more hyperparameter-sensitive, and that tuning agreement intensity lifts Multi-Persona to state-of-the-art.
Problem
Whether multi-agent debate strategies truly improve LLM question answering over simpler prompting or ensembling methods (self-consistency, Medprompt), given no prior systematic comparison and unclear cost-accuracy trade-offs.
Approach
Systematically benchmarks seven prompting/debate systems under many hyperparameter configurations and proposes an agreement-intensity prompt that modulates how much agents agree at the start of a debate, which substantially boosts Multi-Persona accuracy.
Key results
- Medprompt performs best overall (e.g., 0.65 on MedQA), while the best debate system SoM reaches 0.61 on MedQA median configuration.
- Tuning agreement intensity to 90% for Multi-Persona on full MedQA gives a new state-of-the-art for GPT-3.5, with about 15% gain on the 376-question USMLE subset.
- Multi-Persona, which underperforms by about 7% vs Medprompt at default settings, becomes the best protocol after agreement tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune agent agreement intensity in Multi-Persona (ablation) | Multi-Persona · USMLE subset | accuracy: ≈15% | Helped |
| Tune agent agreement intensity in Society of Mind (ablation) | Society of Mind · USMLE subset | accuracy: ≈5% | Helped |
| Tune agent agreement intensity (ablation) | ChatEval · USMLE subset | accuracy | No clear effect |
| Increase prompted agreement intensity (ablation) | Multi-Persona · MedQA and PubMedQA | accuracy | Helped |
| Increase prompted disagreement (ablation) | Multi-Persona · CIAR | accuracy | Helped |
| Use the Multi-Persona debate instead of only the first agent's initial answer | Multi-Persona · All evaluated datasets | accuracy | Hurt |
| Use MAD protocols rather than non-debate prompting strategies | MAD systems · Evaluated datasets | accuracy | Mixed |
| Transfer GPT-3.5 agreement-modulation settings to GPT-4 | Multi-Persona · MedQA | accuracy | Helped |
| Transfer GPT-3.5 agreement-modulation settings to Mixtral 8x7B | Multi-Persona · MedQA | accuracy | Hurt |
Surprising findings
- Multi-Persona's debate reduces accuracy relative to the first agent's initial answer because the devil agent is designed to disagree.
- CIAR follows a reverse agreement pattern: higher disagreement helps on this counter-intuitive dataset.
Limitations
- Relies on a proprietary API model (GPT-3.5) with variable inference and unannounced updates.
- Large-scale API benchmarking is costly, limiting experiment scale and replication.
- Transfer of hyperparameters to Mixtral 8x7B did not hold; GPT-4 experiments limited to a subset.
Experimental setup
Datasets: MedQA PubMedQA MMLU (clinical subset) CosmosQA CIAR GPQA Chess
Benchmarks: MedQA PubMedQA MMLU CosmosQA CIAR GPQA Chess
Metrics: accuracy API cost (USD) time per question tokens per question debate consensus answer change rate
Baselines: Medprompt Self-Consistency Single-agent CoT Society of Minds ChatEval Ensemble Refinement Multi-Persona Solo Performance Prompting
Builds on: Multi-Persona Society of Minds ChatEval Self-consistency Medprompt Ensemble Refinement
Closest papers by meaning
- Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · similarity 0.93)
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? (NeurIPS 2025 · similarity 0.93)
- iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference (AAAI 2026 · similarity 0.92)
- Multi-Agent Debate with Memory Masking (ICLR 2026 · similarity 0.92)
- Key Decision-Makers in Multi-Agent Debates: Who Holds the Power? (AAAI 2026 · similarity 0.91)
- When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.