Research map / Large language models / Language, safety and interpretability
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-agent debate, where several LLM instances independently answer, then iteratively critique and update answers over rounds to reach a consensus, improving reasoning and factuality in black-box settings.
Problem
LLMs produce hallucinated facts and reasoning errors when answering from a single model instance; single-agent prompting methods such as chain-of-thought and self-reflection have limited gains.
Approach
Multiple copies of a language model independently generate answers, then each agent reads other agents' responses and updates its own answer over several rounds, converging to a consensus. Uses only black-box access with identical prompts across tasks.
Key results
- On arithmetic, multi-agent debate reaches 81.8% vs 67.0% for a single agent (3 agents, 2 rounds).
- On MMLU (100 questions), debate reaches 71.1% vs 63.9% for a single agent; on biographies it reaches 73.8% vs 66.0%.
- On GSM8K, debate reaches 85.0% vs 77.0% for a single agent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-agent debate instead of single-agent reflection | gpt-3.5-turbo-0301 · Arithmetic | accuracy: 72.1 ± 4.5 -> 81.8 ± 2.3 | Helped |
| Use multi-agent debate instead of majority voting | gpt-3.5-turbo-0301 · GSM8K | accuracy: 81.0 ± 3.9 -> 85.0 ± 3.5 | Helped |
| Use multi-agent debate instead of majority voting | gpt-3.5-turbo-0301 · Chess Move Prediction | pawn score (ΔPS): 102.2 ± 6.2 -> 122.9 ± 7.6 | Helped |
| Use multi-agent debate instead of single-agent reflection | gpt-3.5-turbo-0301 · Biographies | accuracy: 68.3 ± 2.9 -> 73.8 ± 2.3 | Helped |
| Use multi-agent debate instead of single-agent generation | gpt-3.5-turbo-0301 · MMLU | accuracy: 63.9 ± 4.8 -> 71.1 ± 4.6 | Helped |
| Use multi-agent debate instead of single-agent reflection | gpt-3.5-turbo-0301 · Chess Move Validity | validity: 38.8 ± 2.9 -> 45.2 ± 2.9 | Helped |
| Use different persona-based initialization prompts for agents (ablation) | Multi-Agent (Debate) · MMLU | accuracy: 71.1 to 74.2 | Helped |
| Increase the number of debate agents (ablation) | Multi-Agent (Debate) · Arithmetic | accuracy | Helped |
| Increase debate rounds (ablation) | Multi-Agent (Debate) · Arithmetic | accuracy | Helped |
| Increase debate rounds beyond four (ablation) | Multi-Agent (Debate) · Arithmetic | accuracy | No clear effect |
| Use longer debate prompts rather than short prompts (ablation) | Multi-Agent (Debate) · Reasoning tasks | final consensus accuracy | Helped |
| Summarize other agents' responses instead of concatenating them (ablation) | Multi-Agent (Debate) · Reasoning tasks | performance | Helped |
Surprising findings
- Debate can converge on a consensus that is confidently wrong; models affirm incorrect answers as consistent with other agents.
- Reflection baselines performed poorly on factuality tasks, e.g. 57.7% on MMLU versus 63.9% for a single agent.
Limitations
- Multi-agent debate requires multiple generations and rounds, increasing compute cost.
- Longer debates exceed what current models can fully process, with focus on most recent generations.
- Converged answers are not always correct and models confidently affirm wrong consensus.
Experimental setup
Datasets: GSM8K MMLU BIG-Bench Chess-State Tracking Benchmark
Benchmarks: GSM8K MMLU BIG-Bench Chess-State Tracking Benchmark Biographies Arithmetic Chess move prediction
Metrics: accuracy Stockfish pawn score advantage percentage of ground-truth bullets validated
Baselines: Single Agent Single Agent (Reflection) Multi-Agent (Majority) zero-shot chain-of-thought
Builds on: chain-of-thought prompting self-consistency self-reflection
Closest papers by meaning
- Calibrating Reasoning in Language Models with Internal Consistency (NeurIPS 2024 · similarity 0.91)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.90)
- Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance (NeurIPS 2024 · similarity 0.90)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.90)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.