Research map / Large language models / AI and society
Humans Correct Their Judgements Through Debate, but Weaker AI Models Not
Maria Victoria Carro, Denise Alejandra Mester, Facundo Nieto, Oscar Stanchi, Guido Bergman, Giovanni Franco Gabriel Marraffini, Trinidad Borrell, Guido Freire, Joaquín S Machulsky, Mario Leiva, Ana Lopez, Nicolas Spinelli and 5 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether debate between AI systems helps humans and weaker AI judges correct their judgements, finding humans improve through debate while weaker AI models do not.
Problem
Scalable oversight of AI systems that surpass human intelligence; whether AI Safety via Debate, where two AI debaters argue opposing positions before a judge (human or weaker model), lets judges reach correct decisions.
Approach
Empirical comparison of debate's effect on human judges versus weaker AI judges, showing humans correct their judgements through debate while weaker AI models do not.
Key results
- Human judges improve their judgement accuracy through debate, per the abstract's main finding.
- Weaker AI model judges do not improve their judgements through debate, per the abstract's main finding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add debate before human judgment | human non-expert judges · in-subtopic evaluation | accuracy (%): 41.3 -> 57.61 | Improved |
| use consultancy instead of debate | human judges · consultancy baseline | accuracy (%): 51.25 -> 48.75 | No clear effect |
| add debate before judgment | Claude Sonnet 4 judge (unconditioned) · in-subtopic evaluation | accuracy (%): 85.26 -> 78.95 | Hurt |
| add debate before judgment | GPT-5.2 judge (unconditioned) · in-subtopic evaluation | accuracy (%): 84.85 -> 90.4 | Improved |
| add debate before judgment | Gemma-3-4B-IT judge (unconditioned) · in-subtopic evaluation | accuracy (%): 72.3 -> 47.5 | Hurt |
| add debate before judgment under GPB system prompt | Gemini 2.5 Pro judge · in-subtopic evaluation | accuracy (%): 50.25 -> 60.1 | Improved |
| add debate before judgment under GPB system prompt | Claude Sonnet 4 judge · in-subtopic evaluation | accuracy (%): 52.27 -> 50.75 | No clear effect |
| use confidence-weighted voting for human groups | human judges multi-judge · multi-judge debate | accuracy (%): 45.83 -> 65.28 | Improved |
| add deliberation after debate | human multi-judge groups · multi-human deliberation | accuracy (%): 62.99 -> 62.20 | No clear effect |
| require consensus in LLM deliberation (ablation) | 3 Claude Sonnet 4 judges · multi-judge LLM | accuracy (%): 48.96 -> 44.04 | Hurt |
| use independent voting instead of deliberation (ablation) | 3 Claude Sonnet 4 judges · multi-judge LLM | accuracy (%): 50.51 -> 53.79 | Improved |
| make LLM panel diverse (different models and personas) before deliberation (ablation) | 7-judge LLM panel · multi-judge LLM | accuracy (%): 58.26 -> 64.29 | Improved |
Surprising findings
- Debate helps human judges but not weaker AI judges.
Experimental setup
Metrics: judge accuracy
Builds on: AI Safety via Debate
Closest papers by meaning
- On scalable oversight with weak LLMs judging strong LLMs (NeurIPS 2024 · similarity 0.92)
- AI Debate Aids Assessment of Controversial Claims (NeurIPS 2025 · similarity 0.90)
- Collaborative Disagreement Resolution for Scalable Oversight (ICML 2026 · similarity 0.90)
- Debate Helps Weak-to-Strong Generalization (AAAI 2025 · similarity 0.88)
- When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.88)
- Debating with More Persuasive LLMs Leads to More Truthful Answers (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
