AI Paper Atlas

Research map / Large language models / AI and society

Humans Correct Their Judgements Through Debate, but Weaker AI Models Not

Maria Victoria Carro, Denise Alejandra Mester, Facundo Nieto, Oscar Stanchi, Guido Bergman, Giovanni Franco Gabriel Marraffini, Trinidad Borrell, Guido Freire, Joaquín S Machulsky, Mario Leiva, Ana Lopez, Nicolas Spinelli and 5 more

NeurIPS 2026 · Large language models › AI and society · approach: Human-centered trust and agency

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies whether debate between AI systems helps humans and weaker AI judges correct their judgements, finding humans improve through debate while weaker AI models do not.

Problem

Scalable oversight of AI systems that surpass human intelligence; whether AI Safety via Debate, where two AI debaters argue opposing positions before a judge (human or weaker model), lets judges reach correct decisions.

Approach

Empirical comparison of debate's effect on human judges versus weaker AI judges, showing humans correct their judgements through debate while weaker AI models do not.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add debate before human judgmenthuman non-expert judges · in-subtopic evaluationaccuracy (%): 41.3 -> 57.61Improved
use consultancy instead of debatehuman judges · consultancy baselineaccuracy (%): 51.25 -> 48.75No clear effect
add debate before judgmentClaude Sonnet 4 judge (unconditioned) · in-subtopic evaluationaccuracy (%): 85.26 -> 78.95Hurt
add debate before judgmentGPT-5.2 judge (unconditioned) · in-subtopic evaluationaccuracy (%): 84.85 -> 90.4Improved
add debate before judgmentGemma-3-4B-IT judge (unconditioned) · in-subtopic evaluationaccuracy (%): 72.3 -> 47.5Hurt
add debate before judgment under GPB system promptGemini 2.5 Pro judge · in-subtopic evaluationaccuracy (%): 50.25 -> 60.1Improved
add debate before judgment under GPB system promptClaude Sonnet 4 judge · in-subtopic evaluationaccuracy (%): 52.27 -> 50.75No clear effect
use confidence-weighted voting for human groupshuman judges multi-judge · multi-judge debateaccuracy (%): 45.83 -> 65.28Improved
add deliberation after debatehuman multi-judge groups · multi-human deliberationaccuracy (%): 62.99 -> 62.20No clear effect
require consensus in LLM deliberation (ablation)3 Claude Sonnet 4 judges · multi-judge LLMaccuracy (%): 48.96 -> 44.04Hurt
use independent voting instead of deliberation (ablation)3 Claude Sonnet 4 judges · multi-judge LLMaccuracy (%): 50.51 -> 53.79Improved
make LLM panel diverse (different models and personas) before deliberation (ablation)7-judge LLM panel · multi-judge LLMaccuracy (%): 58.26 -> 64.29Improved

Surprising findings

Experimental setup

Metrics: judge accuracy

Builds on: AI Safety via Debate

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy