Research map / Multi-agent systems and games / LLM agents
Collaborative Disagreement Resolution for Scalable Oversight
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Disagreement Resolution, a collaborative multi-turn consultant protocol inspired by mediation that lets two expert LLMs revise answers toward consensus or isolate a crux, and shows it beats adversarial Debate for weak LLM judges on GPQA, SuperGPQA and HLE-MC.
Problem
Debate-based scalable oversight incentivizes persuasion over truth and often fails when the judge is weaker than the debaters in capability-asymmetric settings.
Approach
Replace adversarial fixed-position debate with a collaborative protocol where consultants identify cruxes, update beliefs, and may retain or adopt the opponent's answer each turn, so the judge verifies a consensus or a narrowed crux rather than adjudicating full arguments.
Key results
- DR achieves 62.1% judge accuracy versus 49.2% for Debate averaged over 12 conditions (+12.9 points).
- With GLM-4.6 vs Kimi K2 Thinking consultants, DR beats Debate by +21.3 points on average; with GPT-4o vs Claude Sonnet 4 the gain is +4.3.
- DR outperforms Debate in 10 of 12 conditions; gains are largest for the weakest judge (+6.3 on average).
Surprising findings
- Double Consultancy outperforms both interactive protocols on HLE-MC with the GPT-4o vs Claude pair.
- Persuasiveness (Best-of-N) and sycophancy interventions do not meaningfully improve DR performance.
Limitations
- Evaluation is inference-time only, without training with the protocol.
- Focuses on naturally occurring disagreements where ground truth may be ill-defined.
- Assumes consultants are honest and follow instructions; dishonest consultants make DR and Debate similarly vulnerable.
Experimental setup
Datasets: GPQA SuperGPQA HLE
Benchmarks: GPQA SuperGPQA HLE-MC
Metrics: judge accuracy bootstrapped 95% confidence interval calibration score (Brier) exit-mode proportions
Baselines: Debate Double Consultancy Naive Judge
Builds on: debate double crux consultancy
Closest papers by meaning
- On scalable oversight with weak LLMs judging strong LLMs (NeurIPS 2024 · similarity 0.91)
- When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.89)
- AI Debate Aids Assessment of Controversial Claims (NeurIPS 2025 · similarity 0.86)
- Debating with More Persuasive LLMs Leads to More Truthful Answers (ICML 2024 · similarity 0.86)
- Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024 · similarity 0.86)
- Multi-Agent Debate for LLM Judges with Adaptive Stability Detection (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.