Research map / Multi-agent systems and games / LLM agents

Multi-Agent Debate for LLM Judges with Adaptive Stability Detection

NeurIPS 2025 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a multi-agent debate framework for LLM judges with a time-varying Beta-Binomial mixture model and KS-test-based adaptive stopping, reporting accuracy gains over majority voting on six judging benchmarks.

Problem

LLM-as-judge ensembles typically aggregate independent judgments via static methods like majority voting, which can fail even when individual agents are correct; iterative debate is also costly without a principled stopping rule.

Approach

Models debate as Bayesian belief updating over latent concepts, proves debate improves over majority vote under assumptions, and tracks the judge-correctness distribution with a time-varying two-component Beta-Binomial mixture fit by EM, halting when KS distance between consecutive rounds stays below 0.05 for two rounds.

Key results

• Debate with 7 agents improves over majority voting (SoM) on LLMBar for Gemini-2.0-Flash, from 77.75% to 81.83%.
• On JudgeBench, Gemini-2.0-Flash debate reaches 68.06% versus 66.13% for SoM and 63.66% for single sampling.
• Debate underperforms SoM on BIG-Bench for Qwen-2.5-7B (72.20% vs 76.60%) and Llama-3.1-8B (74.00% vs 81.80%), so gains are not uniform.

Measured effects reported in the paper

Surprising findings

• Debate underperforms majority voting on BIG-Bench for Qwen-2.5-7B and Llama-3.1-8B, contrary to the framing that debate generally helps.
• Diversity pruning did not improve accuracy as claimed in prior work, which the authors attribute to limited answer spaces in judgment tasks.

Limitations

• Theoretical guarantees rely on strong assumptions (conditional independence, true-concept predictiveness) and the proof's inequalities are loosely justified with epsilon terms.
• Debate gains are inconsistent across models and datasets, and debate underperforms majority voting on several benchmarks.
• Small evaluation subsets (e.g., 180 JudgeAnything entries, sampled 1,000 MLLM-Judge) and large standard errors limit statistical confidence in reported gains.

Experimental setup

Datasets: BIG-Bench sports understanding JudgeBench LLMBar TruthfulQA MLLM-Judge JudgeAnything

Benchmarks: BIG-Bench sports understanding JudgeBench LLMBar TruthfulQA MLLM-Judge JudgeAnything

Metrics: accuracy standard error rounds until stopping

Baselines: Single model SoM (majority vote) MAD RECONCILE

Builds on: LLM-as-a-Judge Multi-agent debate Self-consistency RECONCILE

Closest papers by meaning

• Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models? (NeurIPS 2025 · similarity 0.90)
• When Debate Helps: Proposal Supply and Verification-Aware Readout in Multi-Agent Reasoning (NeurIPS 2026 · similarity 0.90)
• Multi-LLM Debate: Framework, Principals, and Interventions (NeurIPS 2024 · similarity 0.90)
• Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate (ICML 2026 · similarity 0.90)
• Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (ICML 2024 · similarity 0.89)
• The Value of Variance: Mitigating Debate Collapse in Multi-Agent Systems via Uncertainty-Driven Policy Optimization (ICML 2026 · similarity 0.89)