Research map / Multi-agent systems and games / LLM agents

Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DREAM, a two-agent opposing-stance multi-round debate framework for IR relevance labeling that auto-accepts agreed labels and escalates disagreements to humans, and uses it to build BRIDGE, a hole-filled version of BEIR and RobustQA subsets.

Problem

IR benchmarks are incomplete ("holes"): many relevant chunks are unlabeled, biasing retriever evaluation and RAG alignment. Single-LLM labeling is overconfident and confidence-based human escalation is inaccurate and costly.

Approach

Two LLM agents with opposing initial stances (relevant vs. irrelevant) critique each other over up to R rounds; agreement gives the label automatically, persistent disagreement (uncertain cases) is escalated to humans with the debate history, avoiding calibration or threshold tuning.

Key results

• DREAM reaches 95.2% balanced accuracy on non-escalated cases with only 3.5% escalation, versus 73.9% for LLMJudge and 93.8% for Human-Only.
• The pipeline uncovers 29,824 missing relevant chunks (428% over the 6,976 original gold chunks) at about $506 human cost.
• RAGAlign@10 improves from 0.70 on original benchmarks to 0.84 on BRIDGE on average, and 20 of 25 retrievers change rank.

Surprising findings

• Increasing debate rounds beyond two does not consistently reduce spurious consensus and can amplify it.
• Adding more agents (4 Llama) lowered bAcc because debate became conservative toward relevant cases.

Limitations

• Human escalation still required for about 3.5% of cases, and the adjudication result relies on MTurk worker quality.
• Evaluation set of 700 samples is small relative to the full benchmark; reliance on Llama3.3-70B as the sole labeler family for main results.
• Round-based spurious consensus increases with more rounds, limiting scaling by debate depth.

Experimental setup

Datasets: BEIR MS MARCO BEIR NQ RobustQA LoTTE-derived subsets (Lifestyle, Recreation, Science, Technology, Writing)

Benchmarks: BRIDGE BEIR RobustQA

Metrics: balanced accuracy escalation ratio class-wise recall Fleiss kappa Hit@10 nDCG@10 Recall@10 Hole@10 RAGAlign@10 Pearson correlation

Baselines: LLMJudge LARA Human-Only

Builds on: LLMJudge LARA UMBRELA BEIR RobustQA

Closest papers by meaning

• FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (NeurIPS 2025 · similarity 0.87)
• SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction (ICLR 2025 · similarity 0.86)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction (NeurIPS 2026 · similarity 0.85)
• Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.85)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (ICLR 2025 · similarity 0.85)
• Self-Retrieval: End-to-End Information Retrieval with One Large Language Model (NeurIPS 2024 · similarity 0.85)