Research map / Multi-agent systems and games / LLM agents

WideSearch: Benchmarking Agentic Broad Info-Seeking

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent collaboration and safety · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces WideSearch, a 200-task benchmark for wide-scale web information seeking that requires filling large tables, and evaluates single-agent, multi-agent, and commercial systems with a hybrid automated table-matching metric.

Problem

Existing search-agent benchmarks focus on deep, hard-to-find facts or report writing, not on broad, repetitive information collection where an agent must retrieve every atomic item in a large table without omissions or errors.

Approach

Frames wide information seeking as populating a schema-defined table, with a 200-task benchmark curated from real queries and scored by a hybrid pipeline (rule-based checks plus LLM-as-judge) reporting table-level success, row-level F1, and item-level F1.

Key results

• The best multi-agent system (Claude Sonnet 4 Thinking, Avg@4 SR 3.6%; OpenAI o3 at 5.1%) achieves only about 5% success rate, versus 20% for single humans.
• Item-level F1 approaches 80% with 128 attempts in test-time scaling, while table-level success stays below 20%.
• Automated judging pipeline agrees with human annotations at 97.8% to 98.3% across judge models.

Surprising findings

• Deep research commercial systems often produce long reports with multiple tables instead of a single table, hampering automatic evaluation.
• Item-level F1 can be high while table-level success stays near zero, so the bottleneck is completeness and exactness rather than finding individual facts.

Limitations

• Success criterion is all-or-nothing, so small errors zero out tasks and make scores coarse.
• End-to-end commercial systems were evaluated on only three models and some DeepResearch systems were excluded for format non-compliance.
• Small number of human participants (10 annotators on 20 questions) for the human baseline.

Experimental setup

Datasets: WideSearch

Benchmarks: WideSearch

Metrics: success rate row-level F1 item-level F1 Avg@4 Pass@4 Max@4

Baselines: Claude Sonnet 4 (Thinking) Gemini 2.5 Pro OpenAI o3 Kimi K2 DeepSeek-R1 Doubao-Seed-1.6

Builds on: BrowseComp DeepResearch Bench Search-R1

Closest papers by meaning

• Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.91)
• InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents (ICLR 2026 · similarity 0.90)
• GISA: A Benchmark for General Information-Seeking Assistant (NeurIPS 2026 · similarity 0.90)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction (NeurIPS 2026 · similarity 0.89)
• Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers (NeurIPS 2025 · similarity 0.89)
• Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs (ICLR 2026 · similarity 0.89)