Research map / Multi-agent systems and games / LLM agents
InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InfoMosaic-Bench, a 621-task benchmark for multi-source information seeking by tool-augmented agents across six domains with 77 MCP tools, built via the InfoMosaic-Flow synthesis pipeline and evaluated on 14 LLM agents.
Problem
Existing LLM agents rely mainly on open-web search, while many real tasks need precise domain-specific information from MCP tools; it is unclear whether agents can use domain tools and combine them with web search for complex multi-source tasks.
Approach
An organizer-workers agentic pipeline (InfoMosaic-Flow) synthesizes multi-condition questions grounded in verified domain-tool outputs, then iteratively refines them with a web-only verifier to remove single-source shortcuts.
Key results
- GPT-5 with web search only reaches 38.18% accuracy and 67.48% pass rate on InfoMosaic-Bench, the best among 14 agents.
- Domain tools improved GPT-5 overall only marginally (38.18% to 38.61%) and GLM-4.5 from 20.61% to 21.51%, with gains in Map and Video but drops in Medical, Finance, and Multi-domain for GPT-5.
- 22.4% of failures stem from incorrect tool usage or selection, and performance plateaus after about 8 tool calls.
Surprising findings
- Domain-specific tools give only marginal average gains and degrade several domains (e.g., GPT-5 on Medical/Bio, Finance, and Multi-domain).
- Open-weight GLM-4.5 with domain tools is competitive in some settings, while many closed models score below open ones on accuracy in several domains.
Limitations
- Benchmark covers only 6 domains and 77 tools, with a Chinese-centric map tool set.
- Synthesis relies on GPT-5 as planner and executor, which may bias task construction.
- Answer evaluation uses an LLM judge, and human study covers only 120 sampled problems.
Experimental setup
Datasets: Wikipedia Baidu Baike Qunar NORD ClinicalTrials.gov
Benchmarks: InfoMosaic-Bench
Metrics: Accuracy Pass Rate
Baselines: GPT-5 o3 Grok-4 Claude-4.0-Sonnet Qwen2.5-Max Gemini-2.5-Flash o4-mini GLM-4.5 Kimi-K2 Qwen3-235B-A22B Qwen3-32B DeepSeek-V3
Builds on: ReAct BrowseComp MCP-Bench WebWalkerQA
Closest papers by meaning
- UIS-Digger: Towards Comprehensive Research Agent Systems for Real-world Unindexed Information Seeking (ICLR 2026 · similarity 0.92)
- MindSearch: Mimicking Human Minds Elicits Deep AI Searcher (ICLR 2025 · similarity 0.91)
- Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking (ICLR 2026 · similarity 0.90)
- WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.90)
- InteractComp: Evaluating Search Agents With Ambiguous Queries (ICML 2026 · similarity 0.89)
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.