Research map / Multi-agent systems and games / LLM agents

InteractComp: Evaluating Search Agents With Ambiguous Queries

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces InteractComp, a 210-question benchmark where search agents must interact (yes/no clarification to a simulated user) to resolve target-distractor ambiguity, finding models overconfident and rarely interacting.

Problem

Search-agent benchmarks assume complete, unambiguous queries, so agents' ability to recognize ambiguity and interact to resolve it during search is untested.

Approach

Target-distractor construction creates questions whose shared attributes admit a popular distractor, with distinctive attributes hidden in context that a simulated responder reveals only through yes/no/IDK interaction; evaluates search+interact ReAct agents and shows interaction is latent (forced interaction roughly doubles accuracy).

Key results

• GPT-5 achieves 13.73% accuracy in the search+interact setting, while the with-context ceiling reaches up to 71.50% (OpenAI o3).
• Forcing GPT-5 to ask 8 clarifying questions raises accuracy from 20% to 40%.
• Interaction capability stayed flat over 15 months while BrowseComp accuracy rose seven-fold (10% to 70%).

Surprising findings

• Increasing interaction budget from 5 to 20 rounds barely increases how often models ask questions, indicating overconfidence rather than budget limits.
• GPT-4o-mini asks in 73.95% of rounds yet achieves only 7.13% accuracy, showing that more questions does not imply better disambiguation.

Limitations

• Responder is a simulated GPT-4o with yes/no/IDK protocol, only moderately agreeing with humans (Krippendorff alpha 0.707).
• Small benchmark of 210 questions; many ranking differences are within a few points.
• Interaction restricted to binary yes/no questions, which may not reflect natural user dialogue.

Experimental setup

Benchmarks: InteractComp

Metrics: accuracy calibration error (ECE) interaction rate average rounds cost

Baselines: BrowseComp GAIA IN3 Tau-Bench Doubao Deep Search OpenAI Deep Research

Builds on: ReAct BrowseComp

Closest papers by meaning

• Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking (ICLR 2026 · similarity 0.90)
• Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction (NeurIPS 2026 · similarity 0.89)
• Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.89)
• InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents (ICLR 2026 · similarity 0.89)
• WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.89)
• AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web (ICML 2026 · similarity 0.88)