Research map / Large language models / Agents, code and math
GISA: A Benchmark for General Information-Seeking Assistant
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GISA, a 373-query human-annotated benchmark for general information-seeking agents with item/set/list/table answers, deterministic metrics, a live subset, and human search trajectories, and evaluates ReAct LLM agents and commercial search systems.
Problem
Existing agentic search benchmarks reverse-engineer queries from answers, focus on either deep or wide search, rely on static answers prone to memorization, and depend on unstable LLM-based scoring.
Approach
Human-designed forward queries with structured answer formats (item, set, list, table) enabling deterministic exact-match and F1 scoring, unifying deep and wide search, a live subset with monthly updates, and complete human search trajectories as process-level supervision.
Key results
- Best model Claude 4.5 Sonnet (thinking) reaches 19.30% overall exact match on GISA, the best among evaluated systems.
- Commercial systems underperform LLM ReAct agents; GPT-4o Search Preview reaches 5.63% overall EM and Google Search AI Mode 9.38%.
- Qwen3-Max Best@16 rises from 8.90% to 22.22% on a 40-query subset, showing inference-time scaling gains.
Surprising findings
- Commercial deep research and search products do not outperform LLM-based ReAct agents.
- Models that invoke more tools (DeepSeek-V3.2, GLM-4.7) achieve lower scores, suggesting excessive tool use does not help.
Limitations
- Small scale of 373 queries limits use for supervised fine-tuning.
- Text-only; no images or video.
- Tool-call budget capped at 30 per query, which truncated some searches.
Experimental setup
Datasets: GISA
Benchmarks: GISA
Metrics: exact match F1 order score row-level F1 item-level F1 Best@k Majority@k
Baselines: BrowseComp WideSearch DeepWideSearch GPT-4o Search Preview OpenAI o4 Mini Deep Research Perplexity Sonar Pro Search Google Search AI Mode
Builds on: ReAct BrowseComp WideSearch
Closest papers by meaning
- WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.90)
- Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking (ICLR 2026 · similarity 0.89)
- InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents (ICLR 2026 · similarity 0.88)
- InteractComp: Evaluating Search Agents With Ambiguous Queries (ICML 2026 · similarity 0.88)
- MindSearch: Mimicking Human Minds Elicits Deep AI Searcher (ICLR 2025 · similarity 0.88)
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.