Research map / Large language models / Retrieval and knowledge
AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AgentHop, a 1,011-item multi-hop scientific QA benchmark with a seven-tool, budget-constrained sandbox and a four-axis diagnostic decomposition (search, synthesis, tool-use pattern, resource management), evaluated on 19 LLMs.
Problem
Agentic benchmarks report a single leaderboard accuracy, which hides where agents fail (retrieval, synthesis, tool-use, or resource management) under constrained budgets.
Approach
Dissect a single accuracy score into four diagnostic axes computed from one evaluation run: paper recall (search), conversion rate conditioned on recall (synthesis), calls-per-turn and action n-grams (tool-use pattern), and token/turn/budget failure rates (resource management), over a citation-graph sandbox with fixed caps.
Key results
- Gemini-3 Pro achieves the top accuracy of 0.891 on AgentHop, with Claude Opus 4.6 at 0.775 and Claude Sonnet 4.6 at 0.766 within about one point of each other on tool-use-adjusted axes.
- Behavior clusters by family: GPT models commit early, Anthropic and GLM verify before committing (Think after 70-82% of wrong reads for Anthropic), DeepSeek and Kimi over-search.
- Keyword search adds only +3 to +10 pp over reference-following navigation with the citation graph, which carries most of the search work.
Surprising findings
- Capability tier does not predict non-submission failure modes; GPT-5.3 codex loses only 1% to caps while Gemma-4-26B-A4B loses 44%.
- The Qwen MoE 3.5 to 3.6 upgrade raised accuracy via synthesis while multi-target paper recall dropped from 0.472 to 0.302.
Limitations
- Parametric knowledge floor: models trained on cited literature may answer partly from memory, with closed-book accuracy up to 0.73 for closed-frontier models.
- Single domain (nine CS venues) and only four-option multiple-choice format.
- Generator-evaluator overlap: GPT-5.4 generated items and is also evaluated; the disjoint-generator check uses only 94 items.
Experimental setup
Datasets: arXiv papers from nine CS venues (2022-2025) via Semantic Scholar and ar5iv
Benchmarks: AgentHop GPQA-Diamond HLE Terminal-Bench 2.0 NL2Repo SWE-Bench Pro BrowseComp
Metrics: accuracy paper recall section recall conversion rate calls per turn avg tokens avg turns avg budgets failure ratio Spearman correlation
Baselines: Gemini-3 Flash GPT-5.3 codex GPT-5.4 Claude Opus 4.6 Claude Sonnet 4.6 GLM-5.1 DeepSeek-reasoner DeepSeek-chat Kimi-K2.5 MiniMax-M2.7 Gemma-4-31B Qwen3.5-27B
Builds on: ReAct HotpotQA BrowseComp AgenticRAGTracer
Closest papers by meaning
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.91)
- MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.90)
- AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web (ICML 2026 · similarity 0.89)
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.88)
- WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.88)
- Fleet of Agents: Coordinated Problem Solving with Large Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.