Research map / Multi-agent systems and games / LLM agents
Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MADQA, a 2,250-question human-authored benchmark over 800 PDFs for agentic document-collection QA, using Classical Test Theory for splits and a Kuiper-based effort-calibration metric to compare agents with humans.
Problem
Whether multimodal agents over document collections show genuine strategic reasoning or merely stochastic trial-and-error search, which existing document QA benchmarks fail to measure due to single-step framing, recycled documents, or model-generated questions.
Approach
A fully human-authored, multi-hop benchmark over fresh heterogeneous PDF collections, with CTT-based difficulty/discrimination split selection, LLM-judge accuracy, page/doc F1 attribution, and a Kuiper cumulative-difference statistic measuring effort calibration against humans.
Key results
- Gemini 3 Pro BM25 Agent reaches 82.2% accuracy on the MADQA test set, above Gemini 3 Pro File Search at 78.6%.
- Human annotators with oracle retrieval reach 99.4% accuracy, an oracle gap of roughly 18% over the best agents.
- Humans achieve a Kuiper calibration score of 14.6, lower than every agent system (best agent 22.9, Gemini 3 Pro RLM).
Surprising findings
- Humans and Gemini 3 Pro reach similar accuracy (~82%) yet agree on item correctness only weakly (Cohen's kappa 0.24), succeeding on different questions.
- Claude Sonnet 4.5 RLM processed over 270 million input tokens at a cost of $850 yet did not match its BM25 agent counterpart's accuracy.
Limitations
- Corpus is English-language and predominantly U.S.-centric, limiting transfer to other languages and document genres.
- Public documents may have been seen in pretraining, so some answers may be memorized (about 8% guessable via contamination per the paper).
- Results depend on the step budget N=10 and step count as an effort proxy, so rankings can shift with budget.
Experimental setup
Datasets: MADQA
Benchmarks: MADQA
Metrics: accuracy Page F1 Doc F1 Kuiper X-Page accuracy X-Doc accuracy
Baselines: Gemini 3 Pro File Search Gemini 2.5 Flash File Search M3DocRAG GPT-5 File Search HEAVEN ColBERTv2 + Llama-3.1-8B MDocAgent Recursive Language Models Claude Agent with Semtools
Builds on: Classical Test Theory Recursive Language Models MDocAgent M3DocRAG ColBERTv2
Closest papers by meaning
- MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.89)
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.88)
- WideSearch: Benchmarking Agentic Broad Info-Seeking (ICLR 2026 · similarity 0.88)
- InteractComp: Evaluating Search Agents With Ambiguous Queries (ICML 2026 · similarity 0.87)
- Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding (CVPR 2026 · similarity 0.87)
- AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.