Research map / Multi-agent systems and games / LLM agents

SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AutoResearchBench, a 1,000-query benchmark (600 Deep Research, 400 Wide Research) for AI agents doing scientific literature discovery over a 3M-paper arXiv corpus via DeepXiv search, finding frontier models score below 10%.

Problem

Existing agentic web-browsing benchmarks do not measure whether agents can locate scientific papers by fine-grained full-text constraints or exhaustively collect all papers meeting a technical specification, and whether they can abstain when no paper qualifies.

Approach

A full-text-first, human-LLM construction pipeline yields two task families over a controlled arXiv corpus: unique-paper identification or abstention (Deep Research, accuracy) and exhaustive set collection (Wide Research, IoU), evaluated with a standardized ReAct agent and DeepXiv search tool.

Key results

• Claude-Opus-4.6 achieves the best Deep Research accuracy at 9.39%, and Gemini-3.1-Pro-Preview the best Wide Research IoU at 9.31%.
• Most open-source and many closed models score below 5% on Deep Research accuracy and Wide Research IoU.
• Replacing DeepXiv search with open-web search lowers averaged Deep Search accuracy from 5.42% to 3.97% across four matched models.

Surprising findings

• Thinking mode does not consistently help and is generally detrimental for Wide Research, while increasing runtime.
• Longer trajectories and more tool calls do not improve outcomes; GPT-5.4 with 6.1 turns beats DeepSeek-V3.2 with 28.8 turns on Deep Research.

Limitations

• Scope limited to computer science papers in a fixed arXiv corpus and text-only evidence.
• Exhaustive answer sets for Wide Research may be hard to guarantee at large corpus scale.
• Ground-truth set completeness for Wide Research is only audited on a sample.

Experimental setup

Datasets: arXiv corpus via DeepXiv (over 3 million papers)

Benchmarks: AutoResearchBench

Metrics: accuracy IoU precision recall

Baselines: Claude-Opus-4.6 Gemini-3.1-Pro-Preview GPT-5.4 Seed-2.0-Pro Deepseek-V3.2 Kimi-K2.5 Alphaxiv GPT Deep Research

Builds on: ReAct BrowseComp DeepXiv

Closest papers by meaning

• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.93)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.90)
• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.89)
• AI-Researcher: Autonomous Scientific Innovation (NeurIPS 2025 · similarity 0.89)
• FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.89)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.88)