Research map / Multi-agent systems and games / LLM agents

From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a category-theoretic framework for deep research agents, derives a 296-question bilingual structural benchmark across four task types, evaluates 16 systems, and adds tracked search and category tools to API-based agents that yield modest gains.

Problem

Existing deep research agent benchmarks are ad hoc and monolithic, lacking a formal basis to diagnose retrieval-chain, multi-source verification, re-ordering, and false-premise rejection capabilities.

Approach

Models deep research as functors from an intent category to a retrieval subcategory and then to a reasoning category, deriving four structural task families (chains, pullbacks, re-ordering, Yoneda falsification) as a benchmark, plus theory-guided design modules (tracked search and pullback/colimit-style category tools).

Key results

• Best system Grok Deep Research reaches 19.9% total average accuracy on the benchmark, with Perplexity Deep Research leading Type I (10.8%) and Type II (32.7%).
• Adding category tools to tracked search raises GPT-5 from 10.48% to 13.29% on the benchmark and from 45.68% to 47.53% on DeepResearchBench.
• Task 13 (null-field refutation) yields a 0.0% success rate across all evaluated systems.

Surprising findings

• Search-augmented systems such as Qwen3 w/Search and Perplexity-Sonar-Pro w/Search match Grok Deep Research on Type IV (26.5% vs 46.9% is not matched, but both exceed several deep research products), suggesting an agentic loop is not sufficient for false-premise rejection.
• Closed-source Grok Deep Research scores 19.9% on the benchmark while its API-based Grok-4.1-fast variant with category tools reaches only 11.84%, despite the latter exceeding Grok Deep Research on DeepResearchBench.

Limitations

• Category-theoretic claims are largely formal analogies; the functor and proposition proofs are informal and rely on loosely defined morphisms.
• Benchmark is small (296 questions) and per-task sample sizes are small (e.g., 5-20 items per task), limiting statistical reliability.
• Evaluation relies on GPT-5 as an automated judge followed by expert review, with judge bias partially mitigated only by human verification.

Experimental setup

Datasets: DeepResearchBench

Benchmarks: CDR benchmark (296 bilingual questions) DeepResearchBench

Metrics: accuracy search score reasoning score

Baselines: Gemini-3-Pro w/Reasoning Claude-4.5-Sonnet w/Reasoning GPT-5 w/Reasoning Qwen3 w/Reasoning Doubao w/Search Qwen3 w/Search Perplexity-Sonar-Pro w/Search GPT-5 w/Search Claude-4.6-Sonnet w/Search Doubao Deep Research Qwen Deep Research Grok Deep Research

Builds on: DeepResearchBench BrowseComp WebShaper

Closest papers by meaning

• WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research (ICLR 2026 · similarity 0.89)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.88)
• DeepTRACE: Auditing Deep Research AI Systems for Tracking Reliability Across Citations and Evidence (ICLR 2026 · similarity 0.88)
• Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs (ICLR 2026 · similarity 0.88)
• DAGent: Evaluate-then-Grow Planning for Deep Research Agents (NeurIPS 2026 · similarity 0.88)
• Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge (NeurIPS 2025 · similarity 0.88)