Research map / Multi-agent systems and games / LLM agents
Deep Research Arena: The First Exam of LLMs’ Research Abilities via Seminar-Grounded Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DeepResearch Arena, a benchmark of over 10,000 open-ended research tasks generated from 200+ academic seminar transcripts by a multi-agent pipeline (MAHTG), evaluated with keypoint-aligned (KAE) and checklist (ACE) metrics on eight search-augmented LLMs.
Problem
Faithfully evaluating deep research agents is hard because existing benchmarks either risk data leakage from static corpora or rely on small, manually curated expert question sets that lack authentic research discourse.
Approach
Derive research tasks from academic seminar video transcripts via a multi-agent hierarchy (inspiration extraction, task generation, Elo-based ranking), assumed to be absent from pretraining data, and score reports with reference-grounded keypoint metrics plus adaptively generated rubric checklists.
Key results
- o4-mini-deepresearch achieves the highest ACE score (4.03 English) and the most references (29.66), while gemini-2.5-flash has KSR 78.7 and KCR 3.4 on English tasks.
- grok-4 reaches the highest English KSR (83.3) but drops sharply on Chinese tasks (KSR 50).
- KAE and ACE correlate with human judgments with Spearman rho of 0.84 and 0.81 respectively.
Surprising findings
- grok-4 shows strong English factual grounding but much weaker Chinese performance, indicating limited multilingual generalization.
- gemini-2.5-flash uses far more tokens (64.09k English) than other models, a thoroughness-efficiency trade-off.
Limitations
- Evaluation uses only 100 selected tasks from the full dataset, and the leakage test uses 100 tasks.
- Judge models (gemini-2.5-flash, GPT-4o) are LLMs, which may introduce bias despite ACE decoupling.
- The KCR formula in the paper is mistakenly defined for KOR, and the paper reports inconsistent table/text descriptions.
Experimental setup
Datasets: academic seminar video transcripts (200+ seminars, 12 disciplines)
Benchmarks: DeepResearch Arena
Metrics: Keypoint Supported Rate Keypoint Conflict Rate Keypoint Omission Rate ACE checklist score average tokens average references Spearman correlation Pearson correlation Kendall tau
Baselines: gpt-4o-search-preview gpt-4o-mini-search-preview gpt-4.1 w/search gpt-4.1-mini w/search o4-mini-deepresearch gemini-2.5-pro w/search gemini-2.5-flash w/search grok-4 w/search
Builds on: Elo rating system Humanity's Last Exam DeepResearch Bench BrowseComp
Closest papers by meaning
- From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents (NeurIPS 2026 · similarity 0.88)
- MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.87)
- ResearchTown: Simulator of Human Research Community (ICML 2025 · similarity 0.87)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.87)
- Quest: Training Frontier Deep Research Agents with Fully Synthetic Tasks (NeurIPS 2026 · similarity 0.87)
- MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.