Research map / Multi-agent systems and games / LLM agents

DRBench: A Realistic Benchmark for Enterprise Deep Research

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DRBench, a 100-task benchmark for enterprise deep research combining public web sources with synthetic private enterprise data, plus a baseline agent (DRBA) and insight-level LLM-judge metrics.

Problem

Existing deep research benchmarks evaluate web-only retrieval or narrow aspects, and do not test whether agents surface salient insights from both public web and private enterprise files and applications.

Approach

A persona-grounded enterprise benchmark with injected relevant and distractor private insights, hosted in a Dockerized enterprise environment (Nextcloud, Mattermost, email, file browser), scored via atomic insight recall, distractor avoidance, factuality, and report quality.

Key results

• Best local-setting result: OpenAI Deep Research (GPT-5) reaches 41.96 insight recall, 83.64 factuality and 70.35 harmonic mean on the FullBenchmark.
• Adaptive Action Planning on base DRBA with GPT-4o improves insight recall from 13.18 to 15.97 and harmonic mean from 34.82 to 39.74.
• Browser-only GPT-4.1 web agent achieves only 1.11% insight recall and 6.67% factuality.

Surprising findings

• Combining CRP or SRP with AAP did not yield gains and sometimes reduced factuality.
• Agents consistently failed to detect when required information (e.g., FSMA 204) was missing from private files and to issue targeted web searches.

Limitations

• Limited set of task families and domains (100 tasks across 10 domains).
• Insight-level evaluation lacks span-level grounding and relies on LLM-as-judge with residual bias.
• Recall-only insight metric with a +5 cutoff; precision of useful but unlisted insights is not measured.

Experimental setup

Datasets: DRBench (100 tasks, 1093 insights)

Benchmarks: DRBench

Metrics: insight recall distractor avoidance factuality report quality harmonic mean

Baselines: Perplexity OpenAI Deep Research Gemini Generic WebAgents (AgentLab) Llama-3.1-405B-Instruct Qwen-2.5-72B-Instruct

Builds on: G-Eval FactScore TREC-RAG

Closest papers by meaning

• SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.87)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.86)
• From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents (NeurIPS 2026 · similarity 0.86)
• MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (NeurIPS 2026 · similarity 0.86)
• PaperBench: Evaluating AI’s Ability to Replicate AI Research (ICML 2025 · similarity 0.85)
• InnovatorBench: Evaluating Agents’ Ability to Conduct Innovative AI Research (ICLR 2026 · similarity 0.85)