Research map / Multi-agent systems and games / LLM agents
DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DAComp, a 236-task benchmark evaluating data agents on repository-level data engineering (architecture, implementation, evolution) and open-ended data analysis, scored by execution metrics and an LLM judge with hierarchical rubrics.
Problem
Existing data agent benchmarks reduce data engineering to isolated code snippets and open-ended analysis to deterministic answers, failing to capture enterprise full-lifecycle data intelligence workflows.
Approach
Two-track benchmark: repo-level DE tasks scored with component, cascading-failure and strict success metrics on enterprise schemas, plus open-ended DA tasks scored by an LLM judge using hierarchical rubrics with multiple valid solution paths and Good-Same-Bad comparisons against baseline reports.
Key results
- GPT-5 achieves a DE Score of 43.45 and a strict Success Rate of 20.00% on DAComp-DE Implementation/Evolution.
- GPT-5 achieves the top DA score of 56.14 overall (DA-Agent evaluated by Flash judge), with most other models below 50.
- Gemini-2.5-Flash judge reaches rubric item weighted kappa 0.834 and case ICC 0.890 against human experts.
Surprising findings
- Medium-scale code edits are harder than very large edits, since large edits are often repetitive boilerplate.
- Reasoning model o3 computes correct numbers but scores poorly on readability and analytical depth, a 'calculator behavior'.
Limitations
- DE and DA tasks rely on synthetic data populated into schemas.
- Open-ended DA scoring depends on an LLM judge, even though validated against humans.
- Even the best model achieves low strict success rates, so the benchmark is very hard and results sensitive to agent framework.
Experimental setup
Datasets: enterprise SaaS schemas with synthetic data curated web databases
Benchmarks: DAComp-DE DAComp-DA DAComp-zh
Metrics: Component Score Cascading Failure Score Success Rate rubric score GSB score Cohen's kappa ICC(A,1) Kendall tau-b
Baselines: OpenHands CodeAct DA-Agent DE-Agent
Builds on: ReAct OpenHands CodeAct
Closest papers by meaning
- Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (ICLR 2025 · similarity 0.87)
- InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks (ICML 2024 · similarity 0.86)
- CoDA-Bench: Can Code Agents Handle Data-Intensive Tasks? (ICML 2026 · similarity 0.86)
- Synthesis Through Simulation: Generating Coherent Enterprise Data via Scalable Agent-System Interaction (NeurIPS 2026 · similarity 0.85)
- Scaling Generalist Data-Analytic Agents (ICLR 2026 · similarity 0.85)
- InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.