Research map / Large language models / Agents, code and math
DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DSBench, a benchmark of 466 ModelOff data analysis questions and 74 Kaggle-derived data modeling tasks, and evaluates LLMs, LVLMs and agents (AutoGen, Code Interpreter) on them, finding low accuracy and a persistent gap to humans.
Problem
Existing data science benchmarks are simplified (short instructions, single modality, code completion only, tool-specific), so they do not reflect realistic end-to-end data science agent work.
Approach
Builds realistic data science tasks from ModelOff financial Excel analysis competitions (multimodal, long context, multi-table) and Kaggle competitions (end-to-end modeling with train/test split), and introduces Relative Performance Gap (RPG) to normalize heterogeneous metrics across modeling competitions.
Key results
- Best agent (AutoGen with GPT-4o) reaches 34.12% task-level accuracy on data analysis tasks, versus 64.06% for human experts on a 10-competition sample.
- On data modeling, AutoGen with GPT-4 achieves 87.84% task success but only 45.52 RPG, with AutoGen GPT-4o at 34.74 RPG; human RPG is 65.02.
- Vanilla Gemini reaches 31.55% task-level accuracy on data analysis, the best model-only result, above GPT-4o at 28.11%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoGen interaction and tools instead of the model alone | GPT-4o · DSBench data analysis | task-level accuracy /%: 28.11 -> 34.12 | Helped |
| Use AutoGen interaction and tools instead of the model alone | GPT-4 · DSBench data analysis | task-level accuracy /%: 25.97 -> 30.69 | Helped |
| Use AutoGen interaction and tools instead of the model alone | GPT-3.5 · DSBench data analysis | task-level accuracy /%: 20.39 -> 20.82 | Helped |
| Use AutoGen interaction and tools instead of the model alone | GPT-4o mini · DSBench data analysis | task-level accuracy /%: 23.82 -> 28.11 | Helped |
| Use AutoGen instead of Code Interpreter | GPT-4o · DSBench data modeling | RPG: 19.87 -> 34.74 | Helped |
| Use AutoGen instead of Code Interpreter | GPT-3.5 · DSBench data modeling | RPG: 6.52 -> 6.02 | No clear effect |
| Increase data analysis input context length | AutoGen GPT-4o · DSBench data analysis | accuracy | Hurt |
| Vary data modeling input context length | AutoGen GPT-4o · DSBench data modeling | completion rate | No clear effect |
Surprising findings
- Claude (claude-3-5-sonnet) scores only 6.01% on data analysis, far below GPT-4o and Gemini.
- Code Interpreter with GPT-3.5 achieves twice the task success of AutoGen with GPT-3.5 on data modeling yet has a similar RPG.
Limitations
- Data modeling tasks are split from Kaggle training sets (8:2) rather than evaluated on official test sets.
- Human baselines are based on small samples (10 analysis competitions, 22 modeling competitions) and public Kaggle code.
- Only greedy decoding and a single solution per instance were evaluated due to cost.
Experimental setup
Datasets: ModelOff financial Excel challenges (466 questions) Kaggle competitions (74 tasks)
Benchmarks: DSBench data analysis DSBench data modeling
Metrics: task-level accuracy competition-level accuracy task success rate relative performance gap (RPG) cost inference time
Baselines: LLaVA Llama3-8b Llama3-70b GPT-3.5 GPT-4 GPT-4o GPT-4o mini Claude Gemini AutoGen Code Interpreter
Builds on: AutoGen ReAct DS-1000
Closest papers by meaning
- DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning (ICML 2024 · similarity 0.90)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.88)
- InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks (ICML 2024 · similarity 0.87)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.87)
- Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? (NeurIPS 2024 · similarity 0.87)
- \$OneMillion-Bench: How Far are Language Agents from Human Experts? (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.