Research map / Multi-agent systems and games / LLM agents
DSGym: A Standardized and Holistic Framework for Evaluating and Training Data Science Agents
Fan Nie, Junlin Wang, Harper Hua, Federico Bianchi, Yongchan Kwon, Zhenting Qi, Owen Queen, Shang Zhu, James Zou
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DSGym, a standardized framework for evaluating and training data science agents that unifies fragmented benchmarks and emphasizes rigorous data grounding.
Problem
Existing data science agent benchmarks have fragmented evaluation interfaces that hinder cross-benchmark comparison, narrow task coverage, and weak data grounding, with many tasks solvable without the data.
Approach
A unified evaluation and training framework for data science agents that standardizes interfaces across benchmarks and checks data grounding of tasks, supporting both evaluation and training of agents.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune Qwen3-4B on 2,000 execution-verified synthetic trajectories (DSGym-SFT) | Qwen3-4B-Instruct · DABStep-hard | accuracy (%): 2.9 -> 33.07 | Improved |
| fine-tune Qwen3-4B on DSGym-SFT | Qwen3-4B-Instruct · QRData-Verified | accuracy (%): 45.27 -> 59.36 | Improved |
| fine-tune on general-analysis-only SFT data, test out of domain | Qwen3-4B-Instruct · DSBio | accuracy (%): 6.67 -> 21.11 | Improved |
| increase GPT-5.1 reasoning effort | GPT-5.1 · DSPredict-Hard | above-median rate (%): 10.3 -> 14.3 | Improved |
| increase GPT-5.1 reasoning effort (none to high) | GPT-5.1 · DABStep-hard | accuracy (%): 11.9 -> 13.23 | No clear effect |
| withhold data files (shortcut test) on QRData-Verified (ablation) | all evaluated models · QRData-Verified | accuracy: up to ~21% relative drop | Hurt |
| shortcut filtering of tasks answerable by >=3 of 5 LLMs without data | DSGym-Tasks · existing benchmarks | data dependence | Improved |
| DSGym-SFT model on shortcut-removed tasks | Qwen3-4B DSGym-SFT · QRData-Verified | accuracy drop without files | Improved |
Surprising findings
- A substantial portion of tasks in current data science benchmarks can be solved without access to the underlying data.
Closest papers by meaning
- Scaling Generalist Data-Analytic Agents (ICLR 2026 · similarity 0.89)
- DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning (ICML 2024 · similarity 0.88)
- Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing (NeurIPS 2026 · similarity 0.87)
- DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? (ICLR 2025 · similarity 0.87)
- BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics (ICML 2026 · similarity 0.87)
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
