AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

DSGym: A Standardized and Holistic Framework for Evaluating and Training Data Science Agents

Fan Nie, Junlin Wang, Harper Hua, Federico Bianchi, Yongchan Kwon, Zhenting Qi, Owen Queen, Shang Zhu, James Zou

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces DSGym, a standardized framework for evaluating and training data science agents that unifies fragmented benchmarks and emphasizes rigorous data grounding.

Problem

Existing data science agent benchmarks have fragmented evaluation interfaces that hinder cross-benchmark comparison, narrow task coverage, and weak data grounding, with many tasks solvable without the data.

Approach

A unified evaluation and training framework for data science agents that standardizes interfaces across benchmarks and checks data grounding of tasks, supporting both evaluation and training of agents.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune Qwen3-4B on 2,000 execution-verified synthetic trajectories (DSGym-SFT)Qwen3-4B-Instruct · DABStep-hardaccuracy (%): 2.9 -> 33.07Improved
fine-tune Qwen3-4B on DSGym-SFTQwen3-4B-Instruct · QRData-Verifiedaccuracy (%): 45.27 -> 59.36Improved
fine-tune on general-analysis-only SFT data, test out of domainQwen3-4B-Instruct · DSBioaccuracy (%): 6.67 -> 21.11Improved
increase GPT-5.1 reasoning effortGPT-5.1 · DSPredict-Hardabove-median rate (%): 10.3 -> 14.3Improved
increase GPT-5.1 reasoning effort (none to high)GPT-5.1 · DABStep-hardaccuracy (%): 11.9 -> 13.23No clear effect
withhold data files (shortcut test) on QRData-Verified (ablation)all evaluated models · QRData-Verifiedaccuracy: up to ~21% relative dropHurt
shortcut filtering of tasks answerable by >=3 of 5 LLMs without dataDSGym-Tasks · existing benchmarksdata dependenceImproved
DSGym-SFT model on shortcut-removed tasksQwen3-4B DSGym-SFT · QRData-Verifiedaccuracy drop without filesImproved

Surprising findings

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy