Research map / Multi-agent systems and games / LLM agents
InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InfiAgent-DABench, a closed-form benchmark (DAEval) of data analysis questions over CSV files with an agent framework, benchmarks 34 LLMs, and trains DAAgent on the DAInstruct dataset.
Problem
Existing benchmarks do not evaluate LLM agents end-to-end on data analysis tasks, and open-ended data analysis answers are hard to evaluate automatically.
Approach
Use format-prompting to turn open-ended data analysis questions generated by GPT-4 into closed-form questions with "@answer_name[answer]" output formats, enabling automatic exact-match evaluation of ReAct-style agents with a Python sandbox; additionally distill GPT-4 agent trajectories into open-source CodeLlama models (DAAgent).
Key results
- GPT-4 achieves 78.99% accuracy on DAEval, the top result among 34 benchmarked LLMs.
- DAAgent-34B (Code Llama-34B tuned on DAInstruct) reaches 64.59%, surpassing GPT-3.5 (60.70%) by 3.89 points.
- Best open-source model Qwen-72B-Chat reaches 59.92%, trailing GPT-4 by about 19 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Instruction-tune CodeLlama-34B on DAInstruct instead of general instruction tuning | CodeLlama-34B · DAEval validation | accuracy: 31.13 -> 64.59 | Helped |
| Instruction-tune CodeLlama-13B on DAInstruct instead of general instruction tuning | CodeLlama-13B · DAEval validation | accuracy: 26.67 -> 55.86 | Helped |
| Instruction-tune CodeLlama-7B on DAInstruct instead of general instruction tuning | CodeLlama-7B · DAEval validation | accuracy: 24.61 -> 51.36 | Helped |
| Add a GPT-3.5 response reformatting step before evaluation (ablation) | GPT-4 · DAEval validation | accuracy: about 6% | Helped |
| Add a GPT-3.5 response reformatting step before evaluation (ablation) | Mistral-7B · DAEval validation | accuracy: up to 32% | Helped |
Surprising findings
- GPT-4 as an automatic evaluator reached only about 67% consistency with human experts on open-ended data analysis answers, motivating closed-form questions.
- Removing the reformatting step drops GPT-4 accuracy by about 6% and Mistral-7B by up to 32%, showing format compliance is a major confound.
Limitations
- Excludes visualization tasks because they are hard to express as closed-form questions.
- Questions and labels are generated by GPT-4 and labeled by OpenAI ADA, so labels may carry model biases; only 85% of samples passed human assessment.
- Evaluation relies on GPT-3.5 reformatting responses into the required format.
Experimental setup
Datasets: DAEval DAInstruct
Benchmarks: DAEval (InfiAgent-DABench)
Metrics: accuracy (all subquestions correct)
Baselines: GPT-4 GPT-3.5 Claude-2.1 Gemini-Pro Qwen-72B-Chat XAgent AutoGen Qwen-Agent CodeLlama-Instruct
Builds on: ReAct vLLM Code Llama
Closest papers by meaning
- DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? (ICLR 2025 · similarity 0.87)
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.86)
- EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance (NeurIPS 2026 · similarity 0.86)
- DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle (ICLR 2026 · similarity 0.86)
- InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation (ICLR 2025 · similarity 0.86)
- Scaling Generalist Data-Analytic Agents (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.