Research map / Large language models / Agents, code and math
KRAMABENCH: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes
Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Om Chabra, SIVAPRASAD SUDHIR, Anna Zeng, Anton A. Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, Jun Chen, Markos Markakis and 7 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add execution-feedback re-invocation (few-shot vs one-shot) (ablation) | DS-Guru (GPT-o3) · KramaBench Full | end-to-end score (%): 16.67 -> 24.98 | Improved |
| add file snippets to prompt (one-shot vs no-context) (ablation) | DS-Guru (GPT-o3) · KramaBench Full | end-to-end score (%): 5.87 -> 16.67 | Improved |
| use agentic control flow (smolagents DR) instead of structured DS-Guru | smolagents DR (Claude-3.7) · KramaBench Full | end-to-end score (%): 24.98 -> 55.83 | Improved |
| use Claude-3.7 instead of GPT-o3 as smolagents backend (ablation) | smolagents DR · KramaBench Full | end-to-end score (%): 28.07 -> 55.83 | Improved |
| use Claude-3.7 instead of GPT-o3 as DS-Guru backend (ablation) | DS-Guru few-shot · KramaBench Full | end-to-end score (%): 24.98 -> 8.29 | Hurt |
| add evaluator and reflection agent (Reflexion) to single agent (ablation) | smolagents DR (Claude-3.7) · KramaBench Full | end-to-end score (%): 55.83 -> 55.37 | No clear effect |
| use planner/decomposer to tool executor multi-agent (PDT) | smolagents (Claude-3.7) · KramaBench Full | end-to-end score (%): 55.83 -> 12.01 | Hurt |
| supply gold files only (Oracle input) instead of full data lake (ablation) | DS-Guru few-shot (GPT-o3) · KramaBench | end-to-end score (%): 24.98 -> 43.71 | Improved |
| supply gold files only (Oracle input) (ablation) | smolagents DR (Claude-3.7) · KramaBench | end-to-end score (%): 55.83 -> 60.67 | Improved |
| supply gold files only (Oracle input) (ablation) | smolagents DR (GPT-o3) · KramaBench | end-to-end score (%): 28.07 -> 25.86 | No clear effect |
| increase rows sampled per file in OPS (ablation) | DS-Guru few-shot (GPT-o3) · KramaBench | overall score (%): 22.89 -> 22.58 | No clear effect |
| obscure identifiers and numbers in inputs (ablation) | smolagents Reflexion (Claude-3.7) · KramaBench | end-to-end score (%): 55.37 -> 13.94 | Hurt |
Closest papers by meaning
- AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite (ICLR 2026 · similarity 0.86)
- Scaling Generalist Data-Analytic Agents (ICLR 2026 · similarity 0.86)
- SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery (NeurIPS 2026 · similarity 0.85)
- DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? (ICLR 2025 · similarity 0.85)
- DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle (ICLR 2026 · similarity 0.85)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
