AI Paper Atlas

Research map / Large language models / Agents, code and math

KRAMABENCH: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes

Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Om Chabra, SIVAPRASAD SUDHIR, Anna Zeng, Anton A. Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, Jun Chen, Markos Markakis and 7 more

ICLR 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 29 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add execution-feedback re-invocation (few-shot vs one-shot) (ablation)DS-Guru (GPT-o3) · KramaBench Fullend-to-end score (%): 16.67 -> 24.98Improved
add file snippets to prompt (one-shot vs no-context) (ablation)DS-Guru (GPT-o3) · KramaBench Fullend-to-end score (%): 5.87 -> 16.67Improved
use agentic control flow (smolagents DR) instead of structured DS-Gurusmolagents DR (Claude-3.7) · KramaBench Fullend-to-end score (%): 24.98 -> 55.83Improved
use Claude-3.7 instead of GPT-o3 as smolagents backend (ablation)smolagents DR · KramaBench Fullend-to-end score (%): 28.07 -> 55.83Improved
use Claude-3.7 instead of GPT-o3 as DS-Guru backend (ablation)DS-Guru few-shot · KramaBench Fullend-to-end score (%): 24.98 -> 8.29Hurt
add evaluator and reflection agent (Reflexion) to single agent (ablation)smolagents DR (Claude-3.7) · KramaBench Fullend-to-end score (%): 55.83 -> 55.37No clear effect
use planner/decomposer to tool executor multi-agent (PDT)smolagents (Claude-3.7) · KramaBench Fullend-to-end score (%): 55.83 -> 12.01Hurt
supply gold files only (Oracle input) instead of full data lake (ablation)DS-Guru few-shot (GPT-o3) · KramaBenchend-to-end score (%): 24.98 -> 43.71Improved
supply gold files only (Oracle input) (ablation)smolagents DR (Claude-3.7) · KramaBenchend-to-end score (%): 55.83 -> 60.67Improved
supply gold files only (Oracle input) (ablation)smolagents DR (GPT-o3) · KramaBenchend-to-end score (%): 28.07 -> 25.86No clear effect
increase rows sampled per file in OPS (ablation)DS-Guru few-shot (GPT-o3) · KramaBenchoverall score (%): 22.89 -> 22.58No clear effect
obscure identifiers and numbers in inputs (ablation)smolagents Reflexion (Claude-3.7) · KramaBenchend-to-end score (%): 55.37 -> 13.94Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy