Research map / Large language models / Agents, code and math
RADAR: Benchmarking Language Models on Imperfect Tabular Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RADAR, a benchmark of 2,980 table QA instances from 53 expert-curated tasks with programmatic injection of five data artifact types, evaluating LMs on data-aware reasoning across table sizes.
Problem
Language models deployed for autonomous data analysis are rarely tested on whether they detect and correctly handle data artifacts (missing values, bad values, outliers, inconsistent formatting and logic) in tabular data, and how this scales with table size.
Approach
Programmatically perturb clean, expert-curated tables with expert-written, query-specific artifact injectors, paired with recovered-table ground truth, so that naive computation fails and exact-match scoring is objective; table size is varied in tokens and columns while keeping task semantics fixed.
Key results
- o4-mini reaches 100% on clean tables with code execution but performance drops by 59% on perturbed tables.
- Under direct prompting, accuracy falls to nearly zero by 16K tokens as table size grows, while code agents remain stable across table sizes.
- Most models score below 20% under direct prompting on RADAR, with only o4-mini and Gemini 2.5 Pro consistently above 50%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Introduce data artifacts into tables | frontier language models · RADAR perturbed tables | Exact Match accuracy: drops by 59% | Hurt |
| Add Python code execution | language models · RADAR-T | Exact Match accuracy | Helped |
| Evaluate on tables with artifacts rather than clean tables | language models using code agents · RADAR logically inconsistent tables | Exact Match accuracy | Hurt |
| Increase table size | language models using direct prompting · RADAR-S, 16K tokens | Exact Match accuracy | Hurt |
| Increase table size and vary table width | code agents · RADAR-S | Exact Match accuracy | No clear effect |
| Use wider tables with fewer rows at the same token count | language models using direct prompting · RADAR-S | Exact Match accuracy | Helped |
| Use code agents for row dropping rather than direct prompting | language models · RADAR-T row-drop tasks | Exact Match accuracy | Helped |
| Use standardized code operations for value-derivation tasks | language models using code agents · RADAR-T value-derivation tasks | Exact Match accuracy | No clear effect |
Surprising findings
- Code execution does not fix data-artifact handling: models apply generic routines like dropping rows without identifying the actual issue.
- Models that solve a clean table often fail on the logically inconsistent variant of the same table.
Limitations
- Perturbation types are a fixed set and not exhaustive (e.g., sampling bias is not covered).
- Exact-match scoring restricts evaluation to tasks with a single objective corrective action, excluding ambiguous cleaning scenarios.
- No multiple random seeds or error bars reported due to resource constraints.
Experimental setup
Datasets: 53 expert-curated table QA tasks from 27 source tables across 9 domains
Benchmarks: RADAR RADAR-T RADAR-S
Metrics: exact match accuracy
Baselines: TableGPT2-7B StructLM-7B GPT-4.1 Gemini 2.5 Flash Gemma 3 27B DeepSeek-V3 o3-mini o4-mini Gemini 2.5 Pro DeepSeek-R1
Builds on: table QA benchmarks Gemma 3 tool-use code agents
Closest papers by meaning
- MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark (NeurIPS 2025 · similarity 0.88)
- NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables (NeurIPS 2025 · similarity 0.87)
- TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools (NeurIPS 2024 · similarity 0.87)
- OpenTab: Advancing Large Language Models as Open-domain Table Reasoners (ICLR 2024 · similarity 0.87)
- Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries (AAAI 2024 · similarity 0.87)
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.