Research map / Large language models / Agents, code and math
MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMTU, a ~28K-question benchmark over 25 expert-level real-world table tasks drawn from 52 datasets, and evaluates 27+ chat and reasoning LLMs, finding frontier reasoning models score at most ~70%.
Problem
Existing table benchmarks for LLMs focus narrowly on NL-to-SQL and table QA, missing the broader expert-level table tasks (transformation, matching, cleaning, joins, column relationships, KB mapping) that data professionals perform.
Approach
Assembles a large, standardized (instruction, input table(s), ground-truth) benchmark by curating objectively evaluable real-world table tasks from decades of data management, PL, and web-data research, with execution-based and structured-output evaluation, and analyzes model sensitivity to long context, permutation, and format.
Key results
- GPT-5 achieves the best overall score of 69.6% on MMTU, with o3 at 69.1%.
- DeepSeek-R1 is the best open-weight reasoning model at 57.9%, while the best chat model GPT-5-Chat reaches 57.7%.
- Reasoning models outperform chat models by over 10 points at the top, and frontier models show a sharp accuracy drop on large tables (NIHT: GPT-5 0.920 vs GPT-5-Chat 0.546 on table needle retrieval).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a reasoning model instead of a chat-oriented model | Frontier models on MMTU · MMTU | accuracy: 0.577 ± 0.01 -> 0.696 ± 0.01 | Helped |
| Use newer and larger models instead of older and smaller models | LLMs evaluated on MMTU · MMTU | accuracy: 0.310 ± 0.01 -> 0.696 ± 0.01 | Helped |
| Increase table context length | Frontier models on MMTU · MMTU long-table-context quartiles | accuracy | Hurt |
| Shuffle table rows and/or columns | Models evaluated on MMTU · MMTU table permutations | accuracy | Hurt |
| Use column shuffling rather than row shuffling | Models evaluated on MMTU · MMTU table permutations | accuracy | Hurt |
| Change table serialization among Markdown, CSV, JSON, and HTML | Frontier reasoning models on MMTU · MMTU table format variations | accuracy | No clear effect |
| Serialize tables as HTML rather than other tested formats | Frontier models on MMTU · MMTU table format variations | accuracy | Hurt |
| Serialize tables as JSON rather than other tested formats | Chat models on MMTU · MMTU table format variations | accuracy | Helped |
Surprising findings
- Frontier models are becoming less sensitive to table serialization format (markdown/CSV/JSON/HTML), except HTML.
- Column shuffling degrades performance more than row shuffling, despite tables being permutation-invariant.
Limitations
- Tasks restricted to those with objective, unique ground truth; excludes subjective table generation/summarization.
- Text-only table inputs; no multimodal (visual grid) evaluation.
- Task selection biased toward tasks well studied in the research literature, omitting practically important but understudied tasks.
Experimental setup
Datasets: Spider BIRD WikiSQL KaggleDBQA Archer TabFact FinQA TableBench WikiQA HiTab SemTab2019 TEGRA
Benchmarks: MMTU
Metrics: accuracy F1 execution-based accuracy
Baselines: GPT-4o Llama-3.3-70B Mistral-Large-2411 Qwen3-235B-A22B-Thinking-2507 Gemini-2.5-pro o3
Builds on: Needle-in-a-haystack
Closest papers by meaning
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.89)
- RADAR: Benchmarking Language Models on Imperfect Tabular Data (NeurIPS 2025 · similarity 0.88)
- SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation (NeurIPS 2024 · similarity 0.88)
- UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Science (ICLR 2024 · similarity 0.88)
- TableRAG: Million-Token Table Understanding with Language Models (NeurIPS 2024 · similarity 0.88)
- Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.