Research map / Large language models / Agents, code and math
SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SpreadsheetBench, a benchmark of 912 real-world Excel forum spreadsheet manipulation questions with multiple test cases per instruction and an OJ-style evaluation, showing large gaps between LLMs and human experts.
Problem
Existing spreadsheet manipulation benchmarks use synthesized instructions, simplified single-table spreadsheets, and single-test exact-match evaluation, which do not reflect real user demands or robustness of generated solutions.
Approach
Build a benchmark from real Excel forum questions with complex, context-rich instructions and realistic non-standard multi-table spreadsheets, and evaluate code solutions with an online-judge-style metric applying each solution to multiple perturbed spreadsheet test cases (soft and hard restriction scores).
Key results
- GPT-4o achieves 18.35% overall soft and 15.02% hard restriction score, versus 71.33% soft and 62.00% hard for human experts.
- Copilot in Excel reaches 20.00% overall soft restriction on a 5% sample, the highest among models and products.
- Multi-round prompting with code execution feedback raises GPT-3.5 overall soft score from 2.34% to 7.09% and Llama-3 70B from 1.32% to 3.74%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-round inference (ablation) | CodeQwen (7B) · SpreadsheetBench | soft restriction overall: 0.51 -> 3.66 | Helped |
| use multi-round inference (ablation) | DeepseekCoder (33B) · SpreadsheetBench | soft restriction overall: 2.60 -> 5.31 | Helped |
| use multi-round inference (ablation) | Mixtral-8x7B · SpreadsheetBench | soft restriction overall: 3.11 -> 3.88 | Helped |
| use multi-round inference (ablation) | Llama-3 (70B) · SpreadsheetBench | soft restriction overall: 1.32 -> 3.74 | Helped |
| use multi-round inference (ablation) | GPT-3.5 · SpreadsheetBench | soft restriction overall: 2.34 -> 7.09 | Helped |
| use multi-round inference (ablation) | GPT-4o · SpreadsheetBench | soft restriction overall: 18.35 -> 16.96 | Hurt |
| add multi-round execution feedback (ablation) | GPT-3.5 · SpreadsheetBench (200-sample experiment) | soft restriction: 5.38 (±2.19) -> 6.58 (±1.42) | Helped |
| add multi-round execution feedback (ablation) | GPT-4o · SpreadsheetBench (200-sample experiment) | soft restriction: 12.79 (±1.71) -> 16.08 (±1.08) | Helped |
| add ReAct to multi-round execution feedback (ablation) | GPT-3.5 · SpreadsheetBench (200-sample experiment) | soft restriction: 6.58 (±1.42) -> 6.04 (±0.59) | Hurt |
| add ReAct to multi-round execution feedback (ablation) | GPT-4o · SpreadsheetBench (200-sample experiment) | soft restriction: 16.08 (±1.08) -> 14.67 (±1.77) | Hurt |
| use multi-round inference (ablation) | GPT-4o · SpreadsheetBench (200-sample experiment) | soft restriction: 12.79 (±1.71) -> 16.08 (±1.08) | Helped |
| provide 5 spreadsheet rows instead of 1 (ablation) | GPT-4o · SpreadsheetBench (50-sample impact analysis) | overall soft restriction accuracy | Helped |
Surprising findings
- GPT-4o's performance slightly declines under multi-round setting in the main results, attributed to duplicated spreadsheet content from retrieval; averaged runs in the appendix reverse this.
- Adding a ReAct retrieval mechanism tends to reduce performance relative to code-execution feedback alone.
Limitations
- Proprietary and spreadsheet-product evaluations were run on small samples (5% or 50 instructions with one test case), limiting statistical reliability.
- Test case construction does not exhaustively cover corner cases given the benchmark size.
- Human performance was measured on only 50 instructions by four annotators.
Experimental setup
Datasets: ExcelForum posts MrExcel posts ExcelGuru articles Chandoo blogs
Benchmarks: SpreadsheetBench
Metrics: soft restriction score hard restriction score OJ-style exact match
Baselines: TaPEx TaPas Binder CodeQwen1.5 DeepseekCoder Mixtral-8x7B Llama-3 70B GPT-3.5 GPT-4o SheetCopilot Copilot in Excel
Builds on: ReAct Binder SheetCopilot DS-1000
Closest papers by meaning
- MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark (NeurIPS 2025 · similarity 0.88)
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.88)
- Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries (AAAI 2024 · similarity 0.86)
- EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance (NeurIPS 2026 · similarity 0.86)
- FLAME: A Small Language Model for Spreadsheet Formulas (AAAI 2024 · similarity 0.85)
- OpenTab: Advancing Large Language Models as Open-domain Table Reasoners (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.