Research map / Large language models / Agents, code and math
fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets
Sanket Mhatre, Sumit Gulwani, Vu Le, Yasharth Bajpai, Gust Verbruggen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces fxBench, a benchmark for evaluating proactive formula suggestion in spreadsheets, and studies how well models handle this task.
Problem
Proactive formula suggestions in spreadsheets require collecting and representing the right context for a machine learning model, which is challenging to evaluate; the abstract is truncated, so details are limited.
Approach
A benchmark framing formula suggestion as a proactive task in spreadsheet software (Excel and Google Sheets style), with an emphasis on context collection and representation for the model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use denser Minisheet representation instead of windowed SpreadsheetCoder with frontier model | GPT 5.2 · fxBench zero-shot | accuracy relative to upper bound: SpreadsheetCoder 89.6 vs Minisheet 48.0 | Hurt |
| increase viewport from 20x10 to 40x30 (ablation) | windowed context · fxBench context coverage | upper-bound coverage: 69.0% -> 89.9% | Improved |
| increase Minisheet border width w beyond 2 (ablation) | Minisheet · fxBench context coverage | intent printed coverage: 86.3% (w=2) -> 87.5% (w=4) | No clear effect |
| switch table representation format (ablation) | GPT 5.2 / Sonnet 4.6 · fxBench zero-shot full sheet | accuracy: GPT 5.2: CSV 49.7 vs pipe-delimited 58.8 | Mixed |
| add dependency chain to oracle context (ablation) | GPT 5.2 oracle · fxBench oracle | accuracy: +6.2% | Improved |
| add worksheet formulas, same-column formulas or neighbors to oracle context (ablation) | GPT 5.2 oracle · fxBench oracle | accuracy: +5.2%, +4.6%, +4.4% | Improved |
| add cell format to oracle context (ablation) | GPT 5.2 oracle · fxBench oracle | accuracy | No clear effect |
| add natural-language intent description to oracle context (ablation) | GPT 5.2 oracle · fxBench oracle | accuracy: 66.2 -> 97.2 | Improved |
| add all structured signals to oracle context (ablation) | GPT 5.2 oracle · fxBench oracle failures | wrong formula type failures: wrong type 77 -> 43; wrong ranges 15 -> 16; wrong abstentions 15 -> 26 | Mixed |
| supervised fine-tuning on Minisheet with curated data (ablation) | Qwen3-8B · fxBench Minisheet context | pass@1: 9.7% -> 21.2% | Improved |
| train on curated data instead of random sample (ablation) | Qwen3-8B fine-tuning · fxBench Minisheet context | pass@1: 11.3% -> 21.2% | Improved |
| scale curated training data to 39,161 records (ablation) | Qwen3-8B fine-tuning · fxBench Minisheet context | pass@1: 21.2% -> 20.6% | No clear effect |
Experimental setup
Benchmarks: fxBench
Closest papers by meaning
- FLAME: A Small Language Model for Spreadsheet Formulas (AAAI 2024 · similarity 0.88)
- SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation (NeurIPS 2024 · similarity 0.88)
- EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance (NeurIPS 2026 · similarity 0.85)
- MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark (NeurIPS 2025 · similarity 0.82)
- Unsupervised Discovery of Formulas for Mathematical Constants (NeurIPS 2024 · similarity 0.82)
- Composing Biases by Using CP to Decompose Minimal Functional Dependencies for Acquiring Complex Formulae (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
