Research map / Large language models / Language, safety and interpretability
A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets
Tejas Agrawal, Vu Le, Sumit Gulwani, Gust Verbruggen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a stronger LLM (GPT-5 with reasoning vs GPT-5 mini) | zero-shot LLM predictor · NAPE single-action repredict | UAS (%): 18.0 -> 32.7 | Improved |
| fine-tune SmolLM2-360M on synthetic operation sequences | SmolLM2-360M · NAPE single-action repredict | UAS (%): 21.7 -> 26.8 | Improved |
| fine-tune SmolLM2-135M on synthetic operation sequences | SmolLM2-135M · NAPE single-action repredict | UAS (%): 18.3 -> 23.2 | Improved |
| accept every non-empty prediction (ALWAYS) instead of GREEDY (ablation) | GPT-5 multi-action · NAPE multi-action | UAS (%): 22.3 -> -19.2 | Hurt |
| use a stricter acceptance rule (GREEDY-2 / HYBRID-2) instead of GREEDY (ablation) | GPT-5 multi-action · NAPE multi-action | UAS (%): 22.3 -> 20.3 -> 17.5 | Hurt |
| accept on precision >= 90% regardless of utility (P90) instead of GREEDY (ablation) | GPT-5 multi-action · NAPE multi-action | UAS (%): 22.3 -> 17.0 | Hurt |
| increase prediction stride from 1 to 8 (ablation) | GPT-5 multi-action · NAPE multi-action | UAS (%): 22.3 -> 9.8 | Hurt |
| increase context window from 32 to 128 operations (ablation) | GPT-5 multi-action · NAPE multi-action | UAS (%): 22.3 -> 27.6 | Improved |
| increase context window from 128 to 512 or 2048 operations (ablation) | GPT-5 multi-action · NAPE multi-action | UAS (%): 27.6 -> 26.2 (512), 27.4 (2048) | No clear effect |
| cap actions per prediction at 1 instead of unlimited (ablation) | GPT-5 multi-action · NAPE multi-action | UAS (%): 22.3 -> 20.3 | Hurt |
| disable re-prediction after an accepted action (ablation) | GPT-5 single-action · NAPE single-action | UAS (%): 27.4 -> 20.3 | Hurt |
| turn off context shortening of long cell values (ablation) | GPT-5 · NAPE single-action | UAS (%): 27.4 -> 27.7 | No clear effect |
Closest papers by meaning
- SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation (NeurIPS 2024 · similarity 0.85)
- FLAME: A Small Language Model for Spreadsheet Formulas (AAAI 2024 · similarity 0.83)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.83)
- From Guessing to Placeholding: A Cost-Theoretic Framework for Uncertainty-Aware Code Completion (ICML 2026 · similarity 0.83)
- AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos? (ICLR 2024 · similarity 0.83)
- Copilot Arena: A Platform for Code LLM Evaluation in the Wild (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
