Research map / Large language models / Agents, code and math
A Careful Examination of Large Language Model Performance on Grade School Arithmetic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds GSM1k, a 1205-problem human-written held-out version of GSM8k, and uses it to measure overfitting (up to 8% accuracy drop) across many LLMs, linking overfit to per-character likelihood of GSM8k test data.
Problem
Whether high LLM scores on GSM8k reflect data contamination and memorization rather than genuine mathematical reasoning.
Approach
Creates a fresh, human-only (no LLM-assisted) grade-school math benchmark matched to GSM8k on difficulty (human solve rates, step counts, answer magnitudes) and compares model accuracy on GSM8k vs. GSM1k, then correlates the gap with model likelihood of GSM8k test items.
Key results
- Worst models drop up to ~8% accuracy from GSM8k to GSM1k (e.g., Yi-6B-Chat 0.080 gap), while frontier models such as GPT-4 and Claude show near-zero gaps (gpt-4 -0.012).
- Spearman correlation r^2=0.36 (p=0.03) between per-character log-likelihood of GSM8k test items and the GSM8k-GSM1k performance gap.
- Human annotators solved 4.07±0.93 GSM8k vs 4.36±1.11 GSM1k problems in 15 minutes, and identified the lone GSM1k item only 21.83% of the time (20% chance).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on newly human-created GSM1k instead of GSM8k | LLMs evaluated in the paper · GSM1k vs GSM8k | accuracy: up to 8% | Mixed |
| Use an alternative prompt with non-GSM8k examples and different answer phrasing (ablation) | evaluated models · GSM1k and GSM8k | accuracy gap | No clear effect |
| Manually extract answers that are not correctly formatted instead of using automatic answer extraction (ablation) | selected evaluated models · GSM1k and GSM8k | accuracy | No clear effect |
| Higher per-character likelihood of generating GSM8k test examples | models evaluated in the analysis · GSM8k vs GSM1k | performance gap: Spearman's r=0.36; p = 0.03 | Helped |
Surprising findings
- Phi-2 retains over half of GSM1k problems despite a 6% drop, comparable to Llama2-70B, indicating overfit models still reason.
- Likelihood of GSM8k test items does not fully explain overfitting; outlier models such as Mixtral variants and Math-Shepherd show mismatches, and Llemma shows high likelihood with minimal overfit.
Limitations
- GSM1k is not publicly released to prevent contamination, limiting independent verification.
- GSM1k and GSM8k are only approximately matched in distribution; Claude models appear slightly easier on GSM1k.
- Likelihood-based contamination proxy is imperfect; outliers weaken the correlation (r^2=0.36).
Experimental setup
Datasets: GSM8k GSM1k
Benchmarks: GSM8k GSM1k
Metrics: accuracy accuracy gap (GSM8k minus GSM1k) Z-score p-value Spearman correlation
Baselines: GSM8k
Builds on: GSM8k LM Evaluation Harness
Closest papers by meaning
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.92)
- Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process (ICLR 2025 · similarity 0.92)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.92)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.91)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.91)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.