Research map / Large language models / Agents, code and math
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MathCheck, a checklist of four math tasks times four problem variants, auto-generated with GPT-4-Turbo, and builds MathCheck-GSM and MathCheck-GEO to evaluate 26 LLMs and 17 MLLMs.
Problem
Current math benchmarks focus on single problem-solving tasks and per-problem robustness, so they may overfit and fail to measure genuine mathematical reasoning across related tasks.
Approach
A checklist grid crossing task generalization (problem solving, answerable judging, outcome judging, process judging) with reasoning robustness variants (original, problem understanding, irrelevant disturbance, scenario understanding), generated semi-automatically by an LLM from seed benchmark problems; consistency across cells is treated as evidence of real understanding.
Key results
- O1-preview scored 93.2 overall on MathCheck-GSM and O1-mini 92.7, with GPT-4o at 92.0.
- On MathCheck-GEO, GPT-4o scored 65.3 overall, the best among 43 models evaluated.
- MathCheck-GSM correlates with GSM1k (Pearson p reported as higher than GSM8K) and with BPC-loss at p=-0.915 versus -0.822 for GSM8K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate with MathCheck-GSM instead of GSM8K | Mathematical reasoning benchmark evaluation · GSM1k correlation with BPC-Loss | Pearson correlation coefficient: -0.822 -> -0.915 | Helped |
| Train on mathematical problem-solving data | DeepSeek-Math and MetaMath models · MathCheck-GSM tasks beyond problem solving | Performance | No clear effect |
| Use few-shot prompting instead of zero-shot prompting | GPT-3.5-Turbo · MathCheck-GSM | Performance | Hurt |
| Use zero-shot Chain of Thought or Plan-and-Solve prompting | GPT-3.5-Turbo · MathCheck-GSM | Performance | Mixed |
| Use a diverse checklist covering multiple tasks and problem variants rather than only original problem solving | Mathematical reasoning evaluation · GSM1k | Correlation with genuine mathematical ability | Helped |
| Use original problem-solving accuracy as the sole measure rather than the full checklist | Mathematical reasoning evaluation · MathCheck-GSM problems at different complexity levels | Ability to reflect reasoning skill as problems become more difficult | Hurt |
| Use GPT-4-Turbo to rewrite checklist questions | MathCheck-GSM · OP-OJ non-GPT-rewritten questions | Model ranking | No clear effect |
Surprising findings
- Math-specialized models (DeepSeek-Math, MetaMath) gain strongly on problem solving but fall off on other checklist tasks, suggesting training on solving data alone does not improve broader reasoning.
- Few-shot prompting generally yielded worse results than zero-shot for GPT-3.5-Turbo on MathCheck-GSM.
Limitations
- Generated data is rewritten by GPT-4-Turbo, which may introduce self-bias favoring GPT models, though authors argue impact is limited.
- Small seed sets (129 GSM8K problems, 60 geometry problems) and manual validation with pass rates of 81-93% per rewrite type.
- Evaluation of open models uses few-shot for base/math models and zero-shot for generalists, with prompt and hyperparameter choices that may affect comparisons.
Experimental setup
Datasets: GSM8K GeoQA UniGeo Geometry3K GSM1k Big-bench date understanding
Benchmarks: MathCheck-GSM MathCheck-GEO GSM1k correlation BPC-loss compression correlation
Metrics: accuracy macro-F1 Pearson correlation RMSE
Baselines: GSM8K GSM1k O1-preview GPT-4o Llama-3.1-70B-Instruct DeepSeek-Math-7B-RL MetaMath-LLama2-70B
Builds on: GSM8K GeoQA UniGeo Geometry3K GSM1k
Closest papers by meaning
- Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.90)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.89)
- Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals (ICML 2026 · similarity 0.89)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.89)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.88)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.