Research map / Large language models / Agents, code and math
MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MATH-Perturb, two 279-problem variants of level-5 MATH problems (simple vs. hard perturbations), and shows LLMs drop sharply on hard perturbations, attributing part of the failure to a memorization of solution techniques.
Problem
Existing math robustness benchmarks only apply simple perturbations (e.g., changing numbers) that preserve the solution method, so they cannot test whether LLMs recognize when a problem's reasoning pattern no longer applies.
Approach
Hand-crafted minimal edits to level-5 MATH problems that fundamentally change the problem so the original solution method fails (hard perturbation), paired with a same-method simple perturbation as control, and answers guaranteed to differ from the original.
Key results
- On HARD, o1-mini drops 16.49 points (94.27% original to 78.49%) and gemini-2.0-flash-thinking drops about 12.9 points (92.47% to 78.14% in the table).
- Most models show small drops on SAME, but hard perturbations cause large drops across all 18 evaluated models.
- Adding the original problem and solution as one-shot ICL raised HARD accuracy only marginally (under 5 points for most models) because misleading effects offset gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace original problems with hard perturbations | o1-mini · MATH-Perturb All | accuracy (%): 94.27 -> 78.49 | Hurt |
| replace original problems with hard perturbations | Gemini-2.0-flash-thinking-exp · MATH-Perturb All | accuracy (%): 92.47 -> 78.14 | Hurt |
| replace original problems with hard perturbations | o1-preview · MATH-Perturb All | accuracy (%): 87.81 -> 72.40 | Hurt |
| replace original problems with simple perturbations | Phi-3.5-mini-instruct · MATH-Perturb SAME test split | accuracy (%): 24.35 -> 31.30 | Helped |
| add the corresponding original problem and solution as one-shot ICL (ablation) | models on MATH-Perturb SAME · MATH-Perturb SAME (All) | accuracy (%) | Helped |
| add the corresponding original problem and solution as one-shot ICL (ablation) | o1-mini · MATH-Perturb HARD (All) | accuracy (%): 78.49 -> 78.49 | No clear effect |
| add the corresponding original problem and solution as one-shot ICL (ablation) | GPT-4o · MATH-Perturb HARD (All) | accuracy (%): 39.43 -> 43.01 | Helped |
| add the corresponding original problem and solution as one-shot ICL (ablation) | MetaMath-13B-V1.0 · MATH-Perturb HARD (All) | accuracy (%): 5.73 -> 3.58 | Hurt |
| use the corresponding original example instead of a random same-category example (ablation) | GPT-4o · MATH-Perturb HARD (All) | accuracy (%): 37.28 -> 43.01 | Helped |
| use the corresponding original example instead of a random same-category example (ablation) | MetaMath-13B-V1.0 · MATH-Perturb HARD (All) | accuracy (%): 5.38 -> 3.58 | Hurt |
Surprising findings
- Using the original problem and solution as an in-context demonstration often hurts models on HARD problems through a misleading effect, counteracting ICL gains.
- Only about 1-3% of errors on HARD show identical answers to the original problem, but manual inspection estimates 40% of o1-mini errors stem from memorization-style failures.
Limitations
- Benchmark is small (279 problem pairs), limiting statistical power for per-subject comparisons.
- Memorization error percentages are estimated from manual inspection of only 20 error cases per model.
- Some Case III outcomes (hard correct but easy wrong) reflect annotator difficulty misjudgments rather than model behavior.
Experimental setup
Datasets: MATH
Benchmarks: MATH-Perturb (Original) MATH-Perturb (SAME) MATH-Perturb (HARD)
Metrics: accuracy pass@k self-consistency accuracy edit distance cosine similarity
Baselines: o1-preview Gemini-2.0-flash-exp GPT-4o Claude-3.5-Sonnet Qwen2.5-Math-7B-Instruct Llama-3.1-8B-Instruct
Builds on: MATH Functional MATH GSM-Symbolic MATH^2 chain-of-thought
Closest papers by meaning
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.89)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.89)
- DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving (NeurIPS 2024 · similarity 0.89)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.89)
- MathScale: Scaling Instruction Tuning for Mathematical Reasoning (ICML 2024 · similarity 0.88)
- MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.