Research map / Large language models / Agents, code and math
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GSM-Symbolic, a templated variant of GSM8K, and GSM-NoOp, to show LLM math reasoning is variable across instantiations, fragile to added clauses, and sensitive to irrelevant information.
Problem
GSM8K reports a single accuracy on a fixed test set, which may be unreliable due to contamination and does not reveal robustness or reasoning fragility of LLMs.
Approach
Generate many instances per question from symbolic templates (100 templates x 50 samples) to view LLM performance as a distribution, vary names/numbers/clause count, and add seemingly relevant but inconsequential clauses (GSM-NoOp) to probe whether models truly understand the reasoning.
Key results
- Performance on GSM-Symbolic drops relative to GSM8K for most models, and the GSM8K score falls on the right tail of the distribution for 21 of 25 models; Gemma2-9B shows a >12% gap between worst and best sets.
- Adding a single No-Op clause causes large drops, e.g., Phi-3-mini loses over 65% accuracy on GSM-NoOp and o1-preview also declines significantly.
- Accuracy decreases and variance increases monotonically as clauses increase from GSM-Symbolic-Minus-1 to Plus-2 across all models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace original GSM8K questions with GSM-Symbolic instantiations | LLMs evaluated with 8-shot CoT · GSM8K vs GSM-Symbolic | accuracy | Hurt |
| Change only the numerical values in questions | LLMs evaluated with 8-shot CoT · GSM-Symbolic | accuracy | Hurt |
| Change only proper names in questions | LLMs evaluated with 8-shot CoT · GSM-Symbolic | accuracy | No clear effect |
| Add one clause to questions (ablation) | LLMs evaluated with 8-shot CoT · GSM-Symbolic-P1 | accuracy | Hurt |
| Add two clauses to questions (ablation) | LLMs evaluated with 8-shot CoT · GSM-Symbolic-P2 | accuracy | Hurt |
| Remove one clause from questions (ablation) | LLMs evaluated with 8-shot CoT · GSM-Symbolic-M1 | accuracy | Helped |
| Add a seemingly relevant but irrelevant No-Op clause (ablation) | LLMs evaluated with 8-shot CoT · GSM-Symbolic-NoOp | accuracy: over a 65% drop | Hurt |
| Use eight GSM-P1 examples as in-context shots instead of GSM8K shots (ablation) | LLMs evaluated on GSM-P2 · GSM-P2 | accuracy | No clear effect |
| Fine-tune Phi-3.5 on GSM-P1 (ablation) | Phi-3.5 · GSM-P1 | accuracy | Helped |
| Fine-tune Phi-3.5 on GSM-P1 (ablation) | Phi-3.5 · GSM-P2 | accuracy | Hurt |
| Use eight examples of the same question from GSM-Symbolic as shots (ablation) | LLMs evaluated on GSM-NoOp · GSM-NoOp | accuracy | No clear effect |
| Use eight GSM-NoOp examples from different questions as shots (ablation) | Llama-3-8B · GSM-NoOp | accuracy | No clear effect |
Surprising findings
- Providing 8 shots of the same question with correct reasoning chains did not fix the No-Op failures for most models.
- Some models performed significantly better on NoOp-Symb than on the original GSM-NoOp, despite weaker GSM8K performance.
Limitations
- o1-mini and o1-preview APIs do not allow greedy decoding, so results are not fully controlled.
- Statistical significance tests are debated; the authors acknowledge assumptions were not stated clearly.
- Numerical ranges were slightly widened from GSM8K, though arithmetic accuracy was checked.
Experimental setup
Datasets: GSM8K GSM-Symbolic GSM-P1 GSM-P2 GSM-NoOp
Benchmarks: GSM8K GSM-Symbolic GSM-P1 GSM-P2 GSM-NoOp
Metrics: accuracy mean accuracy across instantiations standard deviation accuracy drop
Baselines: GSM8K original accuracy GSM-Plus GSM1K iGSM
Builds on: GSM8K GSM-IC GSM-Plus iGSM
Closest papers by meaning
- GSM-$\infty$: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length? (ICML 2025 · similarity 0.95)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.92)
- AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking (ICLR 2026 · similarity 0.91)
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.91)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.91)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.