Research map / Large language models / Agents, code and math
Can LLMs Solve Longer Math Word Problems Better?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces E-GSM, a GSM8K extension with longer narratives, two CoLeG metrics, a Condition-Retrieving prompt (CoRe) for proprietary LLMs, and an "extension" auxiliary SFT task for open-source LLMs.
Problem
LLMs' ability to solve math word problems with long, extended narratives (context length generalizability, CoLeG) is under-explored, and accuracy appears to degrade as context grows.
Approach
Use GPT-4-turbo to iteratively extend GSM8K problems into longer narratives while preserving conditions and answers; propose a zero-shot Condition-Retrieving instruction (CoRe) before 0-CoT for proprietary models, and add extended-problem CoT data as an auxiliary SFT task for open models.
Key results
- On E-GSM with GPT-3.5-turbo, CoRe improves 0-CoT accuracy on round 0 by 2.51% and round 4 by 5.34%.
- SFT with extension vs. SFT without it improves CoLeG-E by 5.9% and CoLeG-R by 9.12% on average across open-source models; LLaMA-2-7B gains 7.87 (CoLeG-E) and 14.33 (CoLeG-R).
- Across proprietary models, Acc_0 drops on average 15.33% to Acc_4 with zero-shot CoT prompts, and open-source models without extension drop 17.41%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Condition-Retrieving Instruction to 0-CoT | GPT-3.5-turbo · E-GSM | CoLeG-E: 46.63 -> 51.97 | Helped |
| Add Condition-Retrieving Instruction to 0-CoT | GPT-3.5-turbo · E-GSM | Acc_4: 64.42 -> 69.76 | Helped |
| Add extension data during fine-tuning (ablation) | LLaMA-2-7B · E-GSM | CoLeG-E: 20.22 -> 28.09 | Helped |
| Add extension data during fine-tuning (ablation) | LLaMA-2-7B · E-GSM | CoLeG-R: 66.64 -> 80.97 | Helped |
| Add extension data during fine-tuning (ablation) | LLaMA-2-13B · E-GSM | CoLeG-E: 32.40 -> 37.27 | Helped |
| Add extension data during fine-tuning (ablation) | LLaMA-2-70B · E-GSM | CoLeG-E: 45.32 -> 49.81 | Helped |
| Add extension data during fine-tuning (ablation) | Mistral-7B · E-GSM | CoLeG-E: 42.13 -> 48.50 | Helped |
| Add Condition-Retrieving Instruction to 0-CoT | GPT-3.5-turbo · MAWPS | accuracy: 91.40 -> 92.67 | Helped |
| Add Condition-Retrieving Instruction to 0-CoT | GPT-3.5-turbo · SVAMP | accuracy: 71.80 -> 76.30 | Helped |
| Add Condition-Retrieving Instruction to 0-CoT | GPT-3.5-turbo · GSM-IC | accuracy: 88.35 -> 89.60 | Helped |
| Add extension data during fine-tuning (ablation) | LLaMA-2-7B · MAWPS | accuracy: 71.22 -> 72.52 | Helped |
| Add extension data during fine-tuning (ablation) | LLaMA-2-7B · SVAMP | accuracy: 63.40 -> 64.90 | Helped |
Surprising findings
- Semantic understanding (informativeness) stays roughly unchanged after CoRe or extension; gains come from improved reasoning (missing-step) metric.
- Claude-3-opus, the strongest proprietary model, still drops more than 10% in accuracy from round 0 to round 4.
Limitations
- Extended problems are generated by GPT-4-turbo and filtered with an entailment model and LLM solvability checks; human evaluation covers only a subset (50 seed questions, 200 items).
- Proprietary model experiments rely on API access and the CoRe gains are modest on some metrics (e.g., Claude-3-opus CoLeG-E gains are small on some prompts).
- Error analysis: 46% of sampled bad cases failed due to incorrect extraction of known conditions.
Experimental setup
Datasets: GSM8K E-GSM MAWPS SVAMP GSM-IC ASDiv-A
Benchmarks: E-GSM GSM8K MAWPS SVAMP GSM-IC
Metrics: accuracy per round (Acc_r) CoLeG-E CoLeG-R informativeness missing step
Baselines: Plan-and-Solve Plan-and-Solve+ 0-CoT MetaMath SFT without extension
Builds on: chain-of-thought zero-shot CoT Plan-and-Solve prompting MetaMath ROSCOE
Closest papers by meaning
- OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.89)
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.89)
- GSM-$\infty$: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length? (ICML 2025 · similarity 0.88)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.88)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.