Research map / Large language models / Agents, code and math

Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages

ICLR 2025 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies whether problem-solving data is more useful than math corpora during continued pre-training (CPT), compares four synthetic data methods, and contrasts CPT vs SFT, culminating in the MathGPT-8B base model trained from Llama3-8B.

Problem

Math-specific LLMs rely on continued pre-training with math corpora, which gives smaller gains than SFT; it is unclear whether problem-solving data, synthetic variants of it, and the CPT vs SFT stage choice can yield better mathematical reasoning.

Approach

Use problem-solving data (with reasoning steps) during CPT rather than general math corpora, and compare four synthesis methods (response diversification, query expansion, retrospective enhancement, tutorship amplification) where a teacher model checks and corrects student solutions; analyze CPT vs SFT differences by data distribution and difficulty.

Key results

• Tutorship amplification synthesis raised the four-benchmark average from 39.50 (Base2) to 50.60, the best of the four synthesis methods.
• MathGPT-8B reaches a four-benchmark average of 70.62, comparable to Qwen2.5-Math-7B (69.13) and Qwen2-Math-72B (70.92), using about 140B tokens versus 1T for Qwen2.5-Math-7B.
• CPT with problem-solving data outperforms SFT with the same data; SFT gains are about 60% of CPT gains, and SFT is weaker on in-domain high-school data (+2.38 vs +4.63 average).

Measured effects reported in the paper

Surprising findings

• Test4, using only problem-solving data with no math corpus, matched or exceeded the best mixed-data group.
• Retrospective enhancement gave minimal gains, which the authors attribute to erroneous data not being grounded in actual sampling.

Limitations

• Ablations are run on Llama2-7B with a single run per setting and a limited set of four evaluation sets, with GAOKAO and ZHONGKAO translated by the authors.
• Difficulty categorization by reasoning-step count is acknowledged as imprecise.
• The authors note the synthesis methods are relatively naive and the paper does not study post-training alignment.

Experimental setup

Datasets: RedPajama AlgebraicStack StarCoder OpenWebMath NuminaMath Lila GSM8K MATH GAOKAO ZHONGKAO

Benchmarks: GSM8K MATH GAOKAO ZHONGKAO MMLU

Metrics: accuracy average accuracy over four datasets

Baselines: LLEMMA-7B DeepSeekMath-Base-7B Qwen2-Math-7B Qwen2-Math-72B Qwen2.5-Math-7B Qwen2.5-Math-72B Llama3-8B Llama3-70B Qwen2.5-7B Qwen2-72B

Builds on: Llama2 Llama3 RedPajama OpenWebMath Physics of LLM

Closest papers by meaning

• JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models (NeurIPS 2024 · similarity 0.92)
• MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.91)
• MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.91)
• Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.91)
• Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process (ICLR 2025 · similarity 0.91)
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.91)