Research map / Large language models / Reasoning and chain of thought
MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 19.2B-token math pretraining corpus (MathCode-Pile) by pairing LLM-extracted LaTeX reasoning steps with execution-verified Python code, and continues pretraining four 7-8B base models to produce MathCoder2.
Problem
Existing continued math pretraining corpora include code that uses math-related packages but lacks natural-language reasoning context and is mostly designed for non-math fields, limiting its value for mathematical reasoning.
Approach
Extract LaTeX expressions with their conditions and results from math web/textbook text using an instruction-tuned LLM, translate each reasoning step into Python, execute and keep only snippets whose output matches the expected result, and concatenate code with the reasoning step for pretraining.
Key results
- MathCoder2-Llama-3-8B reaches 38.4% on MATH and 69.9% on GSM8K (4-shot), up 17.0 and 15.1 points over Llama-3-8B.
- MathCoder2-DeepSeekMath-7B improves DeepSeekMath-7B on MATH from 36.2% to 38.6%.
- On DeepSeekCoder-1.3B, MathCode-Pile reaches 17.8 MATH vs 9.4 for OpenWebMath and 13.6 for DeepSeekMath Corpus (the latter from a 150B-token run).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add model-translated mathematical code paired with reasoning steps to continued pretraining (ablation) | Llama-3-8B · MATH / GSM8K / SAT / OCW / MMLU-MATH | accuracy: +3.7 MATH, +4.1 GSM8K, +12.5 SAT, +5.1 OCW, +1.3 MMLU-MATH | Helped |
| concatenated reasoning step + code data vs reasoning-step-only or code-only (ablation) | DeepSeekCoder-1.3B · MATH | accuracy: 17.8 vs 16.7 (reasoning-step-only) and 14.6 (trans-code-only) | Helped |
| replace code data with rewritten texts from the same LLM (no-code prompt) (ablation) | DeepSeekCoder-1.3B · MATH | accuracy: 15.7 vs 17.8 for full method | Hurt |
| continued pretraining on MathCode-Pile instead of base model | Llama-3-8B, DeepSeekMath-7B, Mistral-7B, Code-Llama-7B · MATH (4-shot) | accuracy: +17.0 (Llama-3), +2.4 (DeepSeekMath), +23.6 (Mistral), +22.1 (Code-Llama) | Helped |
| continued pretraining on MathCode-Pile instead of base model | Llama-3-8B, DeepSeekMath-7B, Mistral-7B, Code-Llama-7B · GSM8K (4-shot) | accuracy: +15.1, +4.6, +16.0, +37.7 | Helped |
| add translated code paired with reasoning steps before NuminaMath-TIR SFT (ablation) | Llama-3-8B (MathCoder2 vs Basic) · MATH after SFT | accuracy: 65.1 vs 62.9 | Helped |
| filtering irrelevant documents from OpenWebMath and CC-En-math (ablation) | DeepSeekCoder-1.3B · MATH / GSM8K | accuracy: filtered-OpenWebMath 9.0 MATH vs OpenWebMath 9.4; GSM8K 11.4 vs 11.2 | No clear effect |
| adding synthetic data, textbooks, or math code to filtered-OpenWebMath (ablation) | DeepSeekCoder-1.3B · MATH | accuracy: synthetic 10.8, textbooks 9.4, code 9.4 vs filtered-OpenWebMath 9.0 | Mixed |
Surprising findings
- The mathematical code is only 14.1% of tokens yet accounts for 21.8% to 66.2% of total accuracy gains across benchmarks, depending on the benchmark.
- Reasoning-step-only or translated-code-only data each underperform the concatenated reasoning-step-plus-code format.
Limitations
- Corpus focuses on mathematics and English, excluding other STEM subjects and languages.
- Only models from 1.3B to 8B parameters were trained due to compute limits.
- Post-training uses only simple SFT without RL or DPO.
Experimental setup
Datasets: OpenWebMath Common Crawl (Matrix) Education-College-Students Maths-College StarCoderData Mathematical textbooks (Nougat-converted PDFs) NuminaMath-CoT NuminaMath-TIR
Benchmarks: GSM8K MATH SAT-Math OCW MMLU-Math Olympiad Bench SVAMP
Metrics: accuracy (greedy decoding)
Baselines: Qwen2-Math Qwen2.5-Math InternLM2-Math-Base InternLM2.5 DeepSeekMath Llemma Mistral Llama-3 Code-Llama OpenWebMath Proof-Pile-2 MathPile
Builds on: OpenWebMath DeepSeekMath corpus fastText Llemma MAmmoTH
Closest papers by meaning
- MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning (ICLR 2024 · similarity 0.92)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.91)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.91)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.91)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.90)
- JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.