Research map / Large language models / Agents, code and math
MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Llama-2 and CodeLlama on MathCodeInstruct, an 80k dataset of interleaved natural-language, code, and execution-result solutions (plus problem-interpolation data), with loss masked on execution outputs, yielding MathCoder models that set open-source SOTA on GSM8K and MATH.
Problem
Open-source LLMs lag closed-source models on math reasoning and cannot replicate the GPT-4 Code Interpreter style of interleaving reasoning, code generation, and execution feedback.
Approach
Generate GPT-4 Code Interpreter-style LCE (natural language, code, execution result) solutions for GSM8K and MATH, augment data with a problem-interpolation prompt that creates intermediate-difficulty problems solved by self-distillation, then SFT with special block tokens, loss excluding execution results, and real code execution during inference in a Jupyter kernel.
Key results
- MathCoder-CL-34B reaches 81.7% on GSM8K and 45.2% on MATH, the best open-source scores reported, and MathCoder-L-70B reaches 83.9% on GSM8K.
- Even MathCoder-L-7B outperforms WizardMath-70B on three of five datasets, with a +4.5% average gain.
- Actual code execution at inference raises average accuracy from 35.1% to 69.1% versus model-predicted execution results; masking execution-result loss further adds 1.1 points (69.1 to 70.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add problem-interpolation examples to GSM8K+MATH training data (ablation) | CodeLlama-34B MathCoder · GSM8K | accuracy: 77.3 -> 81.7; +4.4 | Helped |
| Add problem-interpolation examples to GSM8K+MATH training data (ablation) | CodeLlama-34B MathCoder · MATH | accuracy: 44.0 -> 45.2; +1.2 | Helped |
| Add problem-interpolation examples to GSM8K+MATH training data (ablation) | CodeLlama-34B MathCoder · SVAMP | accuracy: 78.6 -> 82.5; +3.9 | Helped |
| Add problem-interpolation examples to GSM8K+MATH training data (ablation) | CodeLlama-34B MathCoder · Mathematics | accuracy: 71.6 -> 75.9; +4.3 | Helped |
| Add problem-interpolation examples to GSM8K+MATH training data (ablation) | CodeLlama-34B MathCoder · SimulEq | accuracy: 59.3 -> 65.8; +6.4 | Helped |
| Use actual code execution rather than model-predicted execution results at inference (ablation) | CodeLlama-34B MathCoder · GSM8K | accuracy: 54.1 -> 79.9; +25.8 | Helped |
| Use actual code execution rather than model-predicted execution results at inference (ablation) | CodeLlama-34B MathCoder · MATH | accuracy: 16.9 -> 45.9; +29.0 | Helped |
| Use actual code execution rather than model-predicted execution results at inference (ablation) | CodeLlama-34B MathCoder · SVAMP | accuracy: 69.6 -> 81.9; +12.3 | Helped |
| Use actual code execution rather than model-predicted execution results at inference (ablation) | CodeLlama-34B MathCoder · Mathematics | accuracy: 20.6 -> 74.2; +53.6 | Helped |
| Use actual code execution rather than model-predicted execution results at inference (ablation) | CodeLlama-34B MathCoder · SimulEq | accuracy: 14.2 -> 63.6; +49.4 | Helped |
| Exclude execution-result tokens from the training loss (ablation) | CodeLlama-34B MathCoder with actual execution · GSM8K | accuracy: 79.9 -> 81.7; +1.8 | Helped |
| Exclude execution-result tokens from the training loss (ablation) | CodeLlama-34B MathCoder with actual execution · MATH | accuracy: 45.9 -> 45.2; -0.7 | Hurt |
Surprising findings
- Llama-2-70B based MathCoder (73.1% average) outperforms CodeLlama-34B based MathCoder (70.2%), contrary to the concurrent MAmmoTH findings.
- Problem interpolation with GPT-4 judged 83.2% of new problems harder than GSM8K and 95.6% easier than MATH, while using only GSM8K seeds produced problems too similar to originals.
Limitations
- Relies on GPT-4 for data generation, so capabilities are bounded by GPT-4.
- Unable to solve theorem-proving problems.
- Still struggles with complex geometry problems as a uni-modal model.
Experimental setup
Datasets: GSM8K MATH SVAMP Mathematics SimulEq
Benchmarks: GSM8K MATH SVAMP Mathematics SimulEq
Metrics: accuracy average accuracy
Baselines: ChatGPT-3.5 GPT-4 GPT-4 Code Interpreter PaLM-2 WizardMath Llama-1 RFT Galactica
Builds on: Program-aided language models (PAL) Chain-of-thought prompting Alpaca CodeLlama Llama-2
Closest papers by meaning
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.92)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.91)
- Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification (ICLR 2024 · similarity 0.91)
- JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models (NeurIPS 2024 · similarity 0.90)
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (ICLR 2024 · similarity 0.90)
- OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.