Research map / Large language models / Reasoning and chain of thought
Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses EK-FAC influence functions to attribute math and code reasoning gains to SFT training examples, sequences, and tokens, and flips task difficulty in the data mix to improve Qwen2.5-7B-Instruct.
Problem
Existing data curation for LLM reasoning SFT relies on heuristics, and the cross-domain effects between math and code data and the fine-grained patterns driving reasoning gains remain poorly understood.
Approach
Adapts EK-FAC influence functions to reasoning-oriented SFT, using mean log-likelihood of correct answers as the surrogate objective, and extends attribution to sequence (counterfactual sentence removal) and token levels; uses the findings to flip the difficulty mix of training data.
Key results
- Difficulty-flipped Bespoke-Stratos-17k mix raises AIME24 accuracy from 10.0% to 20.0% and LiveCodeBench from 33.8% to 35.3% for Qwen2.5-7B-Instruct.
- Removing exploration sentences from BS-17k SFT drops MATH500 from 77.2 to 73.8 and LiveCodeBench from 33.8 to 32.0.
- Difficulty-reverse flipping yields the worst results (LiveCodeBench 30.0), supporting the claimed direction of the effect.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use more challenging math problems and simpler coding problems in the training mix | Qwen2.5-Instruct-7B · AIME24 | accuracy: 10.0 -> 20.0 | Helped |
| Use more challenging math problems and simpler coding problems in the training mix | Qwen2.5-Instruct-7B · LiveCodeBench | pass@1 accuracy: 33.8 -> 35.3 | Helped |
| Use more challenging math problems and simpler coding problems in the training mix | Qwen2.5-Instruct-14B · AIME24 | accuracy: 20.0 -> 23.0 | Helped |
| Use more challenging math problems and simpler coding problems in the training mix | Qwen2.5-Instruct-14B · LiveCodeBench | pass@1 accuracy: 45.3 -> 45.5 | Helped |
| Simplify hard math problems and make easy coding tasks more difficult instead of flipping difficulty in the proposed direction (ablation) | Qwen2.5-Instruct-7B · LiveCodeBench | pass@1 accuracy: 33.8 -> 30.0 | Hurt |
| Remove post-correct-answer exploration from training traces (ablation) | Qwen2.5-7B-Instruct · MATH500 | accuracy: 77.2 -> 73.8 | Hurt |
| Remove post-correct-answer exploration from training traces (ablation) | Qwen2.5-7B-Instruct · LiveCodeBench | pass@1 accuracy: 33.8 -> 32.0 | Hurt |
| Add exploration sentences after reaching a correct answer (ablation) | Long-CoT reasoning traces · Math and code reasoning | influence score: Math: 7.0e+07 -> 9.0e+07; Code: 7.5e+06 -> 8.4e+06 | Helped |
| Use high-difficulty math examples instead of lower-difficulty math examples | LLaMA3-8B-Base fine-tuned on MetaMathQA and OSS-Instruct · GSM8K and MBPP | average influence score | Helped |
| Use simpler code problems alongside math data | Qwen2.5-7B-Instruct fine-tuned on Bespoke-Stratos-17k · LiveCodeBench | influence score | Helped |
| Include code-domain examples in math training data | LLaMA3-8B-Base fine-tuned on MetaMathQA and OSS-Instruct · Math reasoning | influence score | Helped |
Surprising findings
- Exploration behavior after reaching a correct answer, usually seen as overthinking, has a positive influence on both math and code reasoning.
- Low-difficulty code problems most benefit code reasoning when mixed with math data, contrary to the expectation that harder data helps most.
Limitations
- Hessian approximated using only MLP parameters with attention treated as fixed.
- Analysis limited to math and coding reasoning; other domains like commonsense reasoning are not studied.
- Small evaluation sets (AIME, LiveCodeBench) with few-point differences make gains noisy.
Experimental setup
Datasets: MetaMathQA OSS-Instruct Bespoke-Stratos-17k OpenThoughts-114k MATH GSM8k
Benchmarks: AIME24 AIME25 MATH500 LiveCodeBench GSM8k MBPP
Metrics: accuracy pass@1 influence score Pearson correlation of rankings
Baselines: Bespoke-Stratos-17k SFT Difficulty-reverse-Flipped
Builds on: influence functions EK-FAC LIMO s1K Sky-T1
Closest papers by meaning
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.89)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.89)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.89)
- Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.88)
- To Code or Not To Code? Exploring Impact of Code in Pre-training (ICLR 2025 · similarity 0.88)
- At Which Training Stage Does Code Data Help LLMs Reasoning? (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.