Research map / Large language models / Instruction and fine-tuning
Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how the proportion of coding data in instruction fine-tuning (0%, 50%, 100%) affects zero-shot reasoning across six LLM families, evaluated on twelve symbolic, logical, and arithmetic tasks.
Problem
Coding data is known to boost LLM reasoning during pretraining, but its effect on out-of-domain reasoning during instruction fine-tuning (IFT) is underexplored.
Approach
Builds three equal-size ShareGPT-derived IFT mixtures with 0%, 50%, and 100% coding data (coding labels assigned by GPT-3.5-turbo), fine-tunes six base model families, and analyzes gains overall, per reasoning domain, and per task.
Key results
- Mistral-7B-v0.1 gains +10.3 average points (24.6% relative) from Code-tuning over General-tuning, the largest gain among six families.
- Llama-3-8B shows the smallest gain at +1.5 points (3.6% relative) from coding data.
- Code-tuned models correct General-tuned errors mostly with pure text responses (81.4% for Llama-1, 90.7% for Llama-2), suggesting gains beyond code-writing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase coding data from 0% to 100% | Llama-1-7B IFT · 12 reasoning tasks | average accuracy (%): 23.2 -> 27.9; +4.6 | Helped |
| Increase coding data from 0% to 100% | Llama-2-7B IFT · 12 reasoning tasks | average accuracy (%): 26.7 -> 30.6; +3.9 | Helped |
| Increase coding data from 0% to 50% (ablation) | Llama-3-8B IFT · 12 reasoning tasks | average accuracy (%): 41.4 -> 42.9 | Helped |
| Increase coding data from 0% to 100% | Llama-3-8B IFT · 12 reasoning tasks | average accuracy (%): 41.4 -> 42.6; +1.5 | Helped |
| Increase coding data from 0% to 100% | Mistral-7B-v0.1 IFT · 12 reasoning tasks | average accuracy (%): 42.0 -> 52.3; +10.3 | Helped |
| Increase coding data from 0% to 100% | Qwen-1.5-7B IFT · 12 reasoning tasks | average accuracy (%): 42.2 -> 47.4; + 5.5 | Helped |
| Increase coding data from 0% to 100% | Gemma-7B IFT · 12 reasoning tasks | average accuracy (%): 22.3 -> 30.0; + 7.7 | Helped |
| Use 100% coding data rather than 50% (ablation) | Llama-3-8B IFT · 12 reasoning tasks | average accuracy (%): 42.9 -> 42.6 | Hurt |
| Add coding data to the IFT mixture | Llama-1 and Llama-2 IFT · Symbolic reasoning | accuracy | Helped |
| Increase coding data from 50% to 100% | Llama-1 and Llama-2 IFT · Logical reasoning | accuracy | Helped |
| Replace diverse General or Half-half data with 100% coding data | Llama-1 and Llama-2 IFT · Arithmetic reasoning | accuracy | Hurt |
| Include coding data in Half-half or Code mixtures | Llama-1-13B and Llama-2-13B IFT · Symbolic and logical tasks | accuracy | Helped |
Surprising findings
- Coding data benefits extend to natural-text reasoning rather than only code-producing answers.
- For arithmetic reasoning, General or Half-half tuning performs best, so pure coding data is not optimal there.
Limitations
- Only 12 tasks in three reasoning domains; general knowledge and coding benchmarks not evaluated.
- Coding data is a single ShareGPT-derived pool; coding language types and data quality not studied.
- Larger models (Llama-2/3 70B) mentioned in conclusion but results not reported in the main text.
Experimental setup
Datasets: ShareGPT Symbolic synthetic tasks (First Letter, Last Letter, Reverse List, Coin Flip) Clutrr List Functions Babi-Inductive Babi-Deductive GSM8K SVAMP ASDiv MAWPS
Benchmarks: First Letter Last Letter Reverse List Coin Flip Clutrr List Functions Babi-Inductive Babi-Deductive GSM8K SVAMP ASDiv MAWPS
Metrics: accuracy
Baselines: General-tuned model (0% coding data) Half-half-tuned model (50% coding data)
Builds on: Alpaca ShareGPT
Closest papers by meaning
- At Which Training Stage Does Code Data Help LLMs Reasoning? (ICLR 2024 · similarity 0.90)
- To Code or Not To Code? Exploring Impact of Code in Pre-training (ICLR 2025 · similarity 0.89)
- An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.88)
- Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning (ICML 2024 · similarity 0.88)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.88)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.