Research map / Large language models / Instruction and fine-tuning

At Which Training Stage Does Code Data Help LLMs Reasoning?

ICLR 2024 · Large language models › Instruction and fine-tuning · approach: Calibration and knowledge in LLM tuning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematically studies whether adding code data at pre-training, instruction-tuning, or both stages improves LLM reasoning, using PanGu-α 2.6B/13B and a code-mixed CodePanGu 2.6B across six reasoning tasks.

Problem

It is unclear at which training stage (pre-training or instruction tuning) adding code data actually improves LLM reasoning, since prior models mix code in with unknown training details and differing scales.

Approach

Controlled comparison at fixed parameter scale and objective: train text-only and text+code variants at pre-training and instruction-tuning stages, evaluate on logical, legal, scientific, analogical and code reasoning, and test dynamic text/code mixing schedules during instruction tuning.

Key results

• Pre-training with code (CodePanGu 2.6B) beats text-only PanGu 2.6B on 5 of 6 reasoning tasks, e.g. E-KAR 36.12 vs 32.24 and MBPP BLEU 5.06 vs 0.52, and beats the 13B text model on most tasks.
• With chain-of-thought on ScienceQA and E-KAR, CodePanGu 2.6B reaches 70.30 and 72.84 vs 68.76 and 69.55 for the text model.
• Instruction tuning with code greatly boosts MBPP when code is used in both stages (24.88 for CC vs 0.00-5.61 for text-only instruction tuning), but gives mixed results on other reasoning tasks; a stepwise-decreasing code ratio gave the best CosQA and MBPP scores.

Measured effects reported in the paper

Surprising findings

• Adding code instruction data in the instruction-tuning stage sometimes lowered general reasoning scores (e.g., ScienceQA and E-KAR), suggesting instruction tuning with code is less effective than pre-training with code for general reasoning.
• The 2.6B code-pretrained model outperformed the 13B text-only model on most reasoning tasks.

Limitations

• Only tested on the PanGu (GPT-style) architecture at 2.6B and 13B scale, not on other LLM families.
• Pre-training mixing-strategy experiments were not run due to resource cost; mixing study only covers the instruction-tuning stage.
• Evaluation uses perplexity-based zero-shot classification and development sets where test sets are unavailable, and results are often small differences on single runs.

Experimental setup

Datasets: CodeParrot Common Crawl Alpaca-GPT-4 PromptCLUE CodeAlpaca CosQA MBPP C-Eval logic JEC-QA ScienceQA E-KAR OCNLI

Benchmarks: C-Eval logic JEC-QA ScienceQA E-KAR CosQA MBPP OCNLI CMNLI C3 DuReader

Metrics: accuracy BLEU exact match F1

Baselines: PanGu-α 2.6B (NL) PanGu-α 13B (NL)

Builds on: PanGu-α CodeParrot CodeAlpaca chain-of-thought prompting

Closest papers by meaning

• To Code or Not To Code? Exploring Impact of Code in Pre-training (ICLR 2025 · similarity 0.91)
• Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning (AAAI 2025 · similarity 0.90)
• Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus (NeurIPS 2024 · similarity 0.89)
• Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.88)
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)