Research map / Large language models / Language, safety and interpretability
To Code or Not To Code? Exploring Impact of Code in Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic controlled pre-training study (470M and 2.8B models) showing that including code in pre-training, in initialization, proportion, synthetic code quality, and cooldown, improves non-code tasks such as NL reasoning, world knowledge, and LLM-judged win-rates.
Problem
Code is routinely included in LLM pre-training even for non-code models, but its precise impact on general non-code performance has not been systematically studied.
Approach
Runs 64 controlled pre-training experiments varying where code enters training (initialization, continued pre-training, cooldown), code proportion, code quality and properties (markup, code-adjacent, synthetic verified Python), and model scale, measuring NL reasoning, world knowledge, code, and LLM-judged win-rates.
Key results
- Relative to text-only pre-training, the balanced→text variant gains 8.2% in NL reasoning, 4.2% in world knowledge, 6.6% in generative win-rate, and a 12x boost in code performance.
- Adding 10% high-quality synthetic code raises NL reasoning by 9% and code benchmarks by 44.9% over web-code-only pre-training.
- Including 20% code in cooldown improves NL reasoning by 3.6%, world knowledge by 10.1%, and code by 20% over the pre-cooldown model, with a 52.3% win-rate versus no cooldown.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Balanced→Text initialization and continued pre-training instead of text-only pre-training | Pre-training recipe · Natural language reasoning | relative performance: +8.2% | Helped |
| Use Balanced→Text initialization and continued pre-training instead of text-only pre-training | Pre-training recipe · World knowledge | relative performance: +4.2% | Helped |
| Use Balanced→Text initialization and continued pre-training instead of text-only pre-training | Pre-training recipe · Dolly-200 English | win-rate difference: +6.6% | Helped |
| Use Balanced→Text initialization and continued pre-training instead of text-only pre-training | Pre-training recipe · Code benchmarks | relative performance: 12x | Helped |
| Increase code proportion from 0% to 25% (ablation) | Pre-training from scratch · Natural language reasoning | relative performance: +3.4% | Helped |
| Increase code proportion from 0% to 25% (ablation) | Pre-training from scratch · World knowledge | relative performance: 3.4% | Hurt |
| Increase code proportion from 25% to 100% (ablation) | Pre-training from scratch · Code benchmarks | relative performance: 2.6x | Helped |
| Use 100% code rather than 0% code (ablation) | Pre-training from scratch · Natural language reasoning | relative performance: 18.3% | Hurt |
| Replace 20% of web code with markup-style data (ablation) | Code-only pre-training · Natural language reasoning | relative performance: +2.8% | Helped |
| Replace 20% of web code with markup-style data (ablation) | Code-only pre-training · Code benchmarks | relative performance: 15.7% | Hurt |
| Replace 15% of web code with code-adjacent data (ablation) | Code-only pre-training · Natural language reasoning | relative performance: +6.3% | Helped |
| Replace 15% of web code with code-adjacent data (ablation) | Code-only pre-training · Code benchmarks | relative performance: 9.4% | Hurt |
Surprising findings
- Using 100% code data in pre-training drops NL reasoning by 18.3% relative to text-only, and world knowledge drops 86% because no knowledge sources are present.
- Cooldown without code does not improve NL reasoning or code performance, only world knowledge.
Limitations
- Does not study the impact of code on safety.
- Scale limited to 470M and 2.8B parameters and 200-400B tokens due to compute cost.
- Synthetic code dataset is proprietary and small (3.2B tokens).
Experimental setup
Datasets: SlimPajama The Stack RedPajama Dolly-15K Dolly-200 English Natural Questions Open TriviaQA BoolQ PiQA SciQ SocialIQA QUAC
Benchmarks: TriviaQA NaturalQuestionsOpen BoolQ PiQA SciQ SocialIQA QUAC SuperGLUE-CB SuperGLUE-COPA StoryCloze HellaSwag Winogrande
Metrics: accuracy exact match pass@1 win-rate
Baselines: text-only pre-training balanced-only code-only pre-training (web-code-only)
Builds on: SlimPajama The Stack StarCoder RedPajama
Closest papers by meaning
- At Which Training Stage Does Code Data Help LLMs Reasoning? (ICLR 2024 · similarity 0.91)
- Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning (AAAI 2025 · similarity 0.89)
- Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions (NeurIPS 2025 · similarity 0.88)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.88)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.