Research map / Large language models / Instruction and fine-tuning
OctoPack: Instruction Tuning Code Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CommitPack/CommitPackFT, a 4TB Git-commit instruction dataset, and HumanEvalPack, a 3-task x 6-language benchmark, to instruction-tune StarCoder (OctoCoder) and CodeGeeX2 (OctoGeeX) with permissive licenses.
Problem
Instruction tuning for code LLMs typically relies on synthetic data generated by closed-source models (e.g., OpenAI), which is legally and practically problematic, and existing code benchmarks cover mainly Python code synthesis, leaving code repair and explanation under-evaluated.
Approach
Use natural human instructions from Git commit messages paired with code diffs as a permissively licensed instruction-tuning source, and extend HumanEval into code repair, explanation, and synthesis tasks across six languages, with execution-based pass@k for explanation via chained regeneration.
Key results
- OctoCoder (StarCoder instruction-tuned on CommitPackFT+OASST) reaches 46.2% pass@1 on Python HumanEval, the best among models not trained on OpenAI outputs.
- OctoCoder averages 35.5% pass@1 across HumanEvalPack's 3 tasks and 6 languages, the best among permissive models (vs. StarCoder 27.6%).
- GPT-4 scores 78.3% average on HumanEvalPack synthesis and 47.8% on code repair, well above all permissive models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| instruction-tune StarCoder with CommitPackFT + OASST instead of the base model | StarCoder · HumanEvalPack Code Synthesis (Python) | pass@1 (%): relative 38% | Helped |
| add CommitPackFT to instruction tuning data (ablation) | StarCoder · HumanEvalPack Code Repair (Python) | pass@1 (%) | Helped |
| use OASST alone instead of CommitPackFT-containing data (ablation) | StarCoder · HumanEvalPack Code Repair (Python) | pass@1 (%) | Hurt |
| include natural-language-target samples such as OASST (ablation) | StarCoder instruction tuning · HumanEvalPack Code Explanation (Python) | pass@1 (%) | Helped |
| use Self-Instruct code-target data without natural-language targets (ablation) | StarCoder instruction tuning · HumanEvalPack Code Explanation (Python) | pass@1 (%) | Hurt |
| mix xP3x with OASST instead of CommitPackFT with OASST (ablation) | StarCoder instruction tuning · HumanEvalPack Code Explanation (Python) | pass@1 (%) | Hurt |
| combine CommitPackFT with OASST (ablation) | StarCoder instruction tuning · HumanEvalPack average across tasks | pass@1 (%): absolute 23% | Helped |
| add more programming-language coverage from CommitPackFT during instruction tuning | StarCoder · HumanEvalPack new-language tasks | pass@1 (%) | Helped |
Surprising findings
- Pure code synthesis is near saturation for GPT-4 (86.6% Python), while code repair and explanation remain much harder.
- Instruction tuning enabled BLOOMZ to generalize to Go and Rust, languages absent from its instruction data.
Limitations
- GPT-4 evaluation may be affected by possible HumanEval contamination and unknown API changes.
- Models struggle to adhere to explanation length limits, and code repair with small targeted changes remains hard.
- Commit data only goes up to 2016 and is restricted to single-file changes.
Experimental setup
Datasets: CommitPack CommitPackFT OASST xP3x StarCoder Self-Instruct HumanEval
Benchmarks: HumanEvalPack
Metrics: pass@1
Baselines: WizardCoder InstructCodeT5+ GPT-4 BLOOMZ StarChat-β CodeGeeX2 StarCoder
Builds on: HumanEval StarCoder Self-Instruct xP3x OASST
Closest papers by meaning
- CodeSync: Synchronizing Large Language Models with Dynamic Code Evolution at Scale (ICML 2025 · similarity 0.87)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.86)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.85)
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct (ICLR 2024 · similarity 0.85)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.