Research map / Large language models / Language, safety and interpretability
Magicoder: Empowering Code Generation with OSS-Instruct
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OSS-Instruct, which prompts gpt-3.5-turbo with open-source code snippets to synthesize 75K coding instruction pairs, used to finetune CodeLlama-7B and DeepSeek-Coder-6.7B into Magicoder models, with Evol-Instruct continuation yielding MagicoderS variants.
Problem
Synthetic instruction data for code generated by LLMs (Self-Instruct, Evol-Instruct) relies on narrow predefined tasks and heuristics, inheriting LLM biases and limiting diversity and realism.
Approach
Use random 1-15 line snippets from real open-source code (starcoderdata) as seeds so a teacher LLM generates a self-contained coding problem and solution, giving diverse, realistic, controllable instruction data; it is orthogonal to Evol-Instruct and can be combined with it.
Key results
- MagicoderS-CL-7B reaches 70.7 pass@1 on HumanEval and 66.5 on HumanEval+, on par with or above ChatGPT (72.6 / 65.9).
- Magicoder-CL-7B (OSS-Instruct only) outperforms WizardCoder-SC-15B on HumanEval+ and MultiPL-E; MagicoderS-CL improves DS-1000 by 8.3 points over WizardCoder-SC-15B.
- MagicoderS-DS-6.7B achieves 76.8 pass@1 on HumanEval, using about 8x fewer training tokens than DeepSeek-Coder-Instruct-6.7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OSS-Instruct tuning | CodeLlama-7B · HumanEval+ | pass@1 (%): 34.1 -> 55.5 | Helped |
| Add Evol-Instruct continuation after OSS-Instruct | Magicoder-CL (CodeLlama-7B) · HumanEval+ | pass@1 (%): 55.5 -> 66.5 | Helped |
| Add OSS-Instruct tuning | CodeLlama-7B · DS-1000 completion overall | pass@1: 28.0 -> 29.9 | Helped |
| Add Evol-Instruct continuation after OSS-Instruct | Magicoder-CL (CodeLlama-7B) · DS-1000 completion overall | pass@1: 29.9 -> 37.5 | Helped |
| Train on non-Python data (32K) (ablation) | CodeLlama-7B · HumanEval+ | pass@1: 34.1 -> 44.5 | Helped |
| Train on Python-only data (43K) instead of both sources (ablation) | Magicoder-CL (CodeLlama-7B) · MultiPL-E other languages | average pass@1: 37.8 -> 32.7 | Hurt |
| Train on non-Python data only instead of both sources (ablation) | Magicoder-CL (CodeLlama-7B) · HumanEval+ | pass@1: 55.5 -> 44.5 | Hurt |
| Replace OSS-Instruct data with comment-function pairs (ablation) | CodeLlama-7B · MultiPL-E | pass@1: 29.6 -> 24.1 | Hurt |
| Replace OSS-Instruct data with comment-function pairs (ablation) | CodeLlama-7B · HumanEval+ | pass@1: 34.1 -> 34.1 | No clear effect |
| Use 20K OSS-Instruct data generated by Mixtral (ablation) | CodeLlama-7B · HumanEval+ | pass@1: 34.1 -> 55.5 | Helped |
| Remove partially implemented/noisy samples (ablation) | CodeLlama-7B trained on OSS-Instruct · HumanEval+ | pass@1: 55.5 -> 54.9 | Hurt |
Surprising findings
- Finetuning on 75K raw comment-function pairs from the same corpus does not improve HumanEval+ (34.1 unchanged) and lowers MultiPL-E performance (24.1 vs 29.6 base).
- Training on 20K OSS-Instruct samples generated by Mixtral-8x7B-Instruct outperforms the Mixtral teacher on HumanEval+ (55.5 vs 39.6), suggesting gains are not only distillation.
Limitations
- Data generated by gpt-3.5-turbo-1106 only; the authors note they did not try GPT-4 as teacher.
- Generated data contains noise such as incomplete solutions, which is kept intentionally and not thoroughly quantified.
- Evaluation focuses on function-level benchmarks; the 7B results are on greedy decoding with limited multilingual training data.
Experimental setup
Datasets: starcoderdata evol-codealpaca-v1 HumanEval MBPP DS-1000 GSM8K APPS
Benchmarks: HumanEval HumanEval+ MBPP MBPP+ MultiPL-E DS-1000 APPS
Metrics: pass@1
Baselines: CodeLlama-Python-7B WizardCoder-CL-7B WizardCoder-SC-15B WizardCoder-CL-34B GPT-3.5 Turbo GPT-4 Turbo DeepSeek-Coder-Instruct-6.7B StarCoder CodeGen-Mono
Builds on: Self-Instruct Evol-Instruct Code Alpaca WizardCoder EvalPlus
Closest papers by meaning
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct (ICLR 2024 · similarity 0.90)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.88)
- Training Language Models on Synthetic Edit Sequences Improves Code Synthesis (ICLR 2025 · similarity 0.88)
- InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · similarity 0.87)
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.87)
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.