Research map / Large language models / Instruction and fine-tuning
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts Evol-Instruct to code by iteratively evolving Code Alpaca instructions with code-specific heuristics and fine-tuning StarCoder-15B and CodeLlama-34B-Python into WizardCoder models.
Problem
Instruction fine-tuning for code LLMs is under-researched; open-source code models lag behind closed-source models on code generation benchmarks.
Approach
Code Evol-Instruct iteratively makes code instructions more complex using coding-specific heuristics (LeetCode-style constraints, erroneous code as misdirection, time/space complexity requirements), with an external dev set as an evolution stop criterion, and fine-tunes Code LLMs on the merged evolved data.
Key results
- WizardCoder-34B reaches 71.5 pass@1 on HumanEval and 61.2 on MBPP, the highest among open-source models in the paper's table.
- WizardCoder-34B surpasses GPT-3.5 (ChatGPT) on HumanEval+ (64.6 vs. 63.4) per the EvalPlus leaderboard setting.
- WizardCoder-15B scores 57.3 on HumanEval, outperforming StarCoder (33.6) and OctoCoder (46.2) by large margins.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune StarCoder 15B with Code Evol-Instruct data | StarCoder 15B · HumanEval | pass@1 (%): 33.6 -> 57.3 | Helped |
| Fine-tune CodeLlama-34B-Python with Code Evol-Instruct data | CodeLlama-34B-Python · HumanEval | pass@1 (%): 53.7 -> 71.5 | Helped |
| Use GPT-4 instead of GPT-3.5 to generate evolved instructions (ablation) | Code Evol-Instruct with StarCoder-15B base · HumanEval | pass@1: 59.8 -> 62.2 | Helped |
| Use GPT-4 instead of GPT-3.5 to generate evolved instructions (ablation) | Code Evol-Instruct with CodeLlama-34B base · HumanEval | pass@1: 73.2 -> 73.8 | Helped |
| Use CodeLlama as the evolution model instead of GPT-3.5 (ablation) | Code Evol-Instruct with StarCoder-15B base · HumanEval | pass@1: 59.8 -> 55.5 | Hurt |
| Use CodeLlama-34B as the evolution model instead of GPT-3.5 (ablation) | Code Evol-Instruct with CodeLlama-34B base · HumanEval | pass@1: 73.2 -> 70.1 | Hurt |
| Train on evolved Round 1 data instead of seed Round 0 data, with similar sample counts (ablation) | WizardCoder training · HumanEval | pass@1: 45.7 -> 56.1 | Helped |
| Train on evolved Round 2 data instead of seed Round 0 data, with similar sample counts (ablation) | WizardCoder training · HumanEval | pass@1: 45.7 -> 53.0 | Helped |
| Train on evolved Round 3 data instead of seed Round 0 data, with similar sample counts (ablation) | WizardCoder training · HumanEval | pass@1: 45.7 -> 54.3 | Helped |
| Train on evolved Round 4 data instead of seed Round 0 data, with similar sample counts (ablation) | WizardCoder training · HumanEval | pass@1: 45.7 -> 51.2 | Helped |
| Train on evolved Round 1 data instead of seed Round 0 data, with the same token count (ablation) | WizardCoder training · HumanEval | pass@1: 44.5 -> 51.8 | Helped |
| Train on evolved Round 2 data instead of seed Round 0 data, with the same token count (ablation) | WizardCoder training · HumanEval | pass@1: 44.5 -> 52.4 | Helped |
Surprising findings
- Evolved data with higher complexity outperforms same-sized seed data, and gains are not explained by more tokens or samples alone.
- Using GPT-4 as the evolution model gives only a marginal gain over GPT-3.5 (73.8 vs. 73.2 on 34B), despite a large coding-ability gap.
Limitations
- Still falls significantly behind GPT-4 on the benchmarks reported.
- Relies on proprietary gpt-3.5-turbo for evolving data and generating responses.
- DS-1000 results only reported for the 15B model because the 34B model was hard to align with the benchmark framework.
Experimental setup
Datasets: Code Alpaca MBPP HumanEval
Benchmarks: HumanEval HumanEval+ MBPP DS-1000 MultiPL-E
Metrics: pass@1
Baselines: GPT-3.5 (ChatGPT) GPT-4 Claude Bard CodeLlama-Instruct CodeLlama-Python StarCoder OctoCoder Unnatural-CodeLlama PanguCoder2
Builds on: Evol-Instruct Code Alpaca self-instruct StarCoder CodeLlama
Closest papers by meaning
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (ICLR 2024 · similarity 0.90)
- Magicoder: Empowering Code Generation with OSS-Instruct (ICML 2024 · similarity 0.90)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.89)
- InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (ICML 2024 · similarity 0.87)
- InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · similarity 0.87)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.