Research map / Large language models / Instruction and fine-tuning
WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Evol-Instruct, which uses ChatGPT to iteratively evolve Alpaca's 52k instructions into more complex and diverse ones, then fine-tunes LLaMA models into WizardLM.
Problem
Human-written instruction data is expensive, slow to produce, and skewed toward easy instructions, limiting instruction-tuned LLMs on complex tasks.
Approach
Automatically evolve instructions in depth (add constraints, deepening, concretizing, more reasoning steps, complicating input) and in breadth (mutation to new long-tail instructions) using an LLM prompted with Evol-Instruct templates, filter failed evolutions with an elimination step, and fine-tune LLaMA on the merged pool.
Key results
- WizardLM-13b averages 58.96 across nine benchmarks vs 54.60 for Vicuna-13b and 43.44 for Alpaca-13b, with HumanEval 24.0 and GSM8k 37.15.
- WizardLM-70b reaches 71.33 average and 99.7 on WizardEval, the best open-source result in the paper.
- Ablation: ChatGPT-judged difficulty rises across evolution rounds C1 to C4 (5.48 to 7.08), and performance improves with complexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on Evol-Instruct data instead of Alpaca instruction data | LLaMA 13B · GSM8K | pass@1: 8.35 -> 37.15 | Helped |
| Fine-tune on Evol-Instruct data instead of Alpaca instruction data | LLaMA 13B · HumanEval | pass@1: 9.2 -> 24.0 | Helped |
| Use ShareGPT instead of Alpaca as seed data for Evol-Instruct (ablation) | WizardLM-13b · Automatic benchmarks | Avg.: 58.96 -> 61.87 | Helped |
| Use ShareGPT instead of Alpaca as seed data for Evol-Instruct (ablation) | WizardLM-13b · GSM8K | pass@1: 37.15 -> 31.46 | Hurt |
| Train on the full 250K evolved data instead of 70k samples (ablation) | WizardLM-13b · Automatic benchmarks | Avg.: 58.96 -> 60.30 | Helped |
| Use LlaMA-2-70B-Chat instead of ChatGPT for evolution (ablation) | WizardLM-13b · Automatic benchmarks | Avg.: 58.96 -> 56.27 | Hurt |
| Use Supernatural Instructions instead of Evol-Instruct data (ablation) | LLaMA-13b · Automatic benchmarks | Avg.: 58.96 -> 37.73 | Hurt |
| Apply Evol-Instruct instead of Alpaca data (ablation) | Mistral-7B · Automatic benchmarks | Avg.: 52.87 -> 65.81 | Helped |
| Increase the pretrained model size to LLaMA-1 65B (ablation) | WizardLM · Automatic benchmarks | Avg.: 58.96 -> 69.40 | Helped |
| Use LLaMA-2 70B as the pretrained model (ablation) | WizardLM · Automatic benchmarks | Avg.: 58.96 -> 71.33 | Helped |
| Increase instruction complexity across evolution rounds (ablation) | WizardLM fine-tuning · Automatic benchmarks | average score | Helped |
Surprising findings
- ShareGPT as the seed (WizardLM-13b ShareGPT Seed) gave higher average than the Alpaca-seeded model, but was worse on GSM8k, attributed to lower math content (4.3% vs 11.8%).
Limitations
- Reliance on GPT-4 and human evaluation has scalability and reliability challenges, as the paper acknowledges.
- The WizardEval test set may not represent all scenarios or domains where LLMs are applied.
- Difficulty judgments come from ChatGPT prompting, and checkpoint results fluctuate across epochs.
Experimental setup
Datasets: Alpaca 52k ShareGPT Supernatural Instructions
Benchmarks: MMLU ARC HellaSwag TruthfulQA HumanEval GSM8k AlpacaEval MT-Bench WizardEval
Metrics: accuracy pass@1 win rate average score
Baselines: Alpaca-13b Vicuna-13b Baize-13b CAMEL-13b Tulu-13b ChatGPT-3.5 Supernatural Instructions
Builds on: Self-Instruct Alpaca Vicuna LLaMA
Closest papers by meaning
- InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (ICML 2024 · similarity 0.92)
- WizardCoder: Empowering Code Large Language Models with Evol-Instruct (ICLR 2024 · similarity 0.90)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.90)
- Can Large Language Models Understand Real-World Complex Instructions? (AAAI 2024 · similarity 0.90)
- Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation (ICLR 2025 · similarity 0.90)
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.