Research map / Large language models / Language, safety and interpretability

DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DistiLLM-2, a contrastive LLM distillation loss that applies skew KL to teacher responses and skew reverse KL to student responses, with curriculum-based alpha and beta schedules, improving student models across instruction-following, math, code, preference alignment, and VLM tasks.

Problem

Standard LLM distillation applies the same loss to teacher- and student-generated responses, ignoring the synergy between loss formulation and data type; naive DPO-style distillation (DPKD) suffers reward hacking.

Approach

Use skew KL on teacher-generated outputs (pulling up teacher-likely regions) and skew reverse KL on student-generated outputs (pushing down student samples) in one contrastive objective, with a linear rather than log-sigmoid formulation, plus curriculum-based adaptive alpha and a gradually increasing SRKL weight beta.

Key results

• On instruction-following, DistiLLM-2 outperforms the second-best method by +2.34% (Qwen2-1.5B), +1.95% (Danube2-1.8B), and +4.53% (Gemma2-2B) average win rate.
• Component analysis on DistiLLM v1 raises average score from 41.55 to 46.50 when contrastive loss, increased beta, and curriculum alpha are all added.
• On VLM distillation (LLaVA-1.5-7B to TinyLLaVA-1.4B), DistiLLM-2 beats GKD and DistiLLM by +2.01% and +4.61% on average.

Measured effects reported in the paper

Surprising findings

• Naive DPO-style distillation (DPKD) suffers reward hacking, degrading student outputs to degenerate sentences (NLL of student responses rises to 91.25).
• Using speculative-decoded responses for the teacher (SKL) term does not help, and higher-quality responses from stronger LLMs do not beat teacher generations.

Limitations

• Relies on a first-order Mercator approximation for the curriculum alpha, which is less accurate when probabilities are far from 1.
• Heavy reliance on GPT-4o/GPT-4o-mini LLM-as-judge for main instruction-following results.
• Mostly evaluated on small student models with LoRA; the prior version's pretraining loss was dropped.

Experimental setup

Datasets: UltraChat200k MetaMathQA WizardCoder RLAIF-V-Dataset AlpacaEval Evol-Instruct UltraFeedback GSM8K MATH HumanEval MBPP OK-VQA

Benchmarks: AlpacaEval Evol-Instruct UltraFeedback GSM8K MATH HumanEval MBPP OK-VQA TextVQA databricks-dolly-15k

Metrics: win rate accuracy pass@1 ROUGE-L

Baselines: KD SeqKD ImitKD GKD DistiLLM Speculative KD DPO DPKD MiniLLM

Builds on: DistiLLM DPO DPKD GKD MiniLLM

Closest papers by meaning

• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.91)
• DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.91)
• Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.91)
• Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)
• MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.90)
• DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.90)