Research map / Label-efficient and robust learning / Knowledge distillation
DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DA-KD, a difficulty-aware knowledge distillation framework that dynamically prunes easy samples via a distillation difficulty score with stratified sampling, plus a bidirectional discrepancy loss, cutting LLM distillation cost roughly in half.
Problem
Knowledge distillation for LLMs is expensive because existing methods use the full dataset, including easy samples whose distillation adds little value.
Approach
Score each sample by the ratio of student to teacher cross-entropy loss (DDS), progressively shrink the per-epoch dataset with a decay schedule while stratified-mixing easy and hard samples, and use a KL-based loss over mixed teacher/student distributions (BDL) that keeps gradients bounded and emphasizes hard samples.
Key results
- On instruction following, DA-KD reaches 28.11 average ROUGE-L for Llama2-2.7B (vs. 27.70 for the best baseline KD-RKL) and 30.44 for Qwen2.5-0.5B (vs. 30.24 for DistiLLM).
- DA-KD needs 1,963 training iterations versus 3,570 for baselines, and 106.35 minutes versus 213.34 (DistiLLM) and 408.24 (GKD) on four A800 GPUs.
- On GSM8K zero-shot accuracy, DA-KD reaches 54.66% on Qwen2.5-1.5B, compared with 50.80% for GKD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DA-KD instead of KD-RKL | Llama2-2.7B distillation · instruction-following evaluation average | ROUGE-L: 27.70 -> 28.11 | Helped |
| use DA-KD instead of Distillm | Llama2-1.3B distillation · instruction-following evaluation average | ROUGE-L: 24.13 -> 26.15 | Helped |
| use DA-KD instead of GKD | Qwen2.5-1.5B distillation · instruction-following evaluation average | ROUGE-L: 32.16 -> 33.01 | Helped |
| use DA-KD instead of Distillm | Qwen2.5-0.5B distillation · instruction-following evaluation average | ROUGE-L: 30.24 -> 30.44 | Helped |
| use DA-KD instead of the strongest listed baseline | Qwen2.5-1.5B distillation · GSM8K | zero-shot accuracy: 50.80 -> 54.66 | Helped |
| use DA-KD instead of Distillm | Qwen2.5-1.5B distillation · SAMSum | ROUGE-L: 39.21 -> 40.05 | Helped |
| use DA-KD instead of GKD | Llama2-1.3B distillation · GSM8K | zero-shot accuracy: 14.18 -> 14.56 | Helped |
| use DA-KD instead of GKD | Llama3.2-1B distillation · GSM8K | zero-shot accuracy: 20.17 -> 22.37 | Helped |
| add DDS-based selection rather than random data selection (ablation) | DiffUp · instruction-following evaluation average | ROUGE-L: 25.08 -> 26.15 | Helped |
| add stratified data mixing (ablation) | DiffUp · instruction-following evaluation average | ROUGE-L: 25.24 -> 26.15 | Helped |
| add difficulty-aware data updating rather than using the entire dataset (ablation) | DA-KD · instruction-following evaluation average | ROUGE-L: 25.91 -> 26.15 | Helped |
| replace KL with BDL (ablation) | DA-KD with DiffUp · instruction-following evaluation average | ROUGE-L: 21.04 -> 26.15 | Helped |
Surprising findings
- Llama2-2.7B and Qwen2.5-1.5B students slightly outperform their teachers on average instruction-following ROUGE-L.
- Performance degrades for smaller students on GSM8K mathematical reasoning, which the authors attribute to the complexity of multi-step reasoning.
Experimental setup
Datasets: databricks-dolly SAMSum GSM8K
Benchmarks: DollyEval SelfInst Super-Natural Instructions Unnatural Instruction VicunaEval SAMSum GSM8K
Metrics: ROUGE-L zero-shot accuracy training iterations training time (minutes)
Baselines: SFT KD-KL KD-RKL SeqKD GKD DistiLLM
Builds on: Knowledge distillation (Hinton 2015) MiniLLM reverse KL GKD DistiLLM
Closest papers by meaning
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.93)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.93)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.90)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.