Research map / Large language models / Retrieval and knowledge
DDK: Distilling Domain Knowledge for Efficient Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDK, a white-box LLM distillation framework that dynamically reweights domain sampling in the distillation dataset according to teacher-student perplexity gaps per domain, with smooth factor updates.
Problem
Knowledge distillation for LLMs treats all data domains uniformly, ignoring that the student-teacher performance gap differs across domains, so well-matched domains get over-attention while lagging domains get too little.
Approach
Compute a per-domain discrepancy factor from the ratio of student to teacher validation perplexity, use it to set domain sampling probabilities during distillation, and smooth the factor across intervals with a history-weighted exponential update to stabilize training.
Key results
- On Qwen-1.5 14B to 1.8B distillation, DDK reaches 55.41 average over 11 benchmarks vs 52.87 for standard KD and 49.39 for the student.
- On LLaMA2 13B to TinyLLaMA 1.1B, DDK averages 29.32 vs 26.18 for KD.
- On StarCoder 15.5B to 3B, DDK improves average exact match to 37.6 vs 34.6 for KD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DDK instead of KD | Qwen-1.5 1.8B · Qwen-1.5 benchmarks | Avg.: 52.87 -> 55.41 | Helped |
| Apply DDK instead of MiniLLM | TinyLLaMA 1.1B · LLaMA benchmarks | Avg.: 26.37 -> 29.32 | Helped |
| Apply DDK instead of KD | StarCoder 3B · CrossCodeEval | EM: 34.6 -> 37.6 | Helped |
| Set distillation weight γ to 0, 0.1, 0.3, or 0.5 rather than 0.2 (ablation) | DDK with Qwen-1.5 1.8B · MMLU validation | score: 45.57; 44.43; 43.44; 44.72; 43.72 | Hurt |
| Vary distillation temperature from T=1.0 to T=0.1, 0.2, 0.5, or 2.0 (ablation) | DDK with Qwen-1.5 1.8B · Qwen-1.5 validation benchmarks | AVG (%): 56.87; 56.96; 57.36; 57.46; 57.12 | No clear effect |
| Remove factor smooth updating (ablation) | DDK with Qwen-1.5 1.8B · MMLU validation | score: 45.57 -> 43.80 | Hurt |
| Use equal sampling across domains instead of domain-guided sampling (ablation) | DDK with Qwen-1.5 1.8B · MMLU validation | score: 45.57 -> 44.75 | Hurt |
| Set distillation interval K to 50, 100, 500, 1,500, or 2,000 rather than 1,000 (ablation) | DDK with Qwen-1.5 1.8B · Qwen-1.5 validation benchmarks | AVG (%): 54.54; 55.68; 56.94; 57.46; 56.51; 55.57 | Hurt |
| Increase training tokens from 5B to 15B (ablation) | DDK with Qwen-1.5 1.8B · Qwen-1.5 validation benchmarks | AVG (%): 55.07 -> 57.48 | Helped |
| Apply DDK instead of TED | Qwen-1.5 1.8B · Qwen-1.5 few-shot benchmarks (5-shot) | Avg.: 57.48 -> 60.36 | Helped |
Surprising findings
- CPT combined with DoReMi domain reweighting gave only marginal gains over plain CPT and even lowered HumanEval (8.75 vs 13.12) on the Qwen 1.8B student.
- Under the Qwen 7B teacher setting, DDK's HumanEval score (11.88) is below the KD baseline (13.75).
Limitations
- Requires tuning several hyperparameters (interval K, smoothing alpha, gamma).
- Only small student sizes (1.1B, 1.8B, 4B) and teachers up to 14B were tested due to GPU limits.
- Validation sets are small (500 samples per domain) and results are single-run without variance reported.
Experimental setup
Datasets: RedPajama Chinese Books Chinese CommonCrawl OpenWebMath The Stack V2
Benchmarks: C-Eval MMLU RACE C3 WinoGrande GSM8K CommonsenseQA ARC-E ARC-C HumanEval MBPP CrossCodeEval
Metrics: accuracy average accuracy exact match edit similarity perplexity
Baselines: CPT CPT & DoReMi KD TED MiniLLM
Builds on: DoReMi MiniLLM TED Knowledge distillation (Hinton et al.)
Closest papers by meaning
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.93)
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.92)
- MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.91)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024 · similarity 0.90)
- DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.