Research map / Label-efficient and robust learning / Knowledge distillation
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAID, a knowledge distillation method that trains the student toward a time-dependent logit interpolation between student and teacher, and reports gains in instruction tuning, continued pre-training, and two compact models (TAID-LLM-1.5B, TAID-VLM-2B).
Problem
Knowledge distillation of causal language models suffers from teacher-student capacity gap, mode averaging, and mode collapse when the student is much smaller than the teacher.
Approach
Distill from a dynamic intermediate distribution formed by logit-level interpolation between the detached student and the teacher, with interpolation parameter t increased over training by an adaptive, objective-change-based momentum rule; a regression-based analysis argues this avoids mode collapse unlike self-distillation.
Key results
- On MT-Bench instruction tuning, TAID reached 4.05 with Phi-3-mini teacher and TinyLlama student versus 2.71 for KL and 3.48 for RKL.
- In continued pre-training on SmolLM-Corpus, TAID achieved the best Open LLM Leaderboard average of 40.10 versus 39.84 for Skew KL and 39.91 for KL.
- TAID-LLM-1.5B scored 52.27 on LightEval, above SmolLM-1.7B (51.31), and TAID-VLM-2B scored 56.43 on Open VLM Leaderboard, above Phi-3-Vision (53.60).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive interpolation updates instead of a linear schedule (ablation) | TAID · MT-Bench, Phi-3-mini-4k-instruct → TinyLlama | score: 3.44 -> 4.05 | Helped |
| use adaptive interpolation updates instead of a linear schedule (ablation) | TAID · MT-Bench, Llama-2-7b-chat → TinyLlama | score: 4.18 -> 4.27 | Helped |
| use adaptive interpolation updates instead of a linear schedule (ablation) | TAID · MT-Bench, StableLM Zephyr 3B → Pythia-410M | score: 2.88 -> 3.05 | Helped |
| use TAID rather than the strongest baseline | TinyLlama distilled from Phi-3-mini-4k-instruct · MT-Bench | score: 3.48 -> 4.05 | Helped |
| use TAID rather than the strongest baseline | TinyLlama distilled from Llama-2-7b-chat · MT-Bench | score: 4.18 -> 4.27 | Helped |
| use TAID rather than the strongest baseline | Pythia-410M distilled from StableLM Zephyr 3B · MT-Bench | score: 2.97 -> 3.05 | Helped |
| use TAID rather than KL distillation | TinyLlama · ARC | score: 44.97 -> 45.48 | Helped |
| use TAID rather than GJS distillation | TinyLlama · HellaSwag | score: 65.67 -> 65.43 | Hurt |
| use TAID rather than Skew RKL distillation | TinyLlama · MMLU | score: 26.07 -> 25.43 | Hurt |
| use TAID rather than KL distillation | TinyLlama · TrustfulQA | score: 37.95 -> 37.92 | No clear effect |
| use TAID rather than KL distillation | TinyLlama · Winogrande | score: 63.22 -> 63.38 | Helped |
| use TAID rather than Skew KL distillation | TinyLlama · GSM8K | score: 3.41 -> 2.96 | Hurt |
Surprising findings
- Standard KL-based baselines and image-classification KD methods (CTKD, DKD) sometimes underperform, with CTKD scoring far below SFT on MT-Bench.
- In the pre-training table, TAID's GSM8K score (2.96) is below Skew KL (3.41), and it trails on several individual tasks despite the best average.
Limitations
- Pre-training experiments were limited to continued pre-training on 20B tokens due to resource limits, and the gap over baselines is small (e.g., 40.10 vs 39.84).
- Theoretical analysis uses least-squares regression with an interpolation regime as a proxy for language modeling, not the actual LM objective.
- Gains on image classification were modest on CIFAR-100.
Experimental setup
Datasets: UltraChat 200k SmolLM-Corpus LAMBADA CIFAR-100 ImageNet OpenWebText
Benchmarks: MT-Bench Open LLM Leaderboard LAMBADA LightEval Open VLM Leaderboard
Metrics: MT-Bench score average accuracy accuracy probability mass
Baselines: KL divergence reverse KL total variation distance Adaptive KL GKD DistiLLM CTKD DKD GJS Skew KL Skew RKL
Builds on: Skew KL divergence GKD DistiLLM CTKD DKD
Closest papers by meaning
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.92)
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.90)
- Entropy-Aware On-Policy Distillation of Language Models (ICML 2026 · similarity 0.90)
- Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.