Research map / Label-efficient and robust learning / Knowledge distillation
MixKD: Towards Efficient Distillation of Large-scale Language Models
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, Lawrence Carin
This paper tried a common change: Add mixup or CutMix
Add MixKD (student mixup, teacher mixup distillation, and backtranslation) · GLUE test tasks · accuracy or F1 Mixed
What 547 accepted papers report for this change
- 416 improved
- 28 no clear effect
- 14 mixed
- 89 hurt
- Lightweight Federated Incremental Learning via Decoupled Replay ICML 2026 · add mixup to uploaded features · CIFAR100 (α=1.0) · A(f): 33.20 -> 35.61Improved
- Quantifying and Optimizing Simplicity via Polynomial Representations ICML 2026 · add embedding mixup · GLUE RTE, MRPC, CoLA · accuracy / Matthews corr.: MRPC 86.74 -> 86.30; CoLA 62.31 -> 61.04; RTE unchanged 70.28Hurt
- Fix the Mind, Not the Move: Interpretable AI Assistance via Knowledge-Gap Localization ICML 2026 · add auxiliary mixup generalization loss · Breakfast · System recall: 0.221 -> 0.762Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MixKD (student mixup, teacher mixup distillation, and backtranslation) | BERT6 student · GLUE test tasks | accuracy or F1 | Mixed |
| Add student mixup loss to teacher mixup distillation with backtranslation (ablation) | BERT6-TMKD+BT · MRPC dev | F1: 89.79±0.27 -> 89.64±0.38 | Hurt |
| Add student mixup loss to teacher mixup distillation with backtranslation (ablation) | BERT6-TMKD+BT · MNLI-m dev | accuracy: 82.05±0.11 -> 82.41±0.12 | Improved |
| Add student mixup loss to teacher mixup distillation with backtranslation (ablation) | BERT6-TMKD+BT · QNLI dev | accuracy: 88.42±0.06 -> 88.76±0.15 | Improved |
| Add student mixup loss to teacher mixup distillation with backtranslation (ablation) | BERT6-TMKD+BT · RTE dev | accuracy: 69.37±0.50 -> 68.02±0.11 | Hurt |
| Increase mixup ratio above 1 (ablation) | MixKD · SST-2 | accuracy | No clear effect |
| Add MixKD rather than the TinyBERT data augmentation module | BERT6 · MNLI | accuracy: 81.5 -> 82.5 | Improved |
| Add MixKD rather than the TinyBERT data augmentation module | BERT6 · SST-2 | accuracy: 91.3 -> 92.1 | Improved |
| Use MixKD instead of fine-tuning alone with limited training data | BERT3 · QQP, MNLI-m, MNLI-mm, and QNLI at 10% and 1% data | accuracy: +2.0%, +3.0%; +3.9%, +3.4%; +4.4%, +3.3%; +2.4%, +4.1% | Improved |
Closest papers by meaning
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.91)
- DDK: Distilling Domain Knowledge for Efficient Large Language Models (NeurIPS 2024 · similarity 0.91)
- WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.90)
- DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models (ICML 2025 · similarity 0.90)
- On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
