Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation from Internal Representations
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, Chenlei Guo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add internal-representation distillation to soft-label distillation | BERT6 · CoLA test | MCC: 38.3 -> 41.4 | Improved |
| Add internal-representation distillation to soft-label distillation | BERT6 · QQP test | F1: 88.0 -> 89.1 | Improved |
| Add internal-representation distillation to soft-label distillation | BERT6 · MRPC test | F1: 73.9 -> 77.1 | Improved |
| Add internal-representation distillation to soft-label distillation | BERT6 · RTE test | accuracy: 59.7 -> 62.2 | Improved |
| Add KL loss on self-attention probabilities to soft-label distillation (ablation) | BERT6 · QQP development | F1: 90.65 -> 91.32 | Improved |
| Add cosine loss on [CLS] activations to soft-label distillation (ablation) | BERT6 · RTE development | accuracy: 62.46 -> 57.04 | Hurt |
| Add both KL and cosine losses at all matching layers to soft-label distillation (ablation) | BERT6 · QQP development | F1: 90.65 -> 91.38 | Improved |
| Use progressive bottom-up internal distillation, then soft-label distillation (ablation) | BERT6 · CoLA development | MCC: 41.72 -> 45.54 | Improved |
| Use stacked bottom-up internal distillation, then soft-label distillation (ablation) | BERT6 · CoLA development | MCC: 41.72 -> 46.09 | Improved |
| Continue soft-label distillation alongside stacked internal distillation (ablation) | BERT6 · MRPC development | F1: 81.39 -> 79.16 | Hurt |
| Add hard-label loss alongside stacked internal and soft-label distillation (ablation) | BERT6 · MRPC development | F1: 79.16 -> 69.68 | Hurt |
| Use internal distillation instead of soft-label-only distillation | BERT6 · QQP, with reduced training data | F1 | Improved |
Closest papers by meaning
- Knowledge Distillation for Large Language Models through Residual Learning (ICLR 2026 · similarity 0.92)
- Does Knowledge Distillation Really Work? (NeurIPS 2021 · similarity 0.90)
- Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher (NeurIPS 2020 · similarity 0.90)
- The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works (ICML 2026 · similarity 0.90)
- Knowledge Distillation from A Stronger Teacher (NeurIPS 2022 · similarity 0.90)
- Contrastive Representation Distillation (ICLR 2020 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
