AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

Knowledge Distillation from Internal Representations

Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, Chenlei Guo

AAAI 2020 · Label-efficient and robust learning › Knowledge distillation · approach: Distillation with causal and adaptive guidance

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add internal-representation distillation to soft-label distillationBERT6 · CoLA testMCC: 38.3 -> 41.4Improved
Add internal-representation distillation to soft-label distillationBERT6 · QQP testF1: 88.0 -> 89.1Improved
Add internal-representation distillation to soft-label distillationBERT6 · MRPC testF1: 73.9 -> 77.1Improved
Add internal-representation distillation to soft-label distillationBERT6 · RTE testaccuracy: 59.7 -> 62.2Improved
Add KL loss on self-attention probabilities to soft-label distillation (ablation)BERT6 · QQP developmentF1: 90.65 -> 91.32Improved
Add cosine loss on [CLS] activations to soft-label distillation (ablation)BERT6 · RTE developmentaccuracy: 62.46 -> 57.04Hurt
Add both KL and cosine losses at all matching layers to soft-label distillation (ablation)BERT6 · QQP developmentF1: 90.65 -> 91.38Improved
Use progressive bottom-up internal distillation, then soft-label distillation (ablation)BERT6 · CoLA developmentMCC: 41.72 -> 45.54Improved
Use stacked bottom-up internal distillation, then soft-label distillation (ablation)BERT6 · CoLA developmentMCC: 41.72 -> 46.09Improved
Continue soft-label distillation alongside stacked internal distillation (ablation)BERT6 · MRPC developmentF1: 81.39 -> 79.16Hurt
Add hard-label loss alongside stacked internal and soft-label distillation (ablation)BERT6 · MRPC developmentF1: 79.16 -> 69.68Hurt
Use internal distillation instead of soft-label-only distillationBERT6 · QQP, with reduced training dataF1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy