Research map / Architectures and efficiency / RNNs and normalization
ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace selected-layer matching with attention-weighted combinations of all teacher layers | BERT student distilled from BERT Base · GLUE validation, 2-layer student | Average: 66.28 -> 67.08 | Helped |
| Replace selected-layer matching with attention-weighted combinations of all teacher layers | BERT student distilled from BERT Base · GLUE validation, 4-layer student | Average: 76.44 -> 77.01 | Helped |
| Replace selected-layer matching with attention-weighted combinations of all teacher layers | BERT student distilled from BERT Base · GLUE validation, 6-layer student | Average: 80.17 -> 80.41 | Helped |
| Add intermediate-layer distillation to final-prediction distillation | BERT student distilled from BERT Base · GLUE validation, 4-layer student | Average: 76.02 -> 76.44 | Helped |
| Replace layer skipping with concatenation-and-projection combinations | CKD BERT student · GLUE validation, 4-layer student | Average: 76.44 -> 76.73 | Helped |
| Use partial bucket overlap instead of no overlap (ablation) | CKD BERT student · GLUE validation, 4-layer student | Average: 76.37 -> 76.73 | Helped |
| Use partial bucket overlap instead of no overlap (ablation) | ALP BERT student · GLUE validation, 4-layer student | Average: 76.82 -> 76.89 | Helped |
| Remove bucket boundaries and attend over all teacher layers (ablation) | ALP BERT student · GLUE validation, 4-layer student | Average: 76.89 -> 77.01 | Helped |
| Remove bucket boundaries and attend over all teacher layers (ablation) | ALP BERT student · GLUE validation, 6-layer student | Average: 80.52 -> 80.41 | Hurt |
| Replace dot-product attention with single-head key-query-value attention (ablation) | ALP BERT student · GLUE validation, 4-layer student | Average: 77.01 -> 76.49 | Hurt |
| Replace dot-product attention with 4-head key-query-value attention (ablation) | ALP BERT student · GLUE validation, 4-layer student | Average: 77.01 -> 76.82 | Hurt |
| Increase key-query-value attention from one head to four heads (ablation) | ALP key-query-value variants · GLUE validation, 4-layer student | Average: 76.49 -> 76.82 | Helped |
Closest papers by meaning
- Cross-Layer Distillation with Semantic Calibration (AAAI 2021 · similarity 0.91)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.90)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.89)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.89)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.