Research map / Label-efficient and robust learning / Knowledge distillation
Cross-Layer Distillation with Semantic Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use semantic-calibrated cross-layer attention for feature-map distillation instead of fixed layer matching | student networks on CIFAR-100 · CIFAR-100 | Top-1 test accuracy (%): 75.27 ± 0.13 vs 73.12 ± 0.10 | Helped |
| Use semantic-calibrated cross-layer attention for feature-map distillation instead of fixed layer matching | student networks on ImageNet · ImageNet | Top-1 test accuracy (%): 70.87 vs 70.62 | Helped |
| Add learned personalized attention over teacher layers rather than equal allocation (ablation) | SemCKD · CIFAR-100, VGG-8 / ResNet-32x4 | Top-1 test accuracy (%): 72.94 ± 0.87 -> 75.27 ± 0.13 | Helped |
| Add feature-map projection to align student and target layer dimensions (ablation) | SemCKD · CIFAR-100, VGG-8 / ResNet-32x4 | Top-1 test accuracy (%): 72.51 ± 0.16 -> 75.27 ± 0.13 | Helped |
| Use nonlinear query/key MLPs rather than simple linear transformations (ablation) | SemCKD · CIFAR-100, VGG-8 / ResNet-32x4 | Top-1 test accuracy (%): 72.78 ± 0.29 -> 75.27 ± 0.13 | Helped |
| Calculate pairwise similarity matrices before deriving query and key vectors (ablation) | SemCKD · CIFAR-100, VGG-8 / ResNet-32x4 | Top-1 test accuracy (%): 74.75 ± 0.14 -> 75.27 ± 0.13 | Helped |
| Assign independent attention weights to each training instance rather than sharing weights (ablation) | SemCKD · CIFAR-100, VGG-8 / ResNet-32x4 | Top-1 test accuracy (%): 74.87 ± 0.21 -> 75.27 ± 0.13 | Helped |
| Soften attention with temperature τ=4 (ablation) | SemCKD · CIFAR-100, VGG-8 / ResNet-32x4 | Top-1 test accuracy (%): +0.47% | Helped |
| Soften attention with temperature τ=4 (ablation) | SemCKD · CIFAR-100, MobileNetV2 / WRN-40-2 | Top-1 test accuracy (%): +0.86% | Helped |
| Use attention temperature τ less than 1, sharpening the weights (ablation) | SemCKD · CIFAR-100, multiple teacher-student combinations | Top-1 test accuracy (%) | Hurt |
| Add pre-activation distillation to SemCKDτ (ablation) | SemCKDτ, MobileNetV2 student · CIFAR-100, MobileNetV2 student settings | Top-1 test accuracy (%) | Hurt |
| Combine SemCKD with CRD feature-embedding distillation | SemCKD · CIFAR-100, tested network combinations | Top-1 test accuracy (%) | Helped |
Closest papers by meaning
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.89)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.89)
- C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation (CVPR 2024 · similarity 0.88)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
- CIFD: Controlled Information Flow to Enhance Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.