Research map / Label-efficient and robust learning / Knowledge distillation
Heterogeneous Knowledge Distillation Using Information Flow Modeling
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use information-flow distillation with a student-compatible auxiliary teacher and critical-period weighting | CNN1 student · CIFAR-10 classification | test accuracy: 71.44 -> 71.97 | Helped |
| use the proposed information-flow distillation method | CNN1 student · CIFAR-10 metric learning without contrastive supervision | mAP (c): 51.73 -> 53.06 | Helped |
| use the proposed information-flow distillation method | CNN1 student · STL-10 distribution shift | top-100 (c): 47.98 -> 50.57 | Helped |
| use the proposed information-flow distillation method | CNN-2 student · CUB-200 | accuracy: 34.96 -> 36.95 | Helped |
| use the proposed information-flow distillation method | CNN-3 cloning a HoG feature extractor · SUN Attribute | mAP (c): 27.22 ± 2.60 -> 27.63 ± 0.62 | Helped |
| add decaying critical-period weighting for intermediate-layer information flow (ablation) | multi-layer PKT-H and single-layer PKT · CIFAR-10 training | convergence speed | Helped |
| reduce the initial intermediate-layer weight from 100 to 1 in the PKT-H decay schedule (ablation) | PKT-H · CIFAR-10 training | convergence | Hurt |
| match student layers to the appropriate auxiliary-teacher layers (CNN-1-A layers 2 and 3) (ablation) | heterogeneous multi-layer KD student · CIFAR-10 metric learning | precision | Helped |
| match the student to overly discriminative ResNet-18 layers 3 and 4 (ablation) | heterogeneous multi-layer KD student · CIFAR-10 metric learning | precision | Hurt |
| match the student to a teacher layer with lower mutual information with targets (ablation) | heterogeneous multi-layer KD student · CIFAR-10 metric learning | precision | Hurt |
| use a slightly less discriminative ResNet-18 Layer 1 for transfer (ablation) | heterogeneous multi-layer KD student · CIFAR-10 metric learning | precision | Helped |
| use an auxiliary teacher closer in complexity to the student rather than a much smaller or larger auxiliary (ablation) | CNN1 and CNN1-L students · CIFAR-10 metric learning | mAP (c): 37.89 -> 53.06; 37.51 -> 39.25 | Helped |
Closest papers by meaning
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.90)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.90)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.90)
- CIFD: Controlled Information Flow to Enhance Knowledge Distillation (NeurIPS 2024 · similarity 0.90)
- Knowledge Distillation with Auxiliary Variable (ICML 2024 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.