Research map / Label-efficient and robust learning / Knowledge distillation
Function-Consistent Feature Distillation
Dongyang Liu, Meina Kan, Shiguang Shan, Xilin CHEN
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add function-consistency matching to intermediate features (ablation) | feature distillation · CIFAR100, ResNet32x4 → ShuffleV1 | top-1 accuracy: 76.77 -> 78.12 | Improved |
| add function-consistency matching to intermediate features (ablation) | feature distillation · CIFAR100, ResNet32x4 → ShuffleV2 | top-1 accuracy: 76.49 -> 78.18 | Improved |
| add appearance matching alongside function matching (ablation) | feature distillation · CIFAR100, ResNet32x4 → ShuffleV1 | top-1 accuracy: 76.73 -> 78.12 | Improved |
| add appearance matching alongside function matching (ablation) | feature distillation · CIFAR100, ResNet32x4 → ShuffleV2 | top-1 accuracy: 77.35 -> 78.18 | Improved |
| add the student-later-part function loss L_func' (ablation) | FCFD · CIFAR100, ResNet32x4 → ShuffleV1 | top-1 accuracy: 77.92 -> 78.12 | Improved |
| add the student-later-part function loss L_func' (ablation) | FCFD · CIFAR100, ResNet32x4 → ShuffleV2 | top-1 accuracy: 78.02 -> 78.18 | Improved |
| add the teacher-later-part function loss L_func (ablation) | FCFD · CIFAR100, ResNet32x4 → ShuffleV1 | top-1 accuracy: 77.19 -> 78.12 | Improved |
| add the teacher-later-part function loss L_func (ablation) | FCFD · CIFAR100, ResNet32x4 → ResNet8x4 | top-1 accuracy: 76.62 -> 76.62 | No clear effect |
| add L_func'-L2 feature matching (ablation) | FCFD function-loss design · CIFAR100, ResNet32x4 → ResNet8x4 | top-1 accuracy: 76.62 -> 76.57 | Hurt |
| add L_func'-L2 feature matching (ablation) | FCFD function-loss design · CIFAR100, ResNet32x4 → ShuffleV1 | top-1 accuracy: 78.12 -> 77.82 | Hurt |
| add L_func'-L2 feature matching (ablation) | FCFD function-loss design · CIFAR100, ResNet32x4 → ShuffleV2 | top-1 accuracy: 78.18 -> 78.14 | No clear effect |
| use FCFD rather than the strongest listed baseline | ResNet32x4 → ResNet8x4 distillation · CIFAR100 | top-1 accuracy: 76.32 -> 76.62 | Improved |
Closest papers by meaning
- Distilling Knowledge by Mimicking Features (TPAMI 2021 · similarity 0.89)
- Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation (AAAI 2022 · similarity 0.88)
- DFD: Distilling the Feature Disparity Differently for Detectors (ICML 2024 · similarity 0.88)
- Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation (NeurIPS 2024 · similarity 0.86)
- Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory (ICLR 2024 · similarity 0.86)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
