Research map / Label-efficient and robust learning / Knowledge distillation
Beyond Student: An Asymmetric Network for Neural Network Inheritance
Yiyun Zhou, Jingwei Shi, Mingjing Xu, Zhonghua Jiang, Jingyuan Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SVD initialization (ablation) | InherNet Large · CIFAR-100 (ResNet56 -> ResNet20) | top-1 accuracy (%): 69.35 -> 73.67 | Improved |
| add SVD initialization (ablation) | InherNet Large · CIFAR-100 (ResNet32x4 -> ResNet8x4) | top-1 accuracy (%): 76.68 -> 78.53 | Improved |
| add gating mechanism (ablation) | InherNet Large · CIFAR-100 (ResNet56 -> ResNet20) | top-1 accuracy (%): 73.22 -> 73.67 | Improved |
| use asymmetric instead of symmetric structure (ablation) | InherNet Large · CIFAR-100 (ResNet56 -> ResNet20) | top-1 accuracy (%): 73.28 -> 73.67 | Improved |
| use one-down-many-ups instead of inverted many-downs-one-up (ablation) | InherNet Large · CIFAR-100 (VGG13 -> VGG8) | top-1 accuracy (%): 71.26 -> 75.16 | Improved |
| use one-down-many-ups instead of inverted many-downs-one-up (ablation) | InherNet Large · CIFAR-100 (ResNet32x4 -> ResNet8x4) | top-1 accuracy (%): 78.79 -> 78.53 | Hurt |
| use one-down-many-ups instead of inverted many-downs-one-up (ablation) | InherNet · GLUE MRPC | accuracy (%): 87.36 -> 88.74 | Improved |
| replace KD student with InherNet Large | ResNet20 student · CIFAR-100 (ResNet56 teacher) | top-1 accuracy (%): 72.33 -> 73.67 | Improved |
| replace KD student with InherNet Large | ResNet8x4 student · CIFAR-100 (ResNet32x4 teacher) | top-1 accuracy (%): 78.28 -> 78.53 | Improved |
| replace T5-Small+KD with InherNet | T5-Small · GLUE (8 tasks) | accuracy/MCC (%): MNLI 53.81 -> 56.34; RTE 74.31 -> 72.93; QQP 84.92 -> 84.69 | Mixed |
| replace CLIP-KD student with InherNet | ResNet-101 teacher · ImageNet zero-shot (CC3M pretrain) | top-1 accuracy (%): 17.30 -> 17.39 | Improved |
| replace CLIP-KD EfficientNet-B0 with InherNet | ResNet-101 teacher · CC3M I2T | R@1 (%): 33.65 -> 32.17 | Hurt |
Closest papers by meaning
- Densely Guided Knowledge Distillation Using Multiple Teacher Assistants (ICCV 2021 · similarity 0.90)
- Similarity-Preserving Knowledge Distillation (ICCV 2019 · similarity 0.89)
- Few Sample Knowledge Distillation for Efficient Network Compression (CVPR 2020 · similarity 0.88)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.87)
- EPSD: Early Pruning with Self-Distillation for Efficient Model Compression (AAAI 2024 · similarity 0.87)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
