Research map / Architectures and efficiency / Pruning and efficient tuning
Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use fixed-depth student (no stochastic depth) (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 74.5 -> 75.5 | Helped |
| use soft labels instead of hard-label distillation (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 75.5 -> 75.8 | Helped |
| add fine-grained manifold distillation (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 75.8 -> 76.5 | Helped |
| add L_intra to manifold loss (alone) (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 75.8 -> 76.0 | Helped |
| add L_inter alone (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 75.8 -> 69.6 | Hurt |
| add L_random alone (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 75.8 -> 76.2 | Helped |
| add L_random to intra+inter (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 76.0 -> 76.5 | Helped |
| add L_inter to intra+random (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 76.4 -> 76.5 | Helped |
| select Shallow/Deep layers instead of only shallow or only deep (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 75.8/75.5 -> 76.3 | Helped |
| reduce random sampling K from 192 to 96 (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 76.5 -> 76.0 | Hurt |
| increase K from 192 to 384 (ablation) | DeiT-Tiny student, CaiT-S24 teacher · ImageNet-1k val | top-1 accuracy (%): 76.5 -> 76.3 | Hurt |
| manifold distillation vs hard distillation on transfer | DeiT-Tiny student, CaiT-S24 teacher · CIFAR-100 transfer | top-1 accuracy (%): 87.34 -> 88.05 | Helped |
Closest papers by meaning
- DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers (CVPR 2022 · similarity 0.90)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.88)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.86)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.86)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.86)
- KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.