Research map / Architectures and efficiency / Attention and transformers
DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DearKD instead of DeiT-Ti | DeiT-Ti · ImageNet | Top-1 accuracy: 72.2 -> 74.8 | Helped |
| use DearKD with 50% rather than 100% training data | DeiT-Ti · ImageNet | Top-1 accuracy: DeiT-Ti 100%: 72.2; DearKD-Ti 50%: 72.3 | Helped |
| add MHCA (ablation) | DeiT-Ti · ImageNet | Top-1 accuracy: 72.3 -> 72.5 | Helped |
| add logit distillation (ablation) | DeiT-Ti · ImageNet | Top-1 accuracy: 72.3 -> 74.3 | Helped |
| add hidden-layer distillation without MHCA (ablation) | DeiT-Ti with logit distillation · ImageNet | Top-1 accuracy: 74.3 -> 74.1 | Hurt |
| add MHCA alongside hidden-layer and logit distillation (ablation) | DeiT-Ti · ImageNet | Top-1 accuracy: 74.1 -> 74.6 | Helped |
| add the second training stage without distillation (ablation) | DeiT-Ti with MHCA and distillation · ImageNet | Top-1 accuracy: 74.6 -> 74.8 | Helped |
| use boundary-preserving intra-divergence instead of ADI diversity loss (ablation) | DF-DearKD · ImageNet, data-free | Top-1 accuracy: 70.1% -> 71.2% | Helped |
| add ADI diversity loss to DeepInversion (ablation) | data-free DeiT-Ti · ImageNet, data-free | Top-1 accuracy: 62.7% -> 70.1% | Helped |
| use boundary-preserving intra-divergence rather than no diversity loss (ablation) | data-free DeiT-Ti · ImageNet, data-free | Top-1 accuracy: 62.7% -> 71.2% | Helped |
| train DF-DearKD with generated rather than real images | DeiT-Ti · ImageNet | Top-1 accuracy: 74.6% -> 71.2% | Hurt |
| use boundary-preserving intra-divergence for sample generation (ablation) | DF-DearKD · Generated ImageNet samples | LPIPS: 0.683 -> 0.692 | Helped |
Closest papers by meaning
- Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation (NeurIPS 2022 · similarity 0.90)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.89)
- Supervised Masked Knowledge Distillation for Few-Shot Transformers (CVPR 2023 · similarity 0.89)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.89)
- KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling (CVPR 2024 · similarity 0.88)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.