Research map / Architectures and efficiency / Attention and transformers
DeiT-LT: Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeiT-LT, a distillation scheme that trains ViT-B from scratch on long-tailed datasets using OOD-augmented images, DRW-reweighted DIST-token distillation, and flat SAM-trained CNN teachers.
Problem
Vision Transformers trained from scratch perform poorly on long-tailed datasets, especially on tail classes, and prior long-tailed methods mostly target CNNs or require large-scale pre-training.
Approach
Distill from small CNN teachers (ResNet-32/50 trained with LDAM-DRW-SAM or PaCo+SAM) using strongly augmented, mixup OOD images so the DIST token becomes a tail-class expert (DRW-weighted loss) while the CLS token specializes on head classes; SAM-flat teachers induce low-rank, generalizable features.
Key results
- On CIFAR-10 LT (rho=100), DeiT-LT with PaCo+SAM teacher reaches 87.5% vs 70.2% for DeiT and 62.6% for ViT.
- On ImageNet-LT, DeiT-LT (PaCo+SAM teacher) reaches 59.1% overall, outperforming the PaCo+SAM teacher's 57.5%.
- On iNaturalist-2018, DeiT-LT (PaCo+SAM teacher) reaches 75.1% overall, vs 73.4% for the PaCo+SAM CNN teacher and 61.0% for DeiT-III.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OOD distillation (ablation) | DeiT-LT · CIFAR-10 LT and CIFAR-100 LT | accuracy: 70.2 -> 84.5; 31.3 -> 48.9 | Helped |
| Add DRW to OOD distillation (ablation) | DeiT-LT · CIFAR-10 LT and CIFAR-100 LT | accuracy: 84.5 -> 87.3; 48.9 -> 54.5 | Helped |
| Use a SAM-trained teacher with OOD distillation and DRW (ablation) | DeiT-LT · CIFAR-10 LT and CIFAR-100 LT | accuracy: 87.3 -> 87.5; 54.5 -> 55.6 | Helped |
| Distill from a PaCo+SAM CNN teacher rather than use the teacher alone | DeiT-LT · CIFAR-10 LT (ρ=100), CIFAR-10 LT (ρ=50), CIFAR-100 LT (ρ=100), CIFAR-100 LT (ρ=50) | accuracy: 86.8 -> 87.5; 88.6 -> 89.8; 52.8 -> 55.6; 56.6 -> 60.5 | Helped |
| Distill from a PaCo+SAM teacher rather than use the teacher alone | DeiT-LT · ImageNet-LT | overall accuracy: 57.5 -> 59.1 | Helped |
| Distill from a PaCo+SAM teacher rather than use the teacher alone | DeiT-LT · iNaturalist-2018 | overall accuracy: 73.4 -> 75.1 | Helped |
| Increase transformer capacity from ViT-Tiny to ViT-Small to ViT-Base (ablation) | DeiT-LT · CIFAR-10 LT (ρ=100) | overall accuracy: 80.8 -> 85.5 -> 87.5 | Helped |
| Increase transformer capacity from ViT-Tiny to ViT-Small to ViT-Base (ablation) | DeiT-LT · CIFAR-100 LT (ρ=100) | overall accuracy: 49.3 -> 54.3 -> 55.6 | Helped |
Surprising findings
- Strongly augmented images on which the teacher has low accuracy (OOD) yield better distillation than in-distribution images, and the student outperforms its CNN teacher.
- Distilling a CNN teacher's final logits transfers low-rank feature structure to ViT blocks, and the CLS and DIST tokens are intentionally made to diverge rather than match as in DeiT.
Limitations
- Tail-class learning relies mostly on distillation, so tail performance is similar to the CNN teacher's.
- Training ViTs with LDAM and other long-tailed losses from scratch led to optimization difficulties.
- Does not compare against pre-training-based methods, which are excluded by design.
Experimental setup
Datasets: CIFAR-10 LT CIFAR-100 LT ImageNet-LT iNaturalist-2018
Benchmarks: CIFAR-10 LT CIFAR-100 LT ImageNet-LT iNaturalist-2018
Metrics: top-1 accuracy overall head/mid/tail accuracy
Baselines: ViT DeiT DeiT-III ViT (cRT) LDAM+DRW PaCo+SAM RIDE MiSLAS BCL ImbSAM
Builds on: DeiT LDAM-DRW Sharpness-Aware Minimization PaCo DeiT-III
Closest papers by meaning
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.87)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.87)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.86)
- Bi-ViT: Pushing the Limit of Vision Transformer Quantization (AAAI 2024 · similarity 0.86)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.86)
- ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.