Research map / Label-efficient and robust learning / Knowledge distillation
UNIC: Universal Classification Models via Multi-teacher Distillation
Yannis Kalantidis, Diane Larlus, Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas LUCAS
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add feature standardization of teacher outputs (ablation) | multi-teacher distillation (DINO and DeiT-III) · ImageNet-1K val | top-1 accuracy: 78.7 -> 81.4 | Improved |
| add dedicated projector heads for CLS and patch tokens (ablation) | multi-teacher distillation · ADE-20K | mIoU: 36.1 -> 36.9 | Improved |
| add ladder of projectors (ablation) | UNIC (DINO and DeiT-III) · ImageNet-1K val | top-1 accuracy: 82.2 -> 82.7 | Improved |
| add ladder of projectors (ablation) | UNIC (DINO and DeiT-III) · ADE-20K | mIoU: 36.9 -> 37.4 | Improved |
| add teacher dropping on top of ladder of projectors (ablation) | UNIC (DINO and DeiT-III) · ImageNet-1K val | top-1 accuracy: 82.7 -> 83.2 | Improved |
| add teacher dropping on top of ladder of projectors (ablation) | UNIC (DINO and DeiT-III) · Transfer (15 datasets avg) | top-1 accuracy: 74.2 -> 73.5 | Hurt |
| replace teacher dropping with Adaloss loss balancing (ablation) | UNIC base setup · ImageNet-1K val | top-1 accuracy: 80.1 vs 83.2 | Hurt |
| multi-teacher distillation versus distilling only from DeiT-III (ablation) | student ViT-B · ImageNet-1K val | top-1 accuracy: 83.1 -> 82.2 | Hurt |
| distill from four teachers instead of two (ablation) | UNIC · Transfer (15 datasets avg) | top-1 accuracy: 74.5 -> 75.1 | Improved |
| replace ImageNet-1K with synthetic ImageNet-SD for distillation (ablation) | UNIC · ImageNet-1K val | top-1 accuracy: 83.8 -> 81.7 | Hurt |
| increase top-only projector hidden layers from 1 to 3 (ablation) | UNIC base setup · ImageNet-1K val | top-1 accuracy: 82.2 -> 81.1 | Hurt |
| shrink student from ViT-Base to ViT-Small (ablation) | UNIC · ADE-20K | mIoU: 39.6 -> 36.1 | Hurt |
Closest papers by meaning
- Unifying Heterogeneous Classifiers With Distillation (CVPR 2019 · similarity 0.84)
- Multiclass Learnability Beyond the PAC Framework: Universal Rates and Partial Concept Classes (NeurIPS 2022 · similarity 0.81)
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels (ICLR 2025 · similarity 0.81)
- Learning Task-Agnostic Representations through Multi-Teacher Distillation (NeurIPS 2025 · similarity 0.80)
- UDON: Universal Dynamic Online distillatioN for generic image representations (NeurIPS 2024 · similarity 0.80)
- Towards Universal Dataset Distillation via Task-Driven Diffusion (CVPR 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
