AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

UNIC: Universal Classification Models via Multi-teacher Distillation

Yannis Kalantidis, Diane Larlus, Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas LUCAS

ECCV 2024 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 23 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add feature standardization of teacher outputs (ablation)multi-teacher distillation (DINO and DeiT-III) · ImageNet-1K valtop-1 accuracy: 78.7 -> 81.4Improved
add dedicated projector heads for CLS and patch tokens (ablation)multi-teacher distillation · ADE-20KmIoU: 36.1 -> 36.9Improved
add ladder of projectors (ablation)UNIC (DINO and DeiT-III) · ImageNet-1K valtop-1 accuracy: 82.2 -> 82.7Improved
add ladder of projectors (ablation)UNIC (DINO and DeiT-III) · ADE-20KmIoU: 36.9 -> 37.4Improved
add teacher dropping on top of ladder of projectors (ablation)UNIC (DINO and DeiT-III) · ImageNet-1K valtop-1 accuracy: 82.7 -> 83.2Improved
add teacher dropping on top of ladder of projectors (ablation)UNIC (DINO and DeiT-III) · Transfer (15 datasets avg)top-1 accuracy: 74.2 -> 73.5Hurt
replace teacher dropping with Adaloss loss balancing (ablation)UNIC base setup · ImageNet-1K valtop-1 accuracy: 80.1 vs 83.2Hurt
multi-teacher distillation versus distilling only from DeiT-III (ablation)student ViT-B · ImageNet-1K valtop-1 accuracy: 83.1 -> 82.2Hurt
distill from four teachers instead of two (ablation)UNIC · Transfer (15 datasets avg)top-1 accuracy: 74.5 -> 75.1Improved
replace ImageNet-1K with synthetic ImageNet-SD for distillation (ablation)UNIC · ImageNet-1K valtop-1 accuracy: 83.8 -> 81.7Hurt
increase top-only projector hidden layers from 1 to 3 (ablation)UNIC base setup · ImageNet-1K valtop-1 accuracy: 82.2 -> 81.1Hurt
shrink student from ViT-Base to ViT-Small (ablation)UNIC · ADE-20KmIoU: 39.6 -> 36.1Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy