Research map / Label-efficient and robust learning / Knowledge distillation
UDON: Universal Dynamic Online distillatioN for generic image representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UDON, a shared-backbone online multi-teacher distillation method with loss-based dynamic domain sampling for universal fine-grained image embeddings, reaching state of the art on UnED.
Problem
Learning a single compact (64-D) universal image embedding that handles fine-grained and instance-level recognition across many visual domains with conflicting cues and very different class/sample distributions.
Approach
Jointly train a shared ViT backbone with a universal student head and per-domain teacher heads, distilling relational and logit knowledge from domain teachers to the student online, and sample domains per batch proportionally to their current training classification losses.
Key results
- With ImageNet21k pretraining, UDON reaches mean R@1 65.3 and P@5 53.9 on UnED, vs. 62.5 R@1 for USCRR (+2.3% R@1 over previous SOTA UJCRR at 62.6 R@1 reported in table).
- With CLIP pretraining, UDON reaches mean R@1 67.7 and P@5 56.9, improving over the previous best (USCSS 65.9 P@5 / UJCRR 64.6 R@1) by about 1.9% P@5 and 1.8% R@1.
- Sharing the backbone between teachers and student outperforms 8 separate teachers (R@1 64.8 vs 64.2) while using ~188M vs ~873M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UDON instead of USCRR | ViT-Base/16 with ImageNet21k initialization · UnED test set (mean) | R@1: 62.5 -> 65.3 | Helped |
| Use UDON instead of USCRR | ViT-Base/16 with ImageNet21k initialization · UnED test set (mean) | P@5: 51.4 -> 53.9 | Helped |
| Use UDON instead of USCSS | ViT-Base/16 with CLIP initialization · UnED test set (mean) | R@1: 65.9 -> 67.7 | Helped |
| Add dynamic loss-based domain sampling (ablation) | UDON · Met and GLDv2; other five domains | R@1: 9.4 -> 15.9; 33.4 -> 40.9; 62.9 -> 62.2; 72.5 -> 71.2; 82.2 -> 80.2; 90.8 -> 88.5; 88.6 -> 88.0 | Mixed |
| Increase teacher embedding dimensionality from 64 to 256 (ablation) | UDON · UnED test set (mean) | R@1: 64.2 -> 65.3 | Helped |
| Add logit distillation (ablation) | UDON with relational distillation · UnED test set (mean) | R@1: 64.8 -> 65.3 | Helped |
| Add relational and logit distillation (ablation) | UDON without distillation · UnED test set (mean) | R@1: 63.8 -> 65.3 | Helped |
| Add classification loss on the universal embedding (ablation) | UDON · UnED test set (mean) | R@1: 64.0 -> 65.3 | Helped |
| Use teacher losses rather than universal-student losses to update sampler weights (ablation) | UDON dynamic sampler · UnED test set (mean) | R@1: 64.9 -> 65.3 | Helped |
| Share the backbone between student and teachers instead of using 8 separate teacher backbones | UDON · UnED test set (mean) | R@1: 64.2 -> 64.8 | Helped |
| Share the backbone between student and teacher instead of using one separate teacher | UDON · UnED test set (mean) | R@1: 63.6 -> 64.8 | Helped |
| Increase logit distillation temperature from 0.1 to 1.0 (ablation) | UDON · UnED test set (mean) | R@1: 65.3 -> 65.0 | Hurt |
Surprising findings
- Specialist teachers with a shared backbone are weaker individually than separate specialist teachers, yet the shared-backbone student distills better.
- Using the universal student's classifier loss instead of the teacher loss to drive the dynamic sampler slightly reduces performance.
Limitations
- 20% decrease in training throughput due to added teacher heads.
- Dynamic sampler improves hard domains (Met, GLDv2) at slight cost to simpler domains.
- Gap to Specialist+Oracle remains for some domains.
Experimental setup
Datasets: UnED Food2k CARS196 SOP InShop iNat Met GLDv2 Rp2k
Benchmarks: UnED
Metrics: Recall@1 modified mean Precision@5
Baselines: USCRR UJCRR USCSS USC w/ Dyn Sampler MLP baseline ImageNet21k off-the-shelf CLIP DINOv2 SigLIP 8 separate teachers 1 separate teacher
Builds on: Normalized Softmax Loss UnED universal embedding benchmark USCRR (Universal Specialist/Classifier baseline) Scenic
Closest papers by meaning
- Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data (NeurIPS 2021 · similarity 0.86)
- Towards Universal Dataset Distillation via Task-Driven Diffusion (CVPR 2025 · similarity 0.86)
- NORM: Knowledge Distillation via N-to-One Representation Matching (ICLR 2023 · similarity 0.85)
- Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval (CVPR 2023 · similarity 0.85)
- Dataset Distillation for Pre-Trained Self-Supervised Vision Models (NeurIPS 2025 · similarity 0.85)
- DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teachers (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.