Research map / Vision-language and multimodal / Human and embodied modeling
Let All Be Whitened: Multi-Teacher Distillation for Efficient Visual Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Multi-teacher distillation (Whiten-MTD) that PCA-whitens each off-the-shelf teacher's embeddings and fuses their similarity matrices to train a single lightweight student for landmark image retrieval and near-duplicate video retrieval.
Problem
Visual retrieval requires balancing accuracy and efficiency; off-the-shelf strong teachers have heavy models and their similarity scores are on incompatible scales, making multi-teacher knowledge transfer biased toward some teachers.
Approach
Apply PCA-whitening and l2-normalization to each teacher's features so their cosine similarity distributions align on the unit sphere, then fuse teacher similarity matrices (max-min strategy) as distillation targets for a single lightweight student.
Key results
- R34-Whiten-MTD achieves 83.05 mAP on ROxford Medium and 92.33 on RParis Medium, outperforming asymmetric retrieval baselines such as EfficientNetB3-R101-DELG-Context (77.44 and 87.94).
- Temperature grid search shows default tau_t = tau_s = 0.05 gives overall best results (74.67 ROxf-M, 84.48 RPar-M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill with R34-Whiten-MTD instead of EfficientNetB3-R101-DELG-Context | visual retrieval · ROxf-M | mAP: 77.44 -> 83.05 | Helped |
| distill with R34-Whiten-MTD instead of EfficientNetB3-R101-DELG-Context | visual retrieval · ROxf-H | mAP: 58.97 -> 64.16 | Helped |
| distill with R34-Whiten-MTD instead of EfficientNetB3-R101-DELG-Context | visual retrieval · RPar-M | mAP: 87.94 -> 92.33 | Helped |
| distill with R34-Whiten-MTD instead of EfficientNetB3-R101-DELG-Context | visual retrieval · RPar-H | mAP: 75.68 -> 83.20 | Helped |
| distill with R34-Whiten-MTD instead of EfficientNetB3-R101-DELG-Context | visual retrieval · ROxf+1M-M | mAP: 63.21 -> 76.88 | Helped |
| distill with R34-Whiten-MTD instead of EfficientNetB3-R101-DELG-Context | visual retrieval · ROxf+1M-H | mAP: 38.20 -> 52.65 | Helped |
| distill with R34-Whiten-MTD instead of EfficientNetB3-R101-DELG-Context | visual retrieval · RPar+1M-M | mAP: 73.37 -> 80.57 | Helped |
| distill with R34-Whiten-MTD instead of EfficientNetB3-R101-DELG-Context | visual retrieval · RPar+1M-H | mAP: 50.09 -> 63.26 | Helped |
| set teacher and student temperatures to 0.01 instead of 0.05 (ablation) | Whiten-MTD · ROxf-M | mAP: 74.67 -> 72.55 | Hurt |
| set teacher and student temperatures to 0.1 instead of 0.05 (ablation) | Whiten-MTD · ROxf-M | mAP: 74.67 -> 72.79 | Hurt |
| set teacher temperature to 0.01 and student temperature to 0.05 (ablation) | Whiten-MTD · ROxf-M | mAP: 74.67 -> 72.67 | Hurt |
| set teacher temperature to 0.01 and student temperature to 0.1 (ablation) | Whiten-MTD · ROxf-M | mAP: 74.67 -> 71.57 | Hurt |
Surprising findings
- R101-DOLG has only 413 significant components out of 1024 dimensions, so its whitened representation lies on a lower-dimensional sphere, yet distillation still works well.
Limitations
- Teachers with rank-deficient representations (e.g., R101-DOLG) break the assumption that whitened features fill the full n_c-dimensional sphere.
- Evaluation restricted to global features without query expansion or re-ranking.
Experimental setup
Datasets: GLDv2-clean Short Video Dataset (SVD) ImageNet
Benchmarks: ROxford RParis SVD
Metrics: mAP MRR
Baselines: R101-GeM R101-AP-GeM R101-SOLAR R101-DELG R101-DOLG MobileNetV2-R101-Reg EfficientNetB3-R101-DELG-Context MobileNetV2-R101-HVS
Builds on: DOLG DELG GeM pooling MoCo v3 Barlow Twins
Closest papers by meaning
- MUSE: Mamba Is Efficient Multi-scale Learner for Text-video Retrieval (AAAI 2025 · similarity 0.85)
- Visual Abstraction: A Plug-and-Play Approach for Text-Visual Retrieval (ICML 2025 · similarity 0.85)
- On Train-Test Class Overlap and Detection for Image Retrieval (CVPR 2024 · similarity 0.84)
- Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval (ICLR 2025 · similarity 0.84)
- Instance-Level Composed Image Retrieval (NeurIPS 2025 · similarity 0.84)
- Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.