Research map / Label-efficient and robust learning / Knowledge distillation
Learning Metrics From Teachers: Compact Networks for Image Embedding
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add relative-teacher distillation to student metric learning | ResNet-18 embedding student · CUB-200-2011 | Recall@1: 51.7 -> 58.0 | Helped |
| Add relative-teacher distillation to student metric learning | ResNet-18 embedding student · Cars-196 | Recall@1: 46.7 -> 76.6 | Helped |
| Add absolute-teacher distillation to student metric learning | ResNet-18 embedding student · Stanford Online Products | Recall@1: 61.7 -> 68.0 | Helped |
| Use relative distances rather than matching teacher embedding coordinates | Distillation for ResNet-18 embeddings · CUB-200-2011 and Cars-196 | Recall@1: 54.9 -> 58.0; 70.6 -> 76.6 | Helped |
| Add hint loss to relative-teacher distillation (ablation) | ResNet-18 embedding student · CUB-200-2011 and Cars-196 | Recall@1: 58.0 -> 57.4; 76.6 -> 76.4 | Hurt |
| Add attention transfer to relative-teacher distillation (ablation) | ResNet-18 embedding student · CUB-200-2011 and Cars-196 | Recall@1: 58.0 -> 58.1; 76.6 -> 76.4 | Mixed |
| Add hint loss to absolute-teacher distillation (ablation) | ResNet-18 embedding student · CUB-200-2011 and Cars-196 | Recall@1: 54.9 -> 55.0; 70.6 -> 71.3 | No clear effect |
| Add attention transfer to absolute-teacher distillation (ablation) | ResNet-18 embedding student · CUB-200-2011 and Cars-196 | Recall@1: 54.9 -> 55.0; 70.6 -> 71.1 | Helped |
| Add unlabeled examples for relative-teacher distillation alongside labeled examples | ResNet-18 embedding student with 50% labeled data · CUB-200-2011 | Recall@1: 56.0 -> 57.2 | Helped |
| Train with relative-teacher distillation on unlabeled data, without metric-learning labels | ResNet-18 embedding student · CUB-200-2011 | Recall@1: 51.0 -> 55.5 | Helped |
| Add relative-teacher distillation to a MobileNet-0.25 student | MobileNet-0.25 embedding student · CUB-200-2011 | Recall@1: 27.5 -> 44.6 | Helped |
| Use a high-quality teacher to guide a low-resolution student | ResNet-18 cross-quality embedding student · CUB-200-2011 low resolution | Recall@1: 44.4 -> 46.2 | Helped |
Closest papers by meaning
- Learning Student Networks in the Wild (CVPR 2021 · similarity 0.87)
- Learning Intra-Batch Connections for Deep Metric Learning (ICML 2021 · similarity 0.87)
- Improved Knowledge Distillation via Teacher Assistant (AAAI 2020 · similarity 0.87)
- Knowledge distillation via softmax regression representation learning (ICLR 2021 · similarity 0.86)
- Scaling Multi-Teacher Distillation for Digital Pathology (NeurIPS 2026 · similarity 0.86)
- How Low Can You Go? Searching for the Intrinsic Dimensionality of Complex Networks using Metric Node Embeddings (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.