Research map / Architectures and efficiency / Neural architecture search
Search to Distill: Pearls Are Everywhere but Not the Eyes
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| search architectures using a distillation-guided reward rather than a hard-label reward (ablation) | AKD student architectures · ImageNet val, 15±1 ms | top-1 accuracy: 59.73 -> 61.4 | Helped |
| train with distillation rather than hard labels (ablation) | NASNet · ImageNet val, 15±1 ms | top-1 accuracy: 59.73 -> 63.9 | Helped |
| train with distillation rather than hard labels (ablation) | AKDNet · ImageNet val, 15±1 ms | top-1 accuracy: 61.4 -> 66.5 | Helped |
| combine distillation-guided architecture search with distillation training (ablation) | student architecture · ImageNet val, 15±1 ms | top-1 accuracy: 59.73 -> 66.5 | Helped |
| combine distillation-guided architecture search with distillation training (ablation) | student architecture · ImageNet val, 25±1 ms | top-1 accuracy: 67.0 -> 69.6 | Helped |
| combine distillation-guided architecture search with distillation training (ablation) | student architecture · ImageNet val, 75±1 ms | top-1 accuracy: 73.0 -> 75.5 | Helped |
| search with a distillation-guided reward rather than hard-label reward (ablation) | AKD student architectures · ImageNet val, 300M FLOPS | top-1 accuracy: 69.92 -> 70.0 | Helped |
| use AKDNet-M rather than MobileNet-v2 with RCO-KD | ImageNet classification · ImageNet val, 32.8ms | top-1 accuracy: 68.2 -> 72.4 | Helped |
| transfer the ImageNet-searched AKDNet-M architecture to face retrieval | AKDNet-M with RCO-KD · MegaFace, 1e6 distractors | Top-1 accuracy: 85.9 -> 90.2 | Helped |
| distill from an ensemble teacher rather than train with hard labels (ablation) | AKDNet-M · MegaFace, 1e6 distractors | Top-1 accuracy: 68.6 -> 90.9 | Helped |
| use RCO-KD rather than original-KD with the ensemble teacher (ablation) | AKDNet-M · MegaFace, 1e6 distractors | Top-1 accuracy: 90.1 -> 90.9 | Helped |
Closest papers by meaning
- Towards Oracle Knowledge Distillation with Neural Architecture Search (AAAI 2020 · similarity 0.93)
- Teacher Guided Neural Architecture Search for Face Recognition (AAAI 2021 · similarity 0.92)
- DisWOT: Student Architecture Search for Distillation WithOut Training (CVPR 2023 · similarity 0.91)
- KD-Zero: Evolving Knowledge Distiller for Any Teacher-Student Pairs (NeurIPS 2023 · similarity 0.91)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.90)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.