Research map / Label-efficient and robust learning / Knowledge distillation
Towards Oracle Knowledge Distillation with Neural Architecture Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OD rather than KD for both architecture search and training (ablation) | KDAS with ResNet-32 backbone · CIFAR-100 | accuracy: 76.92 ± 0.33 -> 77.27 ± 0.11 | Helped |
| Use OD rather than KD for both architecture search and training (ablation) | KDAS with ResNet-32 backbone · TinyImageNet | accuracy: 62.17 ± 0.12 -> 63.04 ± 0.17 | Helped |
| Use OD rather than KD for training after OD-based search (ablation) | KDAS with ResNet-32 backbone · CIFAR-100 | accuracy: 76.92 ± 0.33 -> 77.27 ± 0.11 | Helped |
| Use OD rather than KD for training after OD-based search (ablation) | KDAS with ResNet-32 backbone · TinyImageNet | accuracy: 62.17 ± 0.12 -> 63.04 ± 0.17 | Helped |
| Search with OD rather than KD, keeping OD training (ablation) | KDAS with ResNet-32 backbone · CIFAR-100 | accuracy: 77.04 ± 0.33 -> 77.27 ± 0.11 | No clear effect |
| Search with OD rather than KD, keeping OD training (ablation) | KDAS with ResNet-32 backbone · TinyImageNet | accuracy: 62.73 ± 0.09 -> 63.04 ± 0.17 | Helped |
| Train with OD instead of CE after KD-based search (ablation) | KDAS with ResNet-32 backbone · TinyImageNet | accuracy: 62.92 ± 0.10 -> 62.34 ± 0.10 | Hurt |
| Use OD rather than KD for the student loss (ablation) | ResNet-32 student · CIFAR-100 | accuracy: 74.67 ± 0.10 -> 74.77 ± 0.02 | No clear effect |
| Use OD rather than KD for the student loss (ablation) | ResNet-32 student · TinyImageNet | accuracy: 58.68 ± 0.09 -> 58.66 ± 0.25 | No clear effect |
| Increase student capacity through searched architecture rather than manual architecture design | KDAS versus ResNet-62/ResNet-110 · CIFAR-100 | accuracy: 79.01 ± 0.28 vs 77.39 ± 0.21 | Helped |
| Search the student architecture with KDAS rather than use a manually designed larger student | ResNet-32 student · CIFAR-100 | accuracy: 75.82 ± 0.32 vs 75.18 ± 0.13 | Helped |
| Train the student to imitate only the ensemble members that predict correctly for each example (ablation) | Ensemble-teacher distillation · TinyImageNet | accuracy: 62.17 ± 0.12 -> 63.04 ± 0.17 | Helped |
Closest papers by meaning
- Search to Distill: Pearls Are Everywhere but Not the Eyes (CVPR 2020 · similarity 0.93)
- Teacher Guided Neural Architecture Search for Face Recognition (AAAI 2021 · similarity 0.91)
- DisWOT: Student Architecture Search for Distillation WithOut Training (CVPR 2023 · similarity 0.91)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.88)
- Understanding the Role of the Projector in Knowledge Distillation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.