Research map / Architectures and efficiency / Neural architecture search
UniADS: Universal Architecture-Distiller Search for Distillation Gap
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniADS, a joint search over student architecture (depth/width) and distiller settings (transforms, distance functions, loss weights, temperature) using NSGA-II with Successive Halving, evaluated on CIFAR-100 and ImageNet distillation.
Problem
Teacher-student distillation gap limits KD gains; prior distillation-aware architecture search optimizes only the student architecture and ignores distiller settings such as transforms, distance functions, and loss weights.
Approach
Co-optimize student depth/width and the full distiller configuration (transform, distance, loss weight, temperature) in one unified search space, using NSGA-II for multi-objective evolutionary search (accuracy vs. parameters) and Successive Halving with loss-rejection to prune the space and cut search cost.
Key results
- ResNet18 distilled with UniADS reaches 72.51% top-1 on ImageNet with ResNet50 teacher, a 1.83% gain over the baseline KD-style result reported in Table 4 (69.75% student).
- On CIFAR-100, UniADS improves ResNet-based students by 1.69% to 2.71% over DisWOT-style architecture search and random search (Table 2 summary).
- The paper reports a 40-fold search acceleration from Successive Halving, and NSGA-II converges faster than random search in its ResNet110 comparison.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Jointly search student architecture and distiller settings instead of searching architecture alone | UniADS · CIFAR-100 | Top-1 accuracy (%): 72.45 -> 74.89; 73.94 -> 75.33; 74.86 -> 76.25 | Helped |
| Jointly search student architecture and distiller settings | UniADS · ImageNet-1k; ResNet50 → ResNet18 | Top-1 accuracy (%): 70.68 -> 72.51 | Helped |
| Use NSGA-II genetic search instead of random search (ablation) | UniADS distiller search · CIFAR-100; ResNet110 teacher | search performance | Helped |
| Add Successive Halving to NSGA-II search (ablation) | UniADS distiller search · CIFAR-100; ResNet110 teacher | search efficiency | Helped |
| Add Successive Halving acceleration (ablation) | UniADS distiller search · Distiller search | search speedup: 40-fold acceleration | Helped |
| Increase student network width and depth (ablation) | UniADS · CIFAR-100 | Top-1 accuracy (%) | Helped |
| Add transformation and feature-loss weights to the search space (ablation) | UniADS · CIFAR-100 | Top-1 accuracy (%) | Helped |
| Add additional optimization parameters such as batch size and learning rate to the search (ablation) | UniADS · CIFAR-100 | Top-1 accuracy (%) | No clear effect |
Surprising findings
- Hyperparameter importance analysis finds that some additional optimization parameters such as batch size and learning rate have minimal influence on distillation results.
Limitations
- Search cost and hardware requirements are not clearly reported in the main text.
- Gains over strong baselines on some teacher-student pairs are modest (e.g., ~1.1-1.5% over SOTA KD methods).
Experimental setup
Datasets: CIFAR-100 ImageNet-1k
Benchmarks: CIFAR-100 ImageNet-1k
Metrics: top-1 accuracy top-5 accuracy
Baselines: KD FitNet AT SP CC RKD CRD DIST SRRL SemCKD DisWOT ATKD
Builds on: NSGA-II Successive Halving DisWOT Hinton knowledge distillation CRD
Closest papers by meaning
- Distillation Scaling Laws (ICML 2025 · similarity 0.87)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.84)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.83)
- DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs (ICML 2025 · similarity 0.83)
- Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.83)
- Adaptive Dual Guidance Knowledge Distillation (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.