Research map / Architectures and efficiency / Neural architecture search
AlphaNet: Improved Training of Supernets with Alpha-Divergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace KL-KD with adaptive positive/negative alpha-divergence KD | Weight-sharing NAS supernet · ImageNet, AlphaNet-A4 | top-1 accuracy: 79.3 -> 80.0 | Helped |
| Replace KL-KD with adaptive positive/negative alpha-divergence KD | Slimmable MobileNetV1 · ImageNet, evaluated channel widths | top-1 accuracy: Adaptive-KD further improves on KL based KD for all the channel width configurations evaluated | Helped |
| Replace KL-KD with adaptive positive/negative alpha-divergence KD | Slimmable MobileNetV2 · ImageNet, evaluated channel widths | top-1 accuracy: Adaptive-KD further improves on KL based KD for all the channel width configurations evaluated | Helped |
| Remove knowledge distillation (ablation) | Slimmable MobileNetV1 · ImageNet, evaluated channel widths | top-1 accuracy: MbV1 0.25×: 53.9 without KD; 56.4 with KL-KD | Hurt |
| Remove knowledge distillation (ablation) | Slimmable MobileNetV2 · ImageNet, evaluated channel widths | top-1 accuracy: MbV2 0.35×: 61.9 without KD; 63.2 with KL-KD | Hurt |
| Use symmetric KL, combining KL(p||q) and KL(q||p) (ablation) | Weight-sharing NAS supernet · ImageNet, AlphaNet model family | top-1 accuracy: A4: 79.3 with KL-KD -> 79.3 with Symmetric KL-KD | No clear effect |
| Add attentive sampling to KL-KD (ablation) | Weight-sharing NAS supernet · ImageNet, AlphaNet-A4 | top-1 accuracy: 79.3 -> 79.8 | Helped |
| Use KL-KD temperature T=0.5 instead of T=1 (ablation) | Slimmable MobileNetV1 · ImageNet, MbV1 widths | top-1 accuracy: MbV1 0.25×: 56.4 (T=1) -> 55.1 (T=0.5) | Hurt |
| Use KL-KD temperature T=2 instead of T=1 (ablation) | Slimmable MobileNetV1 · ImageNet, MbV1 widths | top-1 accuracy: MbV1 0.25×: 56.4 (T=1) -> 55.4 (T=2.0) | Hurt |
| Use KL-KD temperature T=4 instead of T=1 (ablation) | Slimmable MobileNetV1 · ImageNet, MbV1 widths | top-1 accuracy: MbV1 0.25×: 56.4 (T=1) -> 48.7 (T=4.0) | Hurt |
| Set alpha-minus to -2 rather than -1 (ablation) | Adaptive-KD supernet · ImageNet, weight-sharing NAS | top-1 accuracy | Hurt |
| Set alpha-minus to 0 rather than -1 (ablation) | Adaptive-KD supernet · ImageNet, weight-sharing NAS | top-1 accuracy | Hurt |
Closest papers by meaning
- Distribution Consistent Neural Architecture Search (CVPR 2022 · similarity 0.91)
- Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search (CVPR 2021 · similarity 0.91)
- Subnet-Aware Dynamic Supernet Training for Neural Architecture Search (CVPR 2025 · similarity 0.90)
- Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.90)
- Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach (CVPR 2024 · similarity 0.89)
- HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.