Research map / Architectures and efficiency / Neural architecture search
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, Quoc Le
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add zero-init of last BN gamma in residual blocks (allows full learning rate) (ablation) | single-stage model · ImageNet val | top-1 accuracy: ~1.0% top-1 drop when lr reduced to 30% instead | Improved |
| add exponentially decaying LR with constant ending (ablation) | single-stage model · ImageNet val | top-1 accuracy | Improved |
| vary minimum LR of constant ending among 3%, 5%, 8%, 10% (ablation) | EfficientNet weight-shared model · ImageNet val | average top-1: 78.7, 78.8, 78.8, 78.8 | No clear effect |
| regularize only the biggest child model (weight decay, dropout) (ablation) | single-stage model, small child · ImageNet val | top-1 accuracy: +0.5 | Improved |
| regularize only the biggest child model (weight decay, dropout) (ablation) | single-stage model, big child · ImageNet val | top-1 accuracy: +0.2 | Improved |
| finetune child at lr 0.01 for 25 epochs (ablation) | BigNASModel-S/M/L/XL · ImageNet val | top-1 accuracy: 76.5 -> 74.6; 78.9 -> 76.7; 79.5 -> 77.9; 80.9 -> 79.0 | Hurt |
| finetune child at lr 0.001 or 0.0001 (ablation) | BigNASModel-S/M/L/XL · ImageNet val | top-1 accuracy: 79.5 -> 79.7 (L, lr 0.0001); 80.9 -> 80.8 (XL) | No clear effect |
| train child architecture jointly in the weight-shared single-stage model instead of from scratch without distillation (ablation) | BigNASModel-S · ImageNet val | top-1 accuracy: 75.3 -> 76.5 | Improved |
| train child architecture jointly in the weight-shared single-stage model instead of from scratch without distillation (ablation) | BigNASModel-XL · ImageNet val | top-1 accuracy: 79.3 -> 80.9 | Improved |
| add simple distillation (A) when training child from scratch (ablation) | EfficientNet-based child architectures · ImageNet val | top-1 accuracy: 75.3 -> 75.3 (S); 77.4 -> 77.4 (M); 78.2 -> 77.9 (L); 79.3 -> 79.0 (XL) | No clear effect |
| use inplace distillation (B) instead of no distillation when training child from scratch (ablation) | EfficientNet-based child architectures · ImageNet val | top-1 accuracy: 75.3 -> 76.3 (S); 77.4 -> 78.6 (M); 78.2 -> 79.2 (L); 79.3 -> 80.4 (XL) | Improved |
| single-stage BigNASModel-S vs MobileNetV3 at similar FLOPs | BigNASModel-S · ImageNet val | top-1 accuracy: 1.3% better at 240 MFLOPs | Improved |
Closest papers by meaning
- AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling (CVPR 2021 · similarity 0.92)
- Distribution Consistent Neural Architecture Search (CVPR 2022 · similarity 0.92)
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (ICLR 2021 · similarity 0.92)
- NAS-Bench-101: Towards Reproducible Neural Architecture Search (ICML 2019 · similarity 0.92)
- Few-Shot Neural Architecture Search (ICML 2021 · similarity 0.91)
- Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
