AI Paper Atlas

Research map / Architectures and efficiency / Neural architecture search

BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models

Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, Quoc Le

ECCV 2020 · Architectures and efficiency › Neural architecture search · approach: Efficient NAS search strategies · 256 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add zero-init of last BN gamma in residual blocks (allows full learning rate) (ablation)single-stage model · ImageNet valtop-1 accuracy: ~1.0% top-1 drop when lr reduced to 30% insteadImproved
add exponentially decaying LR with constant ending (ablation)single-stage model · ImageNet valtop-1 accuracyImproved
vary minimum LR of constant ending among 3%, 5%, 8%, 10% (ablation)EfficientNet weight-shared model · ImageNet valaverage top-1: 78.7, 78.8, 78.8, 78.8No clear effect
regularize only the biggest child model (weight decay, dropout) (ablation)single-stage model, small child · ImageNet valtop-1 accuracy: +0.5Improved
regularize only the biggest child model (weight decay, dropout) (ablation)single-stage model, big child · ImageNet valtop-1 accuracy: +0.2Improved
finetune child at lr 0.01 for 25 epochs (ablation)BigNASModel-S/M/L/XL · ImageNet valtop-1 accuracy: 76.5 -> 74.6; 78.9 -> 76.7; 79.5 -> 77.9; 80.9 -> 79.0Hurt
finetune child at lr 0.001 or 0.0001 (ablation)BigNASModel-S/M/L/XL · ImageNet valtop-1 accuracy: 79.5 -> 79.7 (L, lr 0.0001); 80.9 -> 80.8 (XL)No clear effect
train child architecture jointly in the weight-shared single-stage model instead of from scratch without distillation (ablation)BigNASModel-S · ImageNet valtop-1 accuracy: 75.3 -> 76.5Improved
train child architecture jointly in the weight-shared single-stage model instead of from scratch without distillation (ablation)BigNASModel-XL · ImageNet valtop-1 accuracy: 79.3 -> 80.9Improved
add simple distillation (A) when training child from scratch (ablation)EfficientNet-based child architectures · ImageNet valtop-1 accuracy: 75.3 -> 75.3 (S); 77.4 -> 77.4 (M); 78.2 -> 77.9 (L); 79.3 -> 79.0 (XL)No clear effect
use inplace distillation (B) instead of no distillation when training child from scratch (ablation)EfficientNet-based child architectures · ImageNet valtop-1 accuracy: 75.3 -> 76.3 (S); 77.4 -> 78.6 (M); 78.2 -> 79.2 (L); 79.3 -> 80.4 (XL)Improved
single-stage BigNASModel-S vs MobileNetV3 at similar FLOPsBigNASModel-S · ImageNet valtop-1 accuracy: 1.3% better at 240 MFLOPsImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy