Research map / Architectures and efficiency / Neural architecture search
Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a few-shot NAS method that splits the search space by the number of nonlinear functions (ReLUs) in each subnet, uses half-channel supernets and supernet-balanced sampling to train multiple supernets efficiently.
Problem
One-shot NAS weight sharing causes conflicts between subnets, and existing few-shot NAS methods that split the search space are computationally expensive due to gradient-based splitting and full-width supernets.
Approach
Split the search space into subspaces by counting the number of nonlinear functions (ReLUs) per subnet, which needs no forward/backward passes; reduce channel dimensions per supernet (G=2) so multiple supernets train on one machine; and sample one subnet from each supernet per step (supernet-balanced sampling).
Key results
- On NAS201 CIFAR10, the nonlinear-function splitting achieves Kendall's tau 0.735 with 3 supernets and 1.3M params, versus 0.653 for FS-NAS (5 supernets, 8.4M) and 0.656 for GM-NAS.
- On ImageNet with the MobileNet space, the method reaches 76.9% top-1 at 544M FLOPs (<=600M constraint) and 76.7% at 516M FLOPs, versus 76.6% for GM-NAS at 530M.
- With K=6 and G=2 on ImageNet, splitting cost is negligible and training takes 23.3 GPU days versus 81.0 for GM-NAS, which trains supernets sequentially.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use few-shot supernets with nonlinear-function-count splitting rather than GM-NAS | NAS201 supernet ranking · CIFAR10 test | Kendall's tau: 0.656 -> 0.735 | Helped |
| Split by number of nonlinear functions rather than FLOPs (ablation) | Ours · CIFAR10 test | Kendall's tau: 0.711 -> 0.735 | Helped |
| Split by number of nonlinear functions rather than number of linear regions (ablation) | Ours · CIFAR10 test | Kendall's tau: 0.712 -> 0.735 | Helped |
| Split by number of nonlinear functions rather than feature isotropy (ablation) | Ours · CIFAR10 test | Kendall's tau: 0.693 -> 0.735 | Helped |
| Use Ours rather than K-shot NAS | NAS201 architecture search · CIFAR10 test | accuracy: 94.19 -> 94.30 | Helped |
| Use Ours rather than K-shot NAS | NAS201 architecture search · CIFAR100 test | accuracy: 73.45 -> 73.20 | Hurt |
| Use Ours rather than K-shot NAS | NAS201 architecture search · ImageNet-16-120 test | accuracy: 46.53 -> 46.60 | Helped |
| Use Ours with a ≤530M FLOPs constraint rather than GM-NAS | MobileNet search · ImageNet | top-1 accuracy: 76.6 -> 76.7 | Helped |
| Increase supernets from 6 to 8 (ablation) | Ours (G=2) · ImageNet | top-1 accuracy: 76.7 -> 76.8 | Helped |
| Increase supernets from 6 to 8 (ablation) | Ours (G=2) · ImageNet | training cost: 23.3 -> 34.1 | Hurt |
Surprising findings
- Reducing channel dimensions by half (G=2) preserves subnet performance ranking better for multi-supernet splitting than for a single supernet (SPOS), which suffers ranking inconsistency.
Limitations
- Splitting relies on counting nonlinear functions, which is a coarse criterion; the paper notes it does not directly measure subnet performance.
- Evaluated only on NAS201 and the MobileNet search space for image classification.
- Ablation of the number of supernets and channel reduction is limited to specific K and G settings.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet ImageNet-16-120
Benchmarks: NAS201 MobileNet search space ImageNet CIFAR10 CIFAR100 ImageNet-16-120
Metrics: Kendall's tau top-1 accuracy top-5 accuracy FLOPs parameters GPU days
Baselines: SPOS AngleNet FS-NAS GM-NAS K-shot NAS AGNAS DARTS PC-DARTS GAEA ProxylessNAS Shapley-NAS DrNAS
Builds on: FS-NAS GM-NAS SPOS K-shot NAS FairNAS
Closest papers by meaning
- Few-Shot Neural Architecture Search (ICML 2021 · similarity 0.95)
- HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning (AAAI 2025 · similarity 0.94)
- Subnet-Aware Dynamic Supernet Training for Neural Architecture Search (CVPR 2025 · similarity 0.93)
- Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective (ICML 2025 · similarity 0.92)
- Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach (CVPR 2024 · similarity 0.91)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.