Research map / Architectures and efficiency / Neural architecture search
SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SWAP-Score, a training-free NAS metric based on sample-wise activation patterns with a size-based regularisation, and integrates it into an evolutionary search (SWAP-NAS) that searches CIFAR-10 and ImageNet in minutes on one V100.
Problem
Training-free metrics (zero-cost proxies) used for NAS have unreliable correlation with ground-truth accuracy, generalise poorly across search spaces and tasks, and are biased toward larger models.
Approach
Count distinct binarised activation patterns per sample (sample-wise rather than neuron-wise), which gives a much higher upper bound than standard activation patterns and avoids saturation with high-dimensional inputs; multiply by a Gaussian-shaped size regulariser on parameter count to steer model size.
Key results
- Regularised SWAP-Score reaches Spearman 0.90 with CIFAR-100 validation accuracy on NAS-Bench-201, versus 0.80 for the second-best metric NWOT.
- SWAP-NAS finds a CIFAR-10 architecture with 2.48% test error (SWAP-NAS-C) in about 0.004 GPU days (6 minutes).
- SWAP-NAS on ImageNet reaches 24.0% top-1 error in about 0.006 GPU days (9 minutes), versus 25.5% for QE-NAS in 0.02 GPU days.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sample-wise rather than standard activation patterns at input size 32×32 (ablation) | SWAP-Score · NAS-Bench-201 CIFAR-10 | Spearman correlation: 0.34 -> 0.90 | Helped |
| Add model-size regularisation to SWAP-Score at input size 32×32 (ablation) | SWAP-Score · NAS-Bench-201 CIFAR-10 | Spearman correlation: 0.90 -> 0.93 | Helped |
| Increase input size from 3×3 to 32×32 when using standard activation patterns (ablation) | Standard activation patterns · NAS-Bench-201 CIFAR-10 | Spearman correlation: 0.86 -> 0.34 | Hurt |
| Increase input size from 3×3 to 32×32 when using SWAP-Score (ablation) | SWAP-Score · NAS-Bench-201 CIFAR-10 | Spearman correlation: 0.86 -> 0.90 | Helped |
| Increase input size from 3×3 to 32×32 when using regularised SWAP-Score (ablation) | Regularised SWAP-Score · NAS-Bench-201 CIFAR-10 | Spearman correlation: 0.89 -> 0.93 | Helped |
| Apply regularisation with μ=1.5, σ=1.5 | SWAP-Score · NAS-Bench-201 CIFAR-100 | Spearman correlation: 0.80 -> 0.90 | Helped |
| Use smaller regularisation parameters μ=0.9, σ=0.9 instead of μ=1.5, σ=1.5 (ablation) | SWAP-NAS on CIFAR-10 · CIFAR-10 | test error (%): 2.48±0.09 -> 2.65±0.04 | Hurt |
| Increase regularisation parameters from μ=0.9, σ=0.9 to μ=1.5, σ=1.5 (ablation) | SWAP-NAS on CIFAR-10 · CIFAR-10 | test error (%): 2.65±0.04 -> 2.48±0.09 | Helped |
| Adjust μ and σ to control model size (ablation) | SWAP-NAS · CIFAR-10 | parameters (M): 3.06 -> 3.48 -> 4.3 | Mixed |
| Use regularised SWAP-Score in cell-based NAS-Bench spaces (ablation) | SWAP-Score · NAS-Bench-101, NAS-Bench-201, NAS-Bench-301, TransNAS-Bench-101-Micro | Spearman correlation | Helped |
| Apply regularisation in the stack-based TransNAS-Bench-101-Macro space (ablation) | SWAP-Score · TransNAS-Bench-101-Macro | Spearman correlation | No clear effect |
| Reduce μ while keeping σ=30 (ablation) | Regularised SWAP-Score · NAS-Bench-101 | Spearman correlation: 0.76 -> 0.04 | Hurt |
Surprising findings
- Standard activation-pattern counts saturate at the sample count (32.0 ± 0.0 at 32×32 input), so they lose discriminative power with larger inputs.
- Regularisation with a poorly chosen sigma (e.g., mu=40, sigma=0.3) drops NAS-Bench-101 correlation below 0.05, showing sensitivity to the regulariser's shape.
Limitations
- Current formulation is restricted to ReLU (piecewise linear) activations; extension to GELU and other activations is future work.
- Regularisation parameters mu and sigma must be set per search space using the model size distribution of sampled architectures.
- Regularisation does not improve correlation in the stack-based TransNAS-Bench-101-Macro space.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k ImageNet16-120 Taskonomy MIT Places NAS-Bench-101 NAS-Bench-201 NAS-Bench-301 TransNAS-Bench-101 DARTS search space
Benchmarks: NAS-Bench-101 NAS-Bench-201 NAS-Bench-301 TransNAS-Bench-101-Micro TransNAS-Bench-101-Macro CIFAR-10 ImageNet
Metrics: Spearman rank correlation test error (%) GPU days parameters (M)
Baselines: NWOT ZiCo Zen-Score TE-NAS NAS-Bench-Suite-Zero metrics PNAS DARTS EcoNAS EvNAS PRE-NAS QE-NAS ProxylessNAS
Builds on: NWOT Zen-NAS TE-NAS AmoebaNet NAS-Bench-Suite-Zero
Closest papers by meaning
- Per-Architecture Training-Free Metric Optimization for Neural Architecture Search (NeurIPS 2025 · similarity 0.90)
- Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search (NeurIPS 2022 · similarity 0.90)
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (ICLR 2020 · similarity 0.89)
- NAS evaluation is frustratingly hard (ICLR 2020 · similarity 0.88)
- Robustifying and Boosting Training-Free Neural Architecture Search (ICLR 2024 · similarity 0.87)
- Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.