Research map / Architectures and efficiency / Neural architecture search
AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AZ-NAS, a training-free NAS method that assembles four complementary zero-cost proxies (expressivity, progressivity, trainability, complexity) with non-linear ranking aggregation to rank architectures.
Problem
Training-free NAS zero-cost proxies show weak correlation with final network performance, so rankings of candidate architectures are unreliable.
Approach
Combine four proxies analyzing activations, gradients and FLOPs from different perspectives (expressivity via PCA eigenvalue entropy, progressivity via block-wise expressivity increase, trainability via Hutchinson-estimated Jacobian spectral norm, complexity via FLOPs), computed in one forward/backward pass, and aggregate their ranks non-linearly by summing log of normalized ranks so that networks
Key results
- On NAS-Bench-201 ImageNet16-120, AZ-NAS reaches Kendall's tau 0.710 versus 0.584 for ZiCo and 0.573 for NASWOT, at 42.7 ms/arch.
- On MobileNetV2 search space with 600M FLOPs, AZ-NAS achieves 79.9% top-1 ImageNet accuracy, above ZiCo (79.4%) and ZenNAS (79.1%), with 0.6 GPU days search cost.
- On the 1000M FLOPs MobileNetV2 setting AZ-NAS reaches 81.1% top-1, and on AutoFormer Tiny it reaches 76.1% at 0.03 GPU days.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| assemble the four AZ-NAS proxies instead of using ZiCo alone | NAS-Bench-201 ranking · CIFAR-10 | Kendall’s τ: 0.589 -> 0.741 | Helped |
| assemble the four AZ-NAS proxies instead of using ZiCo alone | NAS-Bench-201 ranking · CIFAR-100 | Kendall’s τ: 0.590 -> 0.723 | Helped |
| assemble the four AZ-NAS proxies instead of using ZiCo alone | NAS-Bench-201 ranking · ImageNet16-120 | Kendall’s τ: 0.584 -> 0.710 | Helped |
| use AZ-NAS rather than ZiCo to select a network | MobileNetV2 search · ImageNet, 450M FLOPs constraint | top-1 validation accuracy: 78.1 -> 78.6 | Helped |
| use AZ-NAS rather than ZiCo to select a network | MobileNetV2 search · ImageNet, 600M FLOPs constraint | top-1 validation accuracy: 79.4 -> 79.9 | Helped |
| use AZ-NAS rather than ZenNAS to select a network | MobileNetV2 search · ImageNet, 1000M FLOPs constraint | top-1 validation accuracy: 80.8 -> 81.1 | Helped |
| add trainability to the expressivity and progressivity proxies (ablation) | AZ-NAS proxy aggregation · CIFAR-10 | Kendall’s τ: 0.601 -> 0.679 | Helped |
| add complexity to expressivity and progressivity proxies (ablation) | AZ-NAS proxy aggregation · CIFAR-10 | Kendall’s τ: 0.601 -> 0.683 | Helped |
| use nonlinear rather than linear ranking aggregation (ablation) | AZ-NAS proxy aggregation · CIFAR-10 | Kendall’s τ: 0.697 -> 0.741 | Helped |
| use trainability alone rather than the full proxy ensemble (ablation) | AZ-NAS · CIFAR-10 | Kendall’s τ: 0.741 -> 0.349 | Hurt |
Surprising findings
- Trainability proxy alone performs poorly (KT 0.349-0.407) but improves ensembles, being the least correlated with other proxies.
Limitations
- Progressivity proxy not applied to AutoFormer search space because ViT attention yields similar features at initialization.
- Proxies are not tailored to ViTs; authors leave zero-cost proxies suited for ViTs to future work.
- Search results on MobileNetV2 and AutoFormer are reported with varying FLOPs/params, making direct comparison sensitive to model size.
Experimental setup
Datasets: NAS-Bench-201 CIFAR-10 CIFAR-100 ImageNet16-120 ImageNet
Benchmarks: NAS-Bench-201 MobileNetV2 search space (ImageNet) AutoFormer search space (ImageNet)
Metrics: Kendall's tau Spearman's rho top-1 accuracy runtime (ms/arch) search cost (GPU days)
Baselines: GradNorm Grasp Snip Synflow NASWOT TE-NAS ZenNAS GradSign ZiCo TF-TAS AutoFormer OFA
Builds on: TE-NAS ZiCo NASWOT ZenNAS Synflow
Closest papers by meaning
- Zero-Cost Proxies for Lightweight NAS (ICLR 2021 · similarity 0.91)
- ParZC: Parametric Zero-Cost Proxies for Efficient NAS (AAAI 2025 · similarity 0.89)
- Surprisingly Strong Performance Prediction with Neural Graph Features (ICML 2024 · similarity 0.89)
- NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance (ICLR 2025 · similarity 0.87)
- SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS (ICLR 2024 · similarity 0.86)
- HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.