Research map / Architectures and efficiency / Neural architecture search
L-SWAG: Layer-Sample Wise Activation with Gradients Information for Zero-Shot NAS on Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L-SWAG, a layer-wise zero-cost proxy combining gradient statistics and sample-wise activation pattern counts, plus LIBRA-NAS, an ensemble that picks complementary proxies per benchmark, evaluated for zero-shot NAS on CNN and ViT search spaces.
Problem
Zero-cost proxies for training-free NAS are mostly validated on convolutional search spaces and lack evaluation on Vision Transformer search spaces; different proxies carry complementary information that is hard to combine.
Approach
A layer-wise zero-cost metric that drops the gradient mean (using a constant numerator) and keeps gradient standard deviation over selected layers, multiplied by a layer-wise count of binarized ReLU/GeLU activation patterns; LIBRA then selects three proxies by correlation, information gain and bias matching.
Key results
- L-SWAG achieves average Spearman correlation 0.72 across benchmarks, versus 0.62 for the second-best proxy NWOT.
- LIBRA finds an Autoformer architecture with 17.0% ImageNet1k test error in 0.1 GPU days, versus 17.8% for AZ-NAS and 18.3% for evolution-based Autoformer search.
- On DARTS CIFAR-10, LIBRA reaches 2.45% test error at 0.08 GPU days, slightly better than SWAG (2.47%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LIBRA instead of SWAG for zero-shot search | NAS on DARTS · Cifar-10 | test error (%): 2.47 -> 2.45 | Helped |
| Use LIBRA instead of SWAG for zero-shot search | NAS on DARTS · ImageNet1k | test error (%): 23.4 -> 23.1 | Helped |
| Use LIBRA instead of SWAG for zero-shot search | NAS on Autoformer Small · ImageNet1k | test error (%): 17.8 -> 17.0 | Helped |
| Remove the gradient mean from L-SWAG (ablation) | L-SWAG · NASBench-201 Cifar-10 | Spearman ρ: 0.75 -> 0.78 | Helped |
| Select a layer interval for gradient statistics (ablation) | L-SWAG · TransNAS-Bench-101 Macro Autoencoder | Spearman ρ: 0.19 -> 0.24 | Helped |
| Add the expressivity term Ψ (ablation) | L-SWAG · TransNAS-Bench-101 Macro Autoencoder | Spearman ρ: 0.01 -> 0.71 | Helped |
| Select layers for gradient statistics and add the expressivity term Ψ (ablation) | L-SWAG · TransNAS-Bench-101 Macro Autoencoder | Spearman ρ: 0.19 -> 0.79 | Helped |
| Choose the second proxy by minimizing information gain (ablation) | LIBRA · NASBench-301 Cifar-10 | Spearman ρ: 0.53 -> 0.71 | Helped |
| Add a third proxy selected by bias matching (ablation) | LIBRA · NASBench-301 Cifar-10 | Spearman ρ: 0.71 -> 0.74 | Helped |
| Add a third proxy selected by bias matching (ablation) | LIBRA · TransNAS-Bench-101 Macro Scene | Spearman ρ: 0.90 -> 0.91 | Helped |
Surprising findings
- Removing the gradient mean (using a constant numerator instead of mu) improves correlation, contrary to the ZiCO formulation.
- Many existing ZC proxies struggle to outperform the simple parameter count on the ViT search space.
Limitations
- LIBRA evaluation is limited to empirical analysis without theoretical grounding.
- Method is not extended to video or other input modalities.
- Random-search baselines are competitive on the ViT search space, making guidance evaluation difficult.
Experimental setup
Datasets: NAS-Bench-201 NAS-Bench-101 NAS-Bench-301 TransNAS-Bench-101 Micro TransNAS-Bench-101 Macro ImageNet CIFAR-10 CIFAR-100 Pets SVHN Spherical-CIFAR-100
Benchmarks: NAS-Bench-201 NAS-Bench-101 NAS-Bench-301 TransNAS-Bench-101 Micro TransNAS-Bench-101 Macro Autoformer Small search space DARTS search space CIFAR-10 ImageNet1k
Metrics: Spearman rho test error (%) search time (GPU days)
Baselines: ZiCO SWAP reg SWAP NWOT Zen-Score Fisher GraSP SynFlow T-CET TE-NAS AZ-NAS TF-TAS
Builds on: ZiCO SWAP NWOT NAS-Bench-SuiteZero AZ-NAS
Closest papers by meaning
- ZiCo: Zero-shot NAS via inverse Coefficient of Variation on Gradients (ICLR 2023 · similarity 0.91)
- ParZC: Parametric Zero-Cost Proxies for Efficient NAS (AAAI 2025 · similarity 0.90)
- Revolutionizing Training-Free NAS: Towards Efficient Automatic Proxy Discovery via Large Language Models (NeurIPS 2025 · similarity 0.90)
- NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet Training (ICLR 2022 · similarity 0.89)
- Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights (CVPR 2025 · similarity 0.87)
- Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.