Research map / Architectures and efficiency / Neural architecture search
Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VKDNW, a training-free NAS proxy built from the entropy of Fisher Information Matrix eigenvalue deciles at initialization, plus an nDCG-based metric for evaluating TF-NAS proxies.
Problem
Neural architecture search is computationally expensive because each candidate must be trained, so training-free proxies that rank architectures without training are needed; existing correlation metrics also fail to reflect how well proxies pick top networks.
Approach
Uses the entropy of the deciles of the empirical Fisher Information Matrix spectrum at initialization (computed on random inputs, with a numerically stable decomposition) as a size-invariant zero-cost proxy, combined with the number of trainable layers; also introduces nDCG as a metric emphasizing top-ranked architectures.
Key results
- VKDNW aggregated ranking reaches KT 0.743, SPR 0.906, nDCG 0.664 on ImageNet16-120 in NAS-Bench-201, outperforming prior zero-cost methods such as AZ-NAS (nDCG 0.534).
- Combining VKDNW, ZCS and GRAF (model-driven) reaches KT 0.859 / nDCG 0.946 on CIFAR-10, surpassing GRAF alone (nDCG 0.935).
- The architecture found by VKDNW-agg in the MobileNetV2 space reaches 78.8% top-1 on ImageNet-1K at about 480M FLOPs, versus 78.6% for AZ-NAS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate VKDNW with Jacov, expressivity, trainability, and FLOPs (ablation) | VKDNW ranking · ImageNet16-120 | nDCG: 0.608 -> 0.664 | Helped |
| Add Jacov to the aggregated ranking (ablation) | VKDNW ranking · ImageNet16-120 | nDCG: 0.623 -> 0.664 | Helped |
| Add expressivity to the aggregated ranking (ablation) | VKDNW ranking · ImageNet16-120 | nDCG: 0.553 -> 0.664 | Helped |
| Add trainability to the aggregated ranking (ablation) | VKDNW ranking · ImageNet16-120 | nDCG: 0.695 -> 0.664 | Hurt |
| Add FLOPs to the aggregated ranking (ablation) | VKDNW ranking · ImageNet16-120 | nDCG: 0.658 -> 0.664 | Helped |
| Use random input rather than real images (ablation) | VKDNW · ImageNet16-120 | nDCG: 0.595 -> 0.617 | Helped |
| Increase batch size from 8 to 64 with random input (ablation) | VKDNW · ImageNet16-120 | Spearman’s ρ: 0.808 -> 0.814 | No clear effect |
| Use VKDNW instead of AZ-NAS for MobileNetV2 search | MobileNetV2 search on ImageNet-1K · ImageNet-1K | Top-1 accuracy: 78.6 -> 78.8 | Helped |
| Add VKDNW features to the model-driven ZCS ranking | Model-driven ranking · ImageNet16-120 | nDCG: 0.534 -> 0.575 | Helped |
| Add ZCS features to the VKDNW model-driven ranking | Model-driven ranking · ImageNet16-120 | nDCG: 0.575 -> 0.843 | Helped |
| Add GRAF features to the VKDNW+ZCS model-driven ranking | Model-driven ranking · ImageNet16-120 | nDCG: 0.843 -> 0.867 | Helped |
Surprising findings
- Random white-noise inputs perform as well as real images for the proxy, with no loss on KT, SPR, or nDCG.
- Most zero-cost proxies underperform a simple FLOPs or parameter-count proxy in prior work, as the paper notes.
Limitations
- Proxy is evaluated mainly on NAS-Bench-201 and one MobileNetV2 space, so generalization to larger search spaces is untested.
- Best model-driven results require a set of trained architectures to fit the regression model, which limits applicability.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet16-120 ImageNet-1K
Benchmarks: NAS-Bench-201 (CIFAR-10, CIFAR-100, ImageNet16-120) MobileNetV2 search space on ImageNet-1K
Metrics: Kendall's tau Spearman's rho nDCG top-1 accuracy
Baselines: FLOPs GradNorm GraSP SNIP SynFlow Jacov NASWOT ZenNAS GradSign ZiCo TE-NAS AZ-NAS
Builds on: Jacov AZ-NAS ZiCo GRAF NAS-Bench-201
Closest papers by meaning
- EcoNAS: Finding Proxies for Economical Neural Architecture Search (CVPR 2020 · similarity 0.90)
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware (ICLR 2019 · similarity 0.90)
- Robustifying and Boosting Training-Free Neural Architecture Search (ICLR 2024 · similarity 0.88)
- L-SWAG: Layer-Sample Wise Activation with Gradients Information for Zero-Shot NAS on Vision Transformers (CVPR 2025 · similarity 0.87)
- NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance (ICLR 2025 · similarity 0.87)
- Insights from the Use of Previously Unseen Neural Architecture Search Datasets (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.