Research map / Architectures and efficiency / Neural architecture search
NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NEAR, a training-free zero-cost proxy scoring networks by the effective rank of pre- and post-activation matrices, and applies it to NAS ranking, MLP layer-size estimation, and activation/initialization selection.
Problem
Predicting final model accuracy of neural networks without training, for NAS and for choosing layer sizes, activation functions and weight initializations, including activations not restricted to ReLU.
Approach
Score a network by summing effective ranks of pre-activation and post-activation matrices over layers, computed on sample inputs without labels or ReLU-specific activation patterns; use a power-law fit of the relative score to estimate layer sizes.
Key results
- NEAR achieves the best average Spearman/Kendall rank of 1.71 across NAS-Bench-101, NATS-Bench-TSS and SSS, versus 2.57 for the second-best MeCo_opt.
- On NAS-Bench-101 NEAR reaches Kendall's tau 0.52 and Spearman's rho 0.70, the highest among the 12 compared proxies.
- For the lMLP machine learning potential, the NEAR-predicted layer sizes (52/58/60) reduce parameters from 22,990 to 13,909 with test loss rising only from 0.012 to 0.015.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NEAR instead of Zen-Score to rank NAS-Bench-101 architectures | zero-cost proxy ranking · NAS-Bench-101 CIFAR-10 | Spearman's rho: 0.64 -> 0.70 | Helped |
| Use NEAR instead of MeCo_opt to rank NATS-Bench-SSS CIFAR-100 architectures | zero-cost proxy ranking · NATS-Bench-SSS CIFAR-100 | Spearman's rho: 0.81 -> 0.82 | Helped |
| Use NEAR instead of SynFlow to rank NATS-Bench-SSS CIFAR-10 architectures | zero-cost proxy ranking · NATS-Bench-SSS CIFAR-10 | Spearman's rho: 0.94 -> 0.91 | Hurt |
| Use NEAR instead of #Params to rank TransNAS-Bench-101-Micro architectures | zero-cost proxy ranking · TransNAS-Bench-101-Micro Cls. Object | Spearman's rho: 0.53 -> 0.74 | Helped |
| Use NEAR instead of #Params to rank HW-GPT-Bench architectures | zero-cost proxy ranking · HW-GPT-Bench small | Spearman's rho with perplexity: -0.99 -> -0.99 | No clear effect |
| Estimate lMLP hidden-layer widths as 52-58-60 instead of 102-61-44 | lMLP trained on dataset B · lMLP dataset B | test loss: 0.012 \pm 0.003 -> 0.015 \pm 0.004 | Hurt |
| Increase the third hidden layer from 60 to 100 neurons (ablation) | lMLP widths 52-58-60 · lMLP dataset B | test loss: 0.015 \pm 0.004 -> 0.015 \pm 0.003 | No clear effect |
| Reduce the third hidden layer from 60 to 40 neurons (ablation) | lMLP widths 52-58-60 · lMLP dataset B | test loss: 0.015 \pm 0.004 -> 0.015 \pm 0.003 | No clear effect |
| Reduce the second hidden layer from 58 to 38 neurons (ablation) | lMLP widths 52-58-60 · lMLP dataset B | test loss: 0.015 \pm 0.004 -> 0.018 \pm 0.004 | Hurt |
| Increase the second hidden layer from 58 to 78 neurons (ablation) | lMLP widths 52-58-60 · lMLP dataset B | test loss: 0.015 \pm 0.004 -> 0.014 \pm 0.004 | Helped |
| Reduce the second hidden layer from 200 to 120 neurons (ablation) | Balanced EMNIST MLP with 200-200 layers · Balanced EMNIST | test loss: 0.368 \pm 0.006 -> 0.383 \pm 0.009 | Hurt |
| Increase the second hidden layer from 200 to 280 neurons (ablation) | Balanced EMNIST MLP with 200-200 layers · Balanced EMNIST | test loss: 0.368 \pm 0.006 -> 0.364 \pm 0.008 | Helped |
Surprising findings
- On HW-GPT-Bench, NEAR did not outperform the simple #Params baseline in correlation with perplexity.
- NEAR's correlation with accuracy on NATS-Bench-SSS is not uniformly best; MeCo_opt and reg_swap do not consistently beat #Params there.
Limitations
- Scores depend on randomly sampled inputs, requiring repeated averaging for stability.
- NEAR captures expressivity but not training dynamics, so it cannot resolve minor differences in final loss.
- Does not consistently beat the #Params baseline on all settings such as HW-GPT-Bench.
Experimental setup
Datasets: NAS-Bench-101 NATS-Bench-TSS NATS-Bench-SSS CIFAR-10 CIFAR-100 ImageNet16-120 TransNAS-Bench-101-Micro HW-GPT-Bench balanced EMNIST lMLP molecular dataset (S_N2 reactions)
Benchmarks: NAS-Bench-101 NATS-Bench-TSS NATS-Bench-SSS TransNAS-Bench-101-Micro HW-GPT-Bench
Metrics: Kendall's tau Spearman's rho average rank test loss classification accuracy
Baselines: Grad_norm SNIP GraSP Fisher SynFlow Zen-Score ZiCo MeCo_opt swap reg_swap FLOPs #Params
Builds on: NASWOT Zen-Score swap reg_swap MeCo SynFlow ZiCo
Closest papers by meaning
- ParZC: Parametric Zero-Cost Proxies for Efficient NAS (AAAI 2025 · similarity 0.88)
- Encodings for Prediction-based Neural Architecture Search (ICML 2024 · similarity 0.88)
- Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective (ICML 2025 · similarity 0.88)
- Training-free Neural Architecture Search through Variance of Knowledge of Deep Network Weights (CVPR 2025 · similarity 0.87)
- Surprisingly Strong Performance Prediction with Neural Graph Features (ICML 2024 · similarity 0.87)
- L-SWAG: Layer-Sample Wise Activation with Gradients Information for Zero-Shot NAS on Vision Transformers (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.