Research map / Architectures and efficiency / Neural architecture search
ParZC: Parametric Zero-Cost Proxies for Efficient NAS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ParZC, a parametric zero-cost proxy that uses a Bayesian mixer network over node-wise ZC statistics trained with a differentiable Kendall's Tau loss (DiffKendall) to rank architectures for zero-shot NAS.
Problem
Zero-cost proxies aggregate node-wise statistics uniformly, ignoring that nodes differ in their contribution to performance, and hand-crafted or automated (parameter-free) proxies are static, slow to find, and fail to transfer across search spaces.
Approach
Replace fixed aggregation of node-wise zero-cost statistics with a learned mixer architecture containing Bayesian linear layers to model per-node uncertainty, trained with a sigmoid-smoothed differentiable Kendall's Tau loss so that the objective matches ranking rather than regression error.
Key results
- On NAS-Bench-201 (CIFAR-10), ParZC reaches 94.36% test accuracy with 68 GPU seconds of search, versus 93.90% for TE-NAS and 93.55% for NASI.
- On NAS-Bench-101 with 178 samples, DiffKendall alone gives 66.83% KD and 85.97% SP, outperforming MSE loss alone (65.69% KD).
- On NB201 CIFAR-10 zero-cost rank correlation, ParZC attains 90.4 SP / 70.6 KD versus 83.0 / 65.0 for EZNAS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Mixer architecture to the predictor (vs MLP baseline) (ablation) | ParZC predictor · NAS-Bench-201, 78 samples | KD (%): 62.29 vs 54.64 (+7.65) | Helped |
| add Bayesian network to the mixer (ablation) | ParZC predictor · NAS-Bench-201, 78 samples | KD (%): 67.69 vs 62.29 (Mixer only), +5.4 | Helped |
| add neural predictor (NP) encoding to Mixer+BN (ablation) | ParZC predictor · NAS-Bench-201, 78 samples | KD (%): 68.89 vs 67.69, +1.2 | Helped |
| use DiffKendall as the sole loss instead of MSE, rank loss, or combinations (ablation) | ParZC on NAS-Bench-101 · NAS-Bench-101, 178 samples | KD (%): 66.83 vs 65.69 (MSE only) | Helped |
| combine DiffKendall with MSE or rank loss (ablation) | ParZC loss · NAS-Bench-101, 178 samples | KD (%): 65.56 and 64.41 vs 66.83 with DiffKendall alone | Hurt |
| ParZC (parametric proxy) vs hand-crafted ZC proxies on NAS-Bench-201 CIFAR-10 | ZC proxies · NB201-CF10 | KD (%): 70.6 vs 65.0 for EZNAS (best baseline) | Helped |
| ParZC vs EZNAS on NB101 | ZC proxies · NAS-Bench-101 CIFAR-10 | KD (%): 63.7 vs ZiCo 46.0 (EZNAS 4.5) | Helped |
| ParZC vs EZNAS on NDS-DARTS | ZC proxies · NDS-DARTS | KD (%): 50.3 vs 56.0 for EZNAS | Hurt |
| ParZC vs EZNAS on NDS-NASNet | ZC proxies · NDS-NASNet | KD (%): 38.5 vs 44.0 for EZNAS (SP 54.9 vs 50.0 higher) | Mixed |
| ParZC vs Synflow in Autoformer ViT search space | ZC proxies · Autoformer ViT search space | Kendall's Tau (%): 41.4 vs 14.8 for Synflow | Helped |
| ParZC searched architecture on ImageNet-1k (Autoformer space) | ViT architecture search · ImageNet-1k | Top-1 (%): 75.5 vs 75.3 for TF-TAS-Ti | Helped |
Surprising findings
- Combining DiffKendall with MSE or rank losses degrades rank correlation, so the pure differentiable Kendall loss performed best.
Limitations
- Relies on node-wise ZC statistics from a fixed set of proxies (Synflow, SNIP, GradNorm, Fisher) and a training set of about 1,000 benchmark-labeled architectures.
- Several results (e.g., NDS DARTS KD) do not beat the best baseline, and EZNAS remains stronger on some NDS spaces.
Experimental setup
Datasets: NAS-Bench-101 NAS-Bench-201 NDS (DARTS, NASNet, ENAS) AutoFormer ViT search space on ImageNet-1k MQ-Bench-101
Benchmarks: NAS-Bench-101 NAS-Bench-201 NDS ImageNet-1k (AutoFormer ViT search space) MQ-Bench-101
Metrics: Kendall's Tau Spearman correlation Pearson correlation test accuracy search cost (GPU seconds)
Baselines: Synflow GradNorm Fisher SNIP NWOT Zen ZiCo EZNAS TF-TAS PINAT CTNAS TNASP
Builds on: EZNAS MLP-Mixer Neural Predictor (NP) ReNAS AutoFormer TF-TAS
Closest papers by meaning
- L-SWAG: Layer-Sample Wise Activation with Gradients Information for Zero-Shot NAS on Vision Transformers (CVPR 2025 · similarity 0.90)
- AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search (CVPR 2024 · similarity 0.89)
- NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance (ICLR 2025 · similarity 0.88)
- Surprisingly Strong Performance Prediction with Neural Graph Features (ICML 2024 · similarity 0.87)
- Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery (AAAI 2024 · similarity 0.87)
- Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.