Research map / Architectures and efficiency / Neural architecture search
Encodings for Prediction-based Neural Architecture Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies neural architecture encodings (structural, score-based, learned) for prediction-based NAS across 13 search spaces, proposes the FLAN hybrid predictor and unified encodings enabling cross-search-space transfer of accuracy predictors.
Problem
Accuracy predictors for NAS are sample-inefficient, and it is unclear which neural architecture encodings work best, and how predictors can transfer across search spaces, tasks and datasets with few samples.
Approach
Categorizes encodings into structural, score-based, unsupervised learned and supervised learned types; proposes FLAN, a GNN predictor with dual graph flows (dense graph flow with residuals plus graph attention) and iterative operation-embedding updates, supplemented by zero-cost proxies and CATE/Arch2Vec encodings; unified encodings append search-space indices to operations so predictors transfer
Key results
- FLAN with ZCP supplemental encoding reaches Kendall-tau 0.698 on NASBench-101 with 1% of samples, up to ~15% above TA-GATES.
- Transfer of predictors across NAS spaces gives at least 8x sample-efficiency improvement over from-scratch predictors with 128 samples.
- End-to-end NAS with FLAN CAZ matches Zero-Cost NAS (W) Rand (3k) test accuracy with 2.12x fewer trained models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ZCP supplemental encoding to FLAN | FLAN · NASBench-101, 1% | Kendall-τ: 0.668 -> 0.698 | Helped |
| Add ZCP supplemental encoding to FLAN | FLAN · NASBench-201, 0.1% | Kendall-τ: 0.538 -> 0.510 | Hurt |
| Add ZCP supplemental encoding to FLAN | FLAN · NASBench-301, 0.5% | Kendall-τ: 0.572 -> 0.573 | Helped |
| Add ZCP supplemental encoding to FLAN | FLAN · ENAS, 5% | Kendall-τ: 0.345 -> 0.397 | Helped |
| Ensemble DGF and GAT in both graph-flow directions (ablation) | FLAN · NASBench-101, 128 training samples | Kendall-τ: 0.7088_{0.0003} -> 0.7322_{0.0002} | Helped |
| Ensemble DGF and GAT in both graph-flow directions (ablation) | FLAN · NASBench-201, 128 training samples | Kendall-τ: 0.7981_{0.0004} -> 0.8200_{0.0004} | Helped |
| Add the DGF residual connection (ablation) | FLAN · NB101, 1% training data | Kendall-τ: 0.59 -> 0.66 | Helped |
| Add zero-cost symmetry breaking (ablation) | FLAN · NB101, 5% training data | Kendall-τ: 0.76 -> 0.79 | Helped |
| Use more than 1 timestep (ablation) | FLAN · NASBench-201 and PNAS | Kendall-τ | Mixed |
| Transfer the predictor from a source NAS space rather than train from scratch | FLAN · Cross-NAS-space transfer | Kendall-τ: at least an 8× sample efficiency improvement | Helped |
| Fine-tune the transferred predictor with a few target samples | FLAN transferred from TB101 Class Scene · TransNASBench-101 target tasks | Kendall-τ | Mixed |
| Add supplementary encodings to FLAN | FLAN · NAS predictor training | sample efficiency: 15% | Helped |
Surprising findings
- Adding few-shot fine-tuning to a transferred predictor can degrade performance on some cross-task targets.
- Adjacency encodings are almost always superior to path encodings when the predictor design is controlled; the reported PATH advantage in naszilla setup did not hold on the authors' MLP design.
Limitations
- Evaluated only on cell-based and tabular/surrogate NAS benchmarks, not on large-scale real search.
- Learning-rate and other training hyperparameters are excluded from the predictor representation.
- Predictor-based NAS is studied only with a BRP-NAS-style iterative sampler.
Experimental setup
Datasets: NASBench-101 NASBench-201 NASBench-301 NDS TransNASBench-101 NAS-Bench-Suite-Zero ENAS
Benchmarks: NASBench-101 NASBench-201 NASBench-301 NDS TransNASBench-101 ENAS
Metrics: Kendall-tau Spearman-rho test accuracy vs number of trained models
Baselines: TA-GATES GATES GCN Arch2Vec CATE GENNAPE CDP BONAS BRP-NAS Zero-Cost NAS (W)
Builds on: TA-GATES BRP-NAS CATE Arch2Vec GENNAPE
Closest papers by meaning
- How Powerful are Performance Predictors in Neural Architecture Search? (NeurIPS 2021 · similarity 0.91)
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (ICLR 2021 · similarity 0.90)
- Neural Architecture Generator Optimization (NeurIPS 2020 · similarity 0.90)
- Insights from the Use of Previously Unseen Neural Architecture Search Datasets (CVPR 2024 · similarity 0.90)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.89)
- Do Not Train It: A Linear Neural Architecture Search of Graph Neural Networks (ICML 2023 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.