Research map / Architectures and efficiency / Neural architecture search
Towards Neural Architecture Search through Hierarchical Generative Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-level generative hierarchy (G-VAE + CCNF for micro cells, GPT-Neo sequence generator for macro architectures) pre-trained on zero-cost-proxy similarities to navigate a huge NAS search space at very low search cost.
Problem
NAS performance is bounded by the quality of a hand-crafted search space; naively searching very large initial spaces is expensive and existing automated search-space design methods require repeated NAS feedback.
Approach
Cluster a huge graph-based micro design space via a G-VAE latent space regularized by triplet loss on zero-cost proxy vectors and a GMM; sample synonymous cells with a conditional continuous normalizing flow; then train a GPT-Neo sequence generator on evolutionary-search histories to generate macro architectures under FLOPs/params conditions, with no feedback loop at search time.
Key results
- On CIFAR-10/100 with 1M params, the SG+CCNF method reaches 97.6% and 81.0% with about 0.08 GPU hours search cost, versus 97.2% and 80.4% for Evo(T-CET) at 19 GPU hours.
- On ImageNet-1k under 450M/600M/1000M FLOPs, the method reports 78.3%, 79.5% and 80.6% top-1 accuracy.
- On NAS-Bench-360, the method beats the WRN reference on 6 of 8 tasks and beats DASH on 4 of 8 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hierarchical search with CCNF to Evo(T-CET) on GraphNet (ablation) | Evo(T-CET) on GraphNet · CIFAR-10 | accuracy (%): 96.4 -> 97.4 | Helped |
| Add hierarchical search with CCNF to Evo(T-CET) on GraphNet (ablation) | Evo(T-CET) on GraphNet · CIFAR-100 | accuracy (%): 79.6 -> 80.9 | Helped |
| Add hierarchical search with CCNF to Evo(T-CET) on GraphNet (ablation) | Evo(T-CET) on GraphNet · ImageNet-1k | accuracy (%): 77.6 -> 78.4 | Helped |
| Train a sequence generator on the hierarchical-search history and use it with CCNF (ablation) | HL-Evo(T-CET)+CCNF · CIFAR-10 | accuracy (%): 97.4 -> 97.6 | Helped |
| Train a sequence generator on the hierarchical-search history and use it with CCNF (ablation) | HL-Evo(T-CET)+CCNF · CIFAR-100 | accuracy (%): 80.9 -> 81.0 | Helped |
| Train a sequence generator on the hierarchical-search history and use it with CCNF (ablation) | HL-Evo(T-CET)+CCNF · ImageNet-1k | accuracy (%): 78.4 -> 78.5 | Helped |
| Use SG+CCNF instead of OFA | ImageNet-1k, 450M FLOPs · ImageNet-1k (450M FLOPs) | accuracy (%): 79.1 -> 78.3±0.2 | Hurt |
| Use SG+CCNF instead of GPT NAS | ImageNet-1k, 450M FLOPs · ImageNet-1k (450M FLOPs) | accuracy (%): 79.1 -> 78.3±0.2 | Hurt |
| Use SG+CCNF instead of ZiCo | ImageNet-1k, 600M FLOPs · ImageNet-1k (600M FLOPs) | accuracy (%): 79.4±0.3 -> 79.5±0.1 | Helped |
| Use SG+CCNF instead of ZiCo | ImageNet-1k, 1000M FLOPs · ImageNet-1k (1000M FLOPs) | accuracy (%): 80.5±0.2 -> 80.6±0.4 | Helped |
| Use the proposed generative NAS method instead of DASH | NAS-Bench-360 · Spherical | error: 71.28 -> 65.72 | Helped |
| Use the proposed generative NAS method instead of DASH | NAS-Bench-360 · Darcy Flow | error: 0.008 -> 0.011 | Hurt |
Limitations
- Method fails noticeably on DeepSEA and Satellite tasks of NAS-Bench-360.
- Improvement on ImageNet accuracy over existing methods is modest; the authors note substantial improvements are not expected on a well-studied task.
- Relies on a pre-trained generative hierarchy and fixed zero-cost proxy set, and final model selection uses a simple T-CET-based selection.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k NAS-Bench-360
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k NAS-Bench-360
Metrics: top-1 accuracy search cost (GPU hours) parameters FLOPs
Baselines: ZenNAS ZiCo T-CET evolution OFA GPT NAS NSENet AutoSpace TE-NAS Zero-Cost-PT DASH
Builds on: Zero-Cost NAS Neural ODE / continuous normalizing flows (FFJORD) GPT-Neo Graph VAE ZenNAS
Closest papers by meaning
- Neural Architecture Generator Optimization (NeurIPS 2020 · similarity 0.92)
- Understanding Architectures Learnt by Cell-based Neural Architecture Search (ICLR 2020 · similarity 0.92)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.91)
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (ICLR 2021 · similarity 0.91)
- NAS-Bench-101: Towards Reproducible Neural Architecture Search (ICML 2019 · similarity 0.90)
- Robustifying and Boosting Training-Free Neural Architecture Search (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.