Research map / Architectures and efficiency / Neural architecture search
Prior Knowledge Guided Neural Architecture Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PG-NAG, a Shapley-value-guided conditional diffusion model that generates high-performance neural architectures from benchmark-derived prior knowledge without search or evaluation.
Problem
Neural architecture search requires evaluating many candidate architectures, which is computationally expensive; existing neural architecture generation methods still depend on predictors to guide generation.
Approach
Quantify each operation's and one-hop subgraph's contribution to accuracy via Shapley values on top architectures from NAS benchmarks, and use these as prior knowledge to modify the noise prior and guide a graph-based conditional diffusion model, removing the need for predictors or search.
Key results
- Generates a DARTS-space architecture with 76.1% ImageNet top-1 and 97.56% CIFAR-10 accuracy at 0.004 GPU days.
- Achieves 93.91% CIFAR-10 and 44.73% ImageNet16-120 on NATS-Bench at 4,147 s search cost, competitive with NAS baselines.
- Obtains the lowest average TIMIT PER (21.78%) and lowest mean test log perplexity (4.57) on NAS-Bench-ASR and NAS-Bench-NLP among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add prior-knowledge guidance from high-performance architectures (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | test accuracy (%): 93.31 -> 94.36 | Helped |
| replace GCN with GIN for operation feature extraction (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | test accuracy (%): 94.36 -> 94.37 | No clear effect |
| replace GCN with GAT for operation feature extraction (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | test accuracy (%): 94.36 -> 94.00 | Hurt |
| replace one-hop-subgraph connection features with connection-only MLP features (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | test accuracy (%): 94.36 -> 94.30 | Hurt |
| use standard DDPM instead of the PG-NAG diffusion model (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | accuracy (%): 93.28 vs 94.36 | Hurt |
| increase the number of diffusion steps from 500 to 800 (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | accuracy (%): 92.73 -> 94.22 | Helped |
| increase prior-knowledge set size from k=10 to k=20 (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | test accuracy (%): 92.31 -> 94.36 | Helped |
| increase prior-knowledge set size from k=20 to k=50 (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | test accuracy (%): 94.36 -> 94.37 | No clear effect |
| use 20 randomly selected architectures instead of high-performance architectures as prior knowledge (ablation) | PG-NAG · NAS-Bench-201 CIFAR-10 | test accuracy (%): 94.36 -> 87.11 | Hurt |
| generate architectures with PG-NAG instead of BOHB | neural architecture generation/search · NATS-Bench CIFAR-10 | accuracy (%): 93.94±0.28 vs 93.91±0.08 | No clear effect |
| generate architectures with PG-NAG instead of BOHB | neural architecture generation/search · NATS-Bench CIFAR-100 | accuracy (%): 72.00±0.86 vs 70.96±0.14 | Hurt |
| generate architectures with PG-NAG instead of BOHB | neural architecture generation/search · ImageNet-16-120 | accuracy (%): 45.70±0.86 vs 44.73±0.35 | Hurt |
Surprising findings
- Using 20 random architectures as prior knowledge instead of high-performance ones substantially degrades generated architecture accuracy, showing prior knowledge quality matters.
Limitations
- Relies on architectures from existing NAS benchmarks as prior knowledge, which limits applicability to spaces without such benchmarks.
- Reported results on new search spaces use selected best or average of only five generated architectures per space.
Experimental setup
Datasets: NAS-Bench-101 NAS-Bench-201 NAS-Bench-301
Benchmarks: ImageNet CIFAR-10 NATS-Bench TransNAS-Bench-101 NAS-Bench-ASR NAS-Bench-NLP NAS-Bench-201
Metrics: top-1 test error accuracy GPU days phoneme error rate test log perplexity search cost
Baselines: DARTS SNAS P-DARTS GDAS PC-DARTS TE-NAS PRE-NAS ENAS BOHB RSPS DiffusionNAG NASBOT
Builds on: DDPM DiffusionNAG classifier-free guidance
Closest papers by meaning
- Building Optimal Neural Architectures using Interpretable Knowledge (CVPR 2024 · similarity 0.88)
- Hierarchical Representations for Efficient Architecture Search (ICLR 2018 · similarity 0.88)
- Efficient Architecture Search by Network Transformation (AAAI 2018 · similarity 0.88)
- Contrastive Neural Architecture Search With Neural Architecture Comparators (CVPR 2021 · similarity 0.87)
- Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search (NeurIPS 2026 · similarity 0.87)
- Towards Neural Architecture Search through Hierarchical Generative Modeling (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.