Research map / Image generation / Fast diffusion sampling
DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffusionNAG, a graph diffusion model over DAG-encoded neural architectures with a positional-embedding transformer score network, conditionally guided by property predictors to generate task-optimal architectures for transferable NAS and BO-based NAS.
Problem
Existing NAS methods waste large search cost sampling and filtering many task-irrelevant architectures, with predictors used only passively for ranking; graph diffusion models also target undirected graphs and fail to model the computational flow of neural architectures.
Approach
Reframe NAS as conditional neural architecture generation: train an unconditional score-based diffusion model over DAG architecture encodings once (without accuracy labels), then steer reverse sampling with the gradient of a dataset-aware or task-specific predictor, plus a positional-embedding transformer with DAG attention masks that yields 100% valid architectures.
Key results
- On MBv3 transferable NAS, DiffusionNAG's top-30 architectures have the highest mean accuracy on CIFAR-10 (97.39%), CIFAR-100 (85.74%), and Oxford-IIIT Pets (94.75%), with p-values below 0.05 against baselines.
- On NB201 transferable NAS, DiffusionNAG matches the best oracle accuracy on CIFAR-10 and CIFAR-100 with 1–2 trained architectures, and reaches 41.80% on Oxford-IIIT Pets, with a reported speedup of 15× to 35×.
- The score network achieves 100% validity on both NB201 and MBv3, whereas GDSS achieves 4.56% on NB201 and 0.00% on MBv3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use predictor-guided conditional generation instead of unconditional generation and predictor-based ranking | DiffusionNAG on MBv3 · CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets | mean accuracy: 97.28 $\pm$0.01 -> 97.39 $\pm$0.01; 85.56 $\pm$0.02 -> 85.74 $\pm$0.04; 81.19 $\pm$0.11 -> 81.47 $\pm$0.05; 94.55 $\pm$0 | Helped |
| use predictor-guided conditional generation instead of predictor-only evaluation | DiffusionNAG on NB201 · Aircraft and Oxford-IIIT Pets | accuracy (%): 59.15 $\pm$0.58 -> 58.83 $\pm$3.75; 40.00 $\pm$0.00 -> 41.80 $\pm$3.82 | Mixed |
| use predictor-guided conditional generation | DiffusionNAG on NB201 · Aircraft and Oxford-IIIT Pets | trained architectures: 26 -> 3; 6 -> 2 | Helped |
| add positional embedding (ablation) | DiffusionNAG score network · MobileNetV3 | validity (%): 42.17 $\pm$1.80 -> 100.00 $\pm$0.00 | Helped |
| add positional embedding (ablation) | DiffusionNAG score network · NAS-Bench-201 | novelty (%): 49.08 $\pm$2.05 -> 49.20 $\pm$1.96 | Helped |
| add positional embedding (ablation) | DiffusionNAG score network · NAS-Bench-201 | uniqueness (%): 98.96 $\pm$0.49 -> 98.70 $\pm$0.66 | Hurt |
| replace directed-graph score network with undirected-graph GDSS | architecture generation · NAS-Bench-201 | validity (%): 100.00 $\pm$0.00 vs 4.56 $\pm$1.44 | Hurt |
| replace directed-graph score network with undirected-graph GDSS | architecture generation · MobileNetV3 | validity (%): 100.00 $\pm$0.00 vs 0.00 $\pm$0.00 | Hurt |
| use DiffusionNAG rather than heuristic acquisition optimization | BO-based NAS · NB201 CIFAR-100 and MBv3 ImageNet 1K | search performance | Helped |
Surprising findings
- The undirected-graph diffusion baseline GDSS almost completely fails to produce valid architectures (0.00% validity on MBv3), showing that directed structure and positional embeddings are essential.
- Without positional embedding, validity on MBv3 drops to 42.17%, indicating that long-range layer ordering is hard to capture.
Limitations
- On the Aircraft dataset in the MBv3 comparison, DiffusionNAG's max accuracy is slightly below TNAS (82.28 vs 82.31), and on NB201 Aircraft it is slightly below TNAS (58.83 vs 59.15).
- Speedup claims depend on counting trained architectures on NB201 benchmarks, and the generation step itself takes up to 2.02 GPU minutes per batch of 256.
Experimental setup
Datasets: NAS-Bench-201 MobileNetV3 search space meta-dataset from Lu et al. (TNAS/MetaD2A) CIFAR-10 CIFAR-100 Aircraft Oxford-IIIT Pets ImageNet 1K
Benchmarks: NAS-Bench-201 MobileNetV3 search space CIFAR-10 CIFAR-100 Aircraft Oxford-IIIT Pets ImageNet 1K
Metrics: accuracy max/mean/min accuracy p-value validity uniqueness novelty trained architectures
Baselines: MetaD2A TNAS REA RS REINFORCE BANANAS NASBOWL HEBO
Builds on: score-based SDE diffusion classifier guidance MetaD2A TNAS GDSS
Closest papers by meaning
- Manifold Preserving Guided Diffusion (ICLR 2024 · similarity 0.87)
- Learning Diffusion Models with Flexible Representation Guidance (NeurIPS 2025 · similarity 0.87)
- Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models (AAAI 2025 · similarity 0.87)
- Denoising Task Routing for Diffusion Models (ICLR 2024 · similarity 0.87)
- Analyzing and Improving the Training Dynamics of Diffusion Models (CVPR 2024 · similarity 0.87)
- TFG: Unified Training-Free Guidance for Diffusion Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.