Research map / Large language models / Agents, code and math
RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RZ-NAS, an evolutionary LLM-guided NAS that mutates architecture genotypes with GPT-4o using text- and code-level prompts, LLM reflection feedback, and training-free zero-cost proxies.
Problem
LLM-to-NAS methods are limited to small search spaces, are costly because each candidate is fully trained, and underperform standard NAS benchmarks and downstream tasks.
Approach
Combines structured prompts giving the LLM text and code descriptions of the search space, network construction, and zero-cost proxy with an LLM reflection module that turns proxy scores and errors into mutation suggestions, so mutations are scored by training-free proxies instead of full training.
Key results
- On NAS-Bench-201 CIFAR-10, RZ-NAS with ZiCo reaches 94.24% test accuracy versus 93.35% for the original ZiCo proxy.
- On the DARTS space CIFAR-10, RZ-NAS with Synflow reduces test error from 5.9% to 4.3%, and with GraSP from 6.0% to 5.4%.
- On ImageNet with a 450M FLOPs budget, RZ-NAS with ZiCo reaches 79.0% top-1 accuracy (21.0% error), reported as higher than DONNA at roughly 62x lower search cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RZ-NAS with ZiCo instead of the original ZiCo proxy search | NAS-Bench-201 search · CIFAR-10 | test accuracy: 93.35 ± 0.00 -> 94.24 ± 0.12 | No clear effect |
| use RZ-NAS with ZiCo instead of the original ZiCo proxy search | NAS-Bench-201 search · CIFAR-100 | test accuracy: 70.71 ± 0.00 -> 73.30 ± 0.21 | Helped |
| use RZ-NAS with ZiCo instead of the original ZiCo proxy search | NAS-Bench-201 search · ImageNet-16-120 | test accuracy: 46.18 ± 0.04 -> 46.24 ± 0.23 | No clear effect |
| use RZ-NAS with Synflow instead of the original Synflow proxy search | DARTS search · CIFAR-10 | test error: 5.9 ± 0.11 -> 4.3 ± 0.07 | Helped |
| use RZ-NAS with Synflow instead of the original Synflow proxy search | DARTS search · CIFAR-100 | test error: 24.1 ± 0.12 -> 21.3 ± 0.20 | Helped |
| use RZ-NAS with ZiCo instead of the original ZiCo proxy search | MobileNet search · ImageNet 450M FLOPs | Top-1 test error: 22.0 -> 21.0 | Helped |
| use RZ-NAS with ZiCo instead of the original ZiCo proxy search | MobileNet search · ImageNet 600M FLOPs | Top-1 test error: 20.6 -> 19.9 | Helped |
| use RZ-NAS with ZiCo instead of the original ZiCo proxy search | MobileNet search · ImageNet 1000M FLOPs | Top-1 test error: 19.5 -> 18.7 | Helped |
| select the highest-Zero-Cost-score architecture rather than selecting randomly for mutation (ablation) | RZ-NAS · CIFAR-10 NAS-Bench-201 | test error: 5.4 ± 0.07 -> 5.38 ± 0.12 | No clear effect |
| rephrase the semantic prompt (rephrase 1) (ablation) | RZ-NAS with GraSP · CIFAR-10 | test error: 5.4 ± 0.07 -> 5.5 ± 0.18 | No clear effect |
| rephrase the semantic prompt (rephrase 2) (ablation) | RZ-NAS with GraSP · CIFAR-10 | test error: 5.4 ± 0.07 -> 5.7 ± 0.10 | No clear effect |
Surprising findings
- Random selection of parent architectures performed about as well as selecting the highest zero-cost score parent (e.g., 5.4 vs 5.38 test error on CIFAR-10 with GraSP).
Limitations
- Relies on the proprietary GPT-4o API, with about $75 cost per proxy run.
- Gains depend on the underlying zero-cost proxy and do not consistently beat the strongest proxy on every task.
- Authors note future work is needed to use more cost-efficient open-source models.
Experimental setup
Datasets: NAS-Bench-201 CIFAR-10 CIFAR-100 ImageNet COCO
Benchmarks: NAS-Bench-201 CIFAR-10 DARTS search space CIFAR-100 DARTS search space ImageNet MobileNet search space COCO object detection
Metrics: test accuracy test error Kendall's tau Spearman's rho mAP search cost (GPU days)
Baselines: GPT-NAS GENIUS LLMatic EvoPrompting DARTS DrNAS Zen-NAS ZiCo GraSP Synflow Gradnorm DONNA
Builds on: Zero-Cost NAS proxies ReEvo Reflexion EvoPrompting DARTS
Closest papers by meaning
- Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search (NeurIPS 2026 · similarity 0.91)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.89)
- Design Principle Transfer in Neural Architecture Search via Large Language Models (AAAI 2025 · similarity 0.88)
- Towards Neural Architecture Search through Hierarchical Generative Modeling (ICML 2024 · similarity 0.88)
- Puzzle: Distillation-Based NAS for Inference-Optimized LLMs (ICML 2025 · similarity 0.88)
- Insights from the Use of Previously Unseen Neural Architecture Search Datasets (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.