Research map / Architectures and efficiency / Neural architecture search
Design Principle Transfer in Neural Architecture Search via Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAPT, which uses GPT-4 to reason linguistic design principles from good architectures and adapts them to prune NAS search spaces, paired with regularized evolution (REA), to speed up transferable NAS on NAS-Bench-201, TransNAS-Bench-101 and DARTS.
Problem
Transferable NAS still searches very large spaces and needs many architecture evaluations; existing transfer methods are entangled with specific search techniques and principle learning is costly and expert-dependent.
Approach
Transfer design principles (natural-language descriptions of how structural components affect performance) learned by an LLM from established architectures in code form to refine the search space, and adapt these principles iteratively using newly found architectures; the transfer is decoupled from the NAS search method.
Key results
- On NAS201, LAPT-REA reaches 94.36% on CIFAR-10 and 73.45% on CIFAR-100 with 4.9 and 8.6 trained architectures, versus REA's >500 architectures.
- On TransNAS-Bench-101, LAPT-REA attains the best average rank among TNAS methods except Arch-Graph (12.3 vs 12.2) and beats BONAS on all 6 tasks.
- On DARTS CIFAR-10, LAPT-REA reaches 97.35% in 0.1 GPU days; on ImageNet it reaches 75.1% top-1 in 2.0 GPU days.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use design principle transfer with REA instead of standalone REA | NAS201 search · CIFAR-100 | accuracy: 71.84 -> 73.45 | Helped |
| use design principle transfer with REA instead of standalone REA | NAS201 search · CIFAR-10 | accuracy: 93.92 -> 94.36 | Helped |
| refine the DARTs search space using transferred design principles | REA on DARTs · CIFAR-10 | GPU days: 4 -> 0.1 | Helped |
| refine the DARTs search space using transferred design principles | REA on DARTs · CIFAR-10 | test accuracy: 97.61 -> 97.35 | Mixed |
| refine the DARTs search space using transferred design principles | REA on DARTs · ImageNet | GPU days: 4.2 -> 2.0 | Helped |
| omit principle adaptation (ablation) | LAPT-REA on Trans101 · Trans101 tasks | model rank | Mixed |
| remove knowledge transfer from prior tasks (ablation) | LAPT-REA on Trans101 · Trans101 tasks | model rank | Hurt |
| use GPT-4 instead of GPT-3.5 for principle learning and adaptation | LAPT on Trans101 · Trans101 tasks | average model rank: 43.2 -> 12.3 | Helped |
| use GPT-4o instead of GPT-3.5 for principle learning and adaptation | LAPT on Trans101 · Trans101 tasks | average model rank: 43.2 -> 12.1 | Helped |
Surprising findings
- Ablation on the LLM showed GPT-3.5 performed far worse than GPT-4 and GPT-4o, indicating strong dependence on LLM capability.
Limitations
- Source and target tasks must share the same search space and high task similarity, as the conclusion notes.
- Performance depends on LLM capability (GPT-3.5 much weaker than GPT-4).
- Several comparisons are only marginal; LAPT-REA is slightly below SOTA on CIFAR-10 accuracy in DARTS.
Experimental setup
Datasets: NAS-Bench-201 TransNAS-Bench-101 DARTS CIFAR-10 CIFAR-100 ImageNet Jigsaw
Benchmarks: NAS-Bench-201 TransNAS-Bench-101 DARTS CIFAR-10 DARTS ImageNet
Metrics: test accuracy number of architectures evaluated GPU days average rank mode rank eEDF search-space quality
Baselines: REA HEBO MetaD2A TNAS-BO BONAS WeakNAS-T Arch-zero Arch-Graph SNAS DARTS GibbsNAS MSNAS
Builds on: regularized evolution transferable NAS chain-of-thought prompting
Closest papers by meaning
- Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search (NeurIPS 2026 · similarity 0.90)
- RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy (ICML 2025 · similarity 0.88)
- Improving Differentiable Neural Architecture Search by Encouraging Transferability (ICLR 2023 · similarity 0.87)
- EG-NAS: Neural Architecture Search with Fast Evolutionary Exploration (AAAI 2024 · similarity 0.85)
- Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach (CVPR 2024 · similarity 0.85)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.