Research map / Architectures and efficiency / Neural architecture search
Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a single tabular lookup with a GIN trained on one seed | NAS-Bench-101 architecture performance estimation · NAS-Bench-101 | MAE: 4.534$e$-3 -> 3.446$\mathbf{e}$-3 | Helped |
| Replace one-seed tabular prediction with a GIN surrogate | NAS-Bench-101 architecture ranking · NAS-Bench-101 | Kendall tau: 0.83 -> 0.87 | Helped |
| Use LGBoost rather than XGBoost for DARTS performance regression | SNB-DARTS · SNB-DARTS test split | R²: 0.832 -> 0.892 | Helped |
| Use XGBoost rather than LGBoost for DARTS rank prediction | SNB-DARTS · SNB-DARTS test split | sKT: 0.816 -> 0.817 | Helped |
| Use a GIN ensemble to estimate noisy architecture performance instead of one table entry | SNB-DARTS · 500 architectures evaluated with 5 seeds | MAE: 1.38e-3 -> 1.13e-3 | Helped |
| Use a GIN ensemble rather than an LGB ensemble to model predictive uncertainty | SNB-DARTS · 500 architectures evaluated with 5 seeds | KL divergence: 68.9 -> 16.4 | Helped |
| Transfer tuned DARTS XGBoost hyperparameters to FBNet | SNB-FBNet · SNB-FBNet test split | R²: 0.84 R² | Helped |
| Train the surrogate only on randomly sampled architectures rather than data collected from multiple optimizers (ablation) | SNB-DARTS · DARTS search trajectories | trajectory fidelity | Mixed |
| Train the surrogate only on architectures with 92% and above performance (ablation) | SNB-DARTS · DARTS search trajectories | trajectory fidelity | Mixed |
| Add a small number of skip connections to the normal cell (ablation) | DARTS architectures · DARTS normal cell | validation performance | Helped |
| Add many parameter-free operations to a cell (ablation) | DARTS architectures · DARTS normal and reduction cells | validation performance | Hurt |
| Extend local search from its initial short-budget phase to a much longer run | Local Search on DARTS · DARTS search | validation performance: 11.8 GPU days; $10^7$ seconds | Helped |
Closest papers by meaning
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search (ICLR 2020 · similarity 0.94)
- NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy (ICLR 2022 · similarity 0.93)
- NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks (NeurIPS 2022 · similarity 0.93)
- NAS evaluation is frustratingly hard (ICLR 2020 · similarity 0.92)
- NAS-Bench-101: Towards Reproducible Neural Architecture Search (ICML 2019 · similarity 0.92)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.