Research map / Architectures and efficiency / Neural architecture search
Insights from the Use of Previously Unseen Neural Architecture Search Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces eight new "unseen" datasets from NAS Challenges and benchmarks CNNs (ResNet-18, AlexNet, VGG16, ConvNext, MnasNet, DenseNet, ResNeXt), NAS methods (PC-DARTS, DrNAS, Bonsai-Net) and random search on them.
Problem
Existing NAS work is evaluated mostly on a small set of benchmark datasets (CIFAR-10, ImageNet) that are not representative of real-world problems, so NAS generalisation to unseen datasets is untested.
Approach
Create eight bespoke datasets, split into Type-1 (solvable by an expert program) and Type-2 (near-impossible for humans at face value), withheld from participants to test NAS generalisation to unseen data, and run challenge submissions alongside standard CNN and NAS baselines.
Key results
- On AddNIST, Bonsai-Net reached 97.91% test accuracy versus 92.08% for ResNet-18 and 5% random chance.
- On CIFARTile, PC-DARTS reached 92.28% versus 51.28% for the best CNN (AlexNet/DenseNet range) and 24.43% for VGG16.
- Random search in the DARTS space outperformed PC-DARTS and DrNAS on AddNIST, Chesseract and Isabella, supporting prior findings that these search spaces rarely beat random search.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Bonsai-Net instead of the strongest CNN baseline | NAS on AddNIST · AddNIST | accuracy: 94.87% -> 97.91% | Helped |
| Use PC-DARTS instead of the strongest CNN baseline | NAS on Language · Language | accuracy: 93.97% -> 90.12% | Hurt |
| Use DrNAS instead of the strongest CNN baseline | NAS on MultNIST · MultNIST | accuracy: 94.01% -> 98.10% | Helped |
| Use PC-DARTS instead of the strongest CNN baseline | NAS on CIFARTile · CIFARTile | accuracy: 51.28% -> 92.28% | Helped |
| Use PC-DARTS instead of the strongest CNN baseline | NAS on Gutenberg · Gutenberg | accuracy: 45.53% -> 49.12% | Helped |
| Use DARTS instead of the strongest CNN baseline | NAS on Isabella · Isabella | accuracy: 63.27% -> 66.35% | Helped |
| Use DrNAS instead of the strongest CNN baseline | NAS on GeoClassing · GeoClassing | accuracy: 94.21% -> 96.03% | Helped |
| Use Bonsai-Net instead of the strongest CNN baseline | NAS on Chesseract · Chesseract | accuracy: 59.60% -> 60.76% | Helped |
| Use Bonsai-Net instead of random search | NAS on Chesseract · Chesseract | accuracy: 68.83% -> 60.76% | Hurt |
Surprising findings
- Random search often matched or beat DARTS-based NAS methods, and Bonsai-Net was significantly beaten by random search on Chesseract.
- ConvNext performed poorly across most datasets, e.g. 38.06% on AddNIST versus over 90% for most other CNNs.
Limitations
- Baseline CNNs were not run at full depth and no hyperparameter optimisation was performed.
- Competition participants had only 24 hours to evaluate across three datasets.
- Isabella dataset cannot be fully shared due to museum licence; only a conversion script is released.
Experimental setup
Datasets: MNIST CIFAR-10 BigEarthNet ASPELL dictionaries Project Gutenberg texts Isabella Stewart Gardner Museum recordings Chess games from eight grandmasters
Benchmarks: AddNIST Language MultNIST CIFARTile Gutenberg Isabella GeoClassing Chesseract
Metrics: test accuracy
Baselines: ResNet-18 AlexNet VGG16 ConvNext MnasNet DenseNet ResNeXt PC-DARTS DrNAS Bonsai-Net random search
Builds on: DARTS PC-DARTS DrNAS Bonsai-Net NAS-Bench-201
Closest papers by meaning
- NAS evaluation is frustratingly hard (ICLR 2020 · similarity 0.93)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.92)
- NAS-Bench-101: Towards Reproducible Neural Architecture Search (ICML 2019 · similarity 0.92)
- Evaluating The Search Phase of Neural Architecture Search (ICLR 2020 · similarity 0.92)
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective (ICLR 2021 · similarity 0.91)
- Encodings for Prediction-based Neural Architecture Search (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.