Research map / Architectures and efficiency / Neural architecture search
einspace: Searching for Neural Architectures from Fundamental Operations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces einspace, an expressive NAS search space defined by a parameterised probabilistic context-free grammar over fundamental tensor operations, and searches it with random search and regularised evolution on Unseen NAS datasets.
Problem
Existing NAS search spaces are too narrow (mostly convolutional cells), so NAS rarely yields fundamental design shifts such as convolution-to-transformer changes; random search is often a strong baseline in them.
Approach
A parameterised PCFG whose production rules compose fundamental operations (branching, aggregation, routing, computation) with shape/mode/branching parameters ensuring validity, plus probabilities tuned to a sub-critical branching rate, so the space unifies ConvNets, transformers and MLP-only architectures and can be seeded with existing architectures.
Key results
- Regularised evolution seeded with a mix of ResNet18, WRN16-4, ViT and MLP-Mixer (RE(Mix)) reaches 97.92% on Language vs 92.16% for ResNet18, a 5.76% gain.
- Random search in einspace averages 59.76% accuracy versus 80.41% for DARTS random search, showing random search underperforms in the expressive space.
- Seeding evolution with ResNet18 (RE(RN18)) improves the initial architecture on every Unseen NAS task.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize evolutionary search with ResNet18 | einspace regularised evolution · Unseen NAS tasks | accuracy | Helped |
| initialize evolutionary search with a mix of ResNet18, WRN16-4, ViT and MLP-Mixer | einspace regularised evolution · Unseen NAS tasks | accuracy | Helped |
| initialize evolutionary search with a mix of existing architectures | einspace regularised evolution · Language | accuracy: +5.76% | Helped |
Surprising findings
- Random search performs poorly in the expressive einspace, unlike in narrow NAS spaces where it is a strong baseline.
- Random search in einspace outperforms the baseline NAS methods on Chesseract (68.83% vs 60.76% for Bonsai-Net baseline; einspace random search 61.46%).
Limitations
- Search is extremely large and unbounded in depth and width, making one-shot methods like DARTS or ENAS hard to apply.
- Multiple derivation trees can map to the same architecture, and mutations can produce redundant identity-equivalent components.
- Recurrent computation and state-space models such as Mamba are not representable in the current space.
Experimental setup
Datasets: Unseen NAS datasets (AddNIST, Language, MultNIST, CIFARTile, Gutenberg, Isabella, GeoClassing, Chesseract) NASBench360
Benchmarks: Unseen NAS NASBench360
Metrics: test accuracy average accuracy average rank
Baselines: ResNet18 PC-DARTS DrNAS Bonsai-Net hNASBench201 DARTS random search
Builds on: DARTS hNASBench201 AutoML-Zero regularised evolution
Closest papers by meaning
- Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars (NeurIPS 2023 · similarity 0.93)
- Insights from the Use of Previously Unseen Neural Architecture Search Datasets (CVPR 2024 · similarity 0.92)
- Understanding Architectures Learnt by Cell-based Neural Architecture Search (ICLR 2020 · similarity 0.92)
- Towards Neural Architecture Search through Hierarchical Generative Modeling (ICML 2024 · similarity 0.91)
- Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.91)
- Encodings for Prediction-based Neural Architecture Search (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.