Research map / Architectures and efficiency / Neural architecture search
Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IBFS, a training-free zero-cost NAS method that ranks architectures for few-shot learning using an information-bottleneck-inspired entropy proxy over Jacobian eigenvalues, and finds meta-architectures for MAML/Reptile-style FSL without training.
Problem
Neural architecture search for few-shot learning is either expensive (searching from scratch or meta-learning NAS costing 100+ GPU days) or reuses architectures from other tasks, which can be sub-optimal.
Approach
Derives that MAML's global convergence can be guaranteed by considering only the first-order (inner-loss) landscape, then scores untrained architectures with an entropy-based information-bottleneck proxy computed from eigenvalues of a batch Jacobian matrix, so architecture ranking needs no training.
Key results
- On NAS-Bench-201 IBFS reaches 94.37% test on CIFAR-10 and 73.09% on CIFAR-100 with a 3.82 s search, versus NASWOT's 91.53% and 66.63%.
- On mini-ImageNet 5-way, IBFS achieves 64.55% (1-shot) and 81.52% (5-shot), above MetaNTK-NAS (64.26%, 80.35%) at 0.1 hr search versus 1.92 hr.
- On ImageNet1k (DARTS space, searched on CIFAR-10), IBFS reports 76.7% top-1 at 0.0042 GPU-days.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IBFS instead of IS-DARTS | NAS-Bench-201 · CIFAR-10 test | accuracy: 94.36±0.00 -> 94.37 ± 0.34 | Helped |
| Use IBFS instead of IS-DARTS | NAS-Bench-201 · CIFAR-100 test | accuracy: 73.31±0.00 -> 73.09 ± 2.08 | Hurt |
| Use IBFS instead of IS-DARTS | NAS-Bench-201 · ImageNet-16-120 test | accuracy: 46.34±0.00 -> 46.33 ± 1.27 | Hurt |
| Use IBFS(C10) instead of WeakNAS | DARTS search space · ImageNet1k | Top-1 accuracy: 76.5 -> 76.7 ↑(0.2) | Helped |
| Use IBFS instead of MetaDiff | few-shot classifier · mini-ImageNet 5-way 1-shot | accuracy: 64.99±0.77 -> 64.55±0.02 | Hurt |
| Use IBFS instead of MetaDiff | few-shot classifier · mini-ImageNet 5-way 5-shot | accuracy: 81.21 ±0.56 -> 81.52±0.08 | Helped |
| Use IBFS instead of MetaDiff | few-shot classifier · tiered-ImageNet 5-way 1-shot | accuracy: 72.33±0.92 -> 72.56±0.02 | Helped |
| Use IBFS instead of MetaNAS (retrained) | few-shot classifier · tiered-ImageNet 5-way 5-shot | accuracy: 86.48±0.06 -> 86.73±0.08 | Helped |
| Use IBFS(C10) instead of Auto-Prox | ImageNet architecture search · ImageNet (AutoFormer benchmark) | Top-1 accuracy: 75.6 -> 76.5 | Helped |
Surprising findings
- Kendall's Tau between the entropy proxy and accuracy stays stable across training epochs, suggesting architecture expressivity is measurable at initialization.
Limitations
- Theorem 4.1 relies on borrowed NTK-based results and infinite-width limit assumptions, and the proof is intricate and loosely connected to the proxy.
- Evaluation is limited to CNN backbones on image classification; transformer results are a single AutoFormer comparison.
- Entropy-based proxy motivation is largely empirical with small model samples (five models) for correlation analysis.
Experimental setup
Datasets: NAS-Bench-201 CIFAR-10 CIFAR-100 ImageNet-16-120 mini-ImageNet tiered-ImageNet ImageNet1k
Benchmarks: NAS-Bench-201 DARTS search space ImageNet1k mini-ImageNet 5-way 1/5-shot tiered-ImageNet 5-way 1/5-shot AutoFormer benchmark ImageNet
Metrics: top-1 accuracy Kendall's Tau search cost (s/GPU-days/hr) few-shot accuracy
Baselines: MAML ANIL MetaOptNet RFS AutoMeta MetaNAS MetaNTK-NAS MetaDiff NASWOT SWAP TENAS NASI-ADA
Builds on: MAML Reptile NASWOT MetaNTK-NAS TENAS
Closest papers by meaning
- Global Convergence of MAML and Theory-Inspired Neural Architecture Search for Few-Shot Learning (CVPR 2022 · similarity 0.92)
- Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.92)
- Meta-Learning of Neural Architectures for Few-Shot Learning (CVPR 2020 · similarity 0.91)
- Neural Fine-Tuning Search for Few-Shot Learning (ICLR 2024 · similarity 0.90)
- HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning (AAAI 2025 · similarity 0.89)
- NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.