Research map / Architectures and efficiency / Neural architecture search

Subnet-Aware Dynamic Supernet Training for Neural Architecture Search

CVPR 2025 · Architectures and efficiency › Neural architecture search · approach: One-shot and few-shot supernets · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a subnet-aware dynamic supernet training scheme (complexity-aware LR scheduler plus momentum separation) that plugs into N-shot NAS methods to improve ranking consistency and search results.

Problem

Static supernet training in N-shot NAS treats all subnets identically, causing unfairness toward low-complexity subnets and noisy momentum from diverse subnet gradients, which degrades ranking consistency.

Approach

A complexity-aware LR scheduler (CaLR) sets per-subnet LR decay ratios based on parameter count, slowing decay for high-complexity subnets; momentum separation (MS) assigns separate momentum buffers to clusters of subnets sharing an operation on a chosen edge, while weights stay shared.

Key results

• On the MobileNet space, SPOS-M + Ours reaches 76.7% top-1 vs 76.2% for SPOS-M, and SPOS-L + Ours reaches 77.1% vs 76.6%.
• On NAS-Bench-201 CIFAR-10, SPOS + Ours raises Kendall's Tau from 0.751 to 0.814 and FairNAS + Ours from 0.766 to 0.828.
• Applying the method adds negligible search cost, e.g., peak memory rises from 1875 MB to 1901 MB for SPOS on NAS-Bench-201.

Measured effects reported in the paper

Surprising findings

• Using an opposite-direction LR decay (decay ratio proportional to complexity) degrades CB and C3 relative to the baseline.
• Random clustering for momentum separation gives no gain in Kendall's Tau, while operation-based clustering does.

Limitations

• Not applicable to differentiable NAS methods, which jointly optimize weights and operation coefficients.
• Number of momentum clusters must be tuned; too many clusters (125 or 625) degrade ranking consistency.

Experimental setup

Datasets: ImageNet CIFAR-10 CIFAR-100 ImageNet16-120

Benchmarks: ImageNet (MobileNet space) NAS-Bench-201 CIFAR-10 NAS-Bench-201 CIFAR-100 NAS-Bench-201 ImageNet16-120

Metrics: top-1 accuracy Kendall's Tau Complexity Bias (CB) Complexity-Convergence Correlation (C3) peak memory GPU hours

Baselines: SPOS FairNAS FSNAS GreedyNAS PA&DA ShiftNAS

Builds on: SPOS FairNAS FSNAS NAS-Bench-201

Closest papers by meaning

• Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.93)
• GreedyNAS: Towards Fast One-Shot NAS With Greedy Supernet (CVPR 2020 · similarity 0.92)
• Few-Shot Neural Architecture Search (ICML 2021 · similarity 0.91)
• SUMNAS: Supernet with Unbiased Meta-Features for Neural Architecture Search (ICLR 2022 · similarity 0.91)
• HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning (AAAI 2025 · similarity 0.91)
• Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach (CVPR 2024 · similarity 0.91)