Research map / Architectures and efficiency / Neural architecture search

Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach

CVPR 2024 · Architectures and efficiency › Neural architecture search · approach: Bi-level differentiable NAS · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Supernet Shifting, which interleaves evolutionary architecture search with per-iteration supernet fine-tuning on top-sampled architectures to improve local order-preserving and enable supernet transfer to new datasets.

Problem

Two-stage one-shot NAS supernets rank top architectures poorly (low local order-preserving, Kendall's tau 0.17 for top-10), and pre-trained supernets are not reusable for new datasets.

Approach

Accumulate training loss for each evaluated architecture during evolutionary search and update the supernet every iteration, so the supernet shifts toward frequently sampled top architectures while keeping the uniform-sampling-trained global ranking; transfer is done by reusing the feature encoder and training a new classifier head during search.

Key results

• On ImageNet-1K, the searched architecture reaches 74.28% top-1 at 318M FLOPs versus 74.01% for SPOS at 323M FLOPs.
• Supernet shifting raises SPOS's ImageNet-1K top-1 by about 0.35% across ShuffleNet-V2 and by up to 1.01% for FairNAS in MobileNet-V2 search space (Table 4).
• Transferring a pre-trained supernet to ImageNet-100 cuts search time from 48-50 GPU hours to 6 while reaching 85.83% top-1 at 299M FLOPs.

Measured effects reported in the paper

Surprising findings

• Global order-preserving of the uniformly trained supernet is already good, but local Kendall's tau among top-10 architectures is only about 0.17 before retraining.

Limitations

• Shifting depends heavily on the sampling behavior of evolutionary search, which has no explicit sampling distribution.
• Mutation makes the sampling distribution uncontrollable.
• Fairness and effectiveness of comparing architectures in a continually changing supernet is not verified.

Experimental setup

Datasets: ImageNet-1K ImageNet-100 CIFAR-100

Benchmarks: ImageNet-1K ImageNet-100 CIFAR-100

Metrics: top-1 accuracy top-5 accuracy FLOPs Kendall's tau GPU hours

Baselines: Random select OneShot SPOS FairNAS GreedyNAS AttentiveNAS DARTS

Builds on: SPOS FairNAS DARTS OneShot NAS

Closest papers by meaning

• SUMNAS: Supernet with Unbiased Meta-Features for Neural Architecture Search (ICLR 2022 · similarity 0.93)
• Neural Architecture Search as Sparse Supernet (AAAI 2021 · similarity 0.91)
• Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.91)
• Rethinking Architecture Selection in Differentiable NAS (ICLR 2021 · similarity 0.91)
• Subnet-Aware Dynamic Supernet Training for Neural Architecture Search (CVPR 2025 · similarity 0.91)
• HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning (AAAI 2025 · similarity 0.90)