Research map / Architectures and efficiency / Neural architecture search
Multi-objective Differentiable Neural Architecture Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MODNAS, a differentiable multi-objective NAS that uses a hypernetwork conditioned on preference vectors and device embeddings, trained with multiple gradient descent, to profile Pareto fronts across many hardware devices in a single search run.
Problem
Finding diverse Pareto-optimal architectures trading off accuracy and hardware metrics (latency, energy) across multiple devices is expensive, since existing constrained NAS methods require a separate search per constraint and per device.
Approach
A MetaHypernetwork maps a scalarization (preference vector) and a device embedding to architecture distributions, sampled via ReinMax differentiable sampling in a weight-shared supernetwork and a meta-predictor for hardware objectives; the hypernetwork is updated with multiple gradient descent over devices and preferences, enabling zero-shot transfer to new devices.
Key results
- On NAS-Bench-201 across 19 devices (13 search, 6 test), MODNAS achieves higher hypervolume than all baselines on every device while using a single search run versus 19 separate runs for baselines.
- On HAT WMT'14 En-De, MODNAS evaluates only 24 generated architectures and outperforms baselines that receive 2.5x runtime budget (1300 to 6000 evaluations).
- With 3 objectives (error, latency, energy) on FPGA, MODNAS scales without extra search cost or hyperparameter tuning and achieves hypervolume close to the global optimum front on NAS-Bench-201.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MODNAS instead of the multi-objective baselines | NAS-Bench-201 on CIFAR-10 across 19 devices · NAS-Bench-201, CIFAR-10 | hypervolume | Helped |
| Use MGD updates instead of mean, sequential, or single-task MC-sample gradient updates (ablation) | MODNAS MetaHypernetwork · NAS-Bench-201 | hypervolume | Helped |
| Use ReinMax instead of the STGS estimator (ablation) | MODNAS Architect · Initial experiments | performance | Helped |
| Disable the cosine penalty term (ablation) | MODNAS in convolutional search spaces · Convolutional search spaces | performance | No clear effect |
| Use MODNAS instead of the multi-objective baselines | MODNAS · HAT, WMT'14 En-De | hypervolume | Helped |
| Use MODNAS instead of the baselines | MODNAS · MobileNetV3, ImageNet-1k | hypervolume | Helped |
| Use Zico zero-cost proxy for performance estimation instead of actual accuracy (ablation) | NSGA-II · MobileNetV3, two devices | hypervolume | Hurt |
| Use MODNAS instead of the multi-objective baselines | MODNAS · HW-GPT-Bench, GPT-S | Pareto-front quality | Mixed |
| Increase the latency constraint to 1, leaving only cross-entropy optimized (ablation) | MODNAS · NAS-Bench-201 | hypervolume | Hurt |
| Increase the latency constraint to 1, leaving only cross-entropy optimized (ablation) | MODNAS · NAS-Bench-201 | architecture performance | Helped |
| Use MODNAS-SoTL to select candidates before training Pareto-set architectures from scratch (ablation) | MODNAS-SoTL · NAS-Bench-201 | evaluation cost | Helped |
Surprising findings
- MetaD2A+HELP, despite multiple constrained search runs, prioritizes performance over diversity and yields less varied solutions.
Limitations
- Gradient-based search may favor one objective, leading to local optima that hinder exploration of the full Pareto front.
- Relies on differentiable proxies (supernetwork accuracy, meta-predictors) that may not align with ground-truth values.
Experimental setup
Datasets: NAS-Bench-201 CIFAR-10 ImageNet-1k WMT'14 En-De OpenWebText HW-NAS-Bench
Benchmarks: NAS-Bench-201 MobileNetV3 (OFA) ImageNet-1k HAT WMT'14 En-De HW-GPT-Bench GPT-S
Metrics: hypervolume generational distance inverse generational distance BLEU perplexity
Baselines: Random Search Local Search MO-ASHA MO-RE NSGA-II CMA-ES LaMOO RS-BO qEHVI MetaD2A + HELP LEMONADE RHPN
Builds on: DARTS GDAS ReinMax HELP HW-GPT-Bench
Closest papers by meaning
- One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking (CVPR 2021 · similarity 0.88)
- Per-Architecture Training-Free Metric Optimization for Neural Architecture Search (NeurIPS 2025 · similarity 0.88)
- MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation (CVPR 2020 · similarity 0.87)
- MOTE-NAS: Multi-Objective Training-based Estimate for Efficient Neural Architecture Search (NeurIPS 2024 · similarity 0.87)
- EG-NAS: Neural Architecture Search with Fast Evolutionary Exploration (AAAI 2024 · similarity 0.87)
- RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.