Research map / Architectures and efficiency / Neural architecture search
Differentiable Model Scaling using Differentiable Topk
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Differentiable Model Scaling (DMS), which searches width and depth of networks via a fully differentiable top-k operator with importance normalization and a learnable pruning-ratio threshold, under a resource-constraint loss.
Problem
Efficiently searching optimal width and depth of neural networks; prior gradient-based NAS methods model structural hyperparameters indirectly or with non-differentiable gradient estimates, and stochastic/one-shot/multi-shot NAS is very costly.
Approach
A differentiable top-k operator that uses a single learnable pruning-ratio threshold over rank-normalized element importance (uniform in [0,1]) and a sigmoid soft mask, so the number of channels/blocks/heads is directly and smoothly optimized with task loss plus a resource-constraint loss.
Key results
- On ImageNet, DMS-searched EfficientNet-B0 variant reaches 78.5% top-1 vs 77.1% for EfficientNet-B0 and outperforms ZiCo by 1.3% at 0.4 GPU days search cost.
- On COCO, DMS-Yolo-v8-n reaches 39.4 mAP vs 37.4 for Yolo-v8-n (+2.0).
- On pruned Llama-7B (5.47B params), DMS gets Wikitext2 perplexity 17.13 vs 17.39 for LLM-Pruner and higher zero-shot accuracy on BoolQ (75.23 vs 66.79).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use differentiable topk instead of gradient-estimate topk | DMS on ResNet-50 · ImageNet | Top-1 (%): 61.6 -> 70.7 | Helped |
| Replace EA search with DMS | OFA supernet at 0.45G MACs · ImageNet | search cost (GPU hours): 0 + 1.1 -> 0 + 0.05 | Helped |
| Search EfficientNet-B0 width and depth with DMS | EfficientNet-B0 · ImageNet | Top-1 (%): 77.1 -> 78.5 | Helped |
| Search EfficientNet-B1 width and depth with DMS | EfficientNet-B1 · ImageNet | Top-1 (%): 79.1 -> 80.0 | Helped |
| Search EfficientNet-B2 width and depth with DMS | EfficientNet-B2 · ImageNet | Top-1 (%): 80.1 -> 81.1 | Helped |
| Apply DMS structure search | Yolo-v8-n · COCO | mAP (%): 37.4 -> 39.4 | Helped |
| Prune and finetune with DMS instead of LLM-Pruner | Llama-7B pruned to 5.47B · Wikitext2 | perplexity: 17.39 -> 17.13 | Helped |
| Remove importance normalization (ablation) | DMS differentiable topk on ResNet-50 · ImageNet | Top-1 (%): 70.7 -> 0.1 | Hurt |
| Add moving average to Taylor importance (ablation) | DMSnp on ResNet-50 · ImageNet, 1G MACs | Top-1 (%): 72.5 -> 73.1 | Helped |
| Increase supernet size from ResNet-50 to ResNet-152 without pretraining (ablation) | DMSnp · ImageNet, 1G MACs | Top-1 (%): 73.1 -> 74.6 | Helped |
| Reduce search time from 10 to 20 epochs (ablation) | DMSnp on ResNet-50 · ImageNet, 1G MACs | Top-1 (%): 73.1 -> 72.8 | Hurt |
Surprising findings
- Without importance normalization, the differentiable top-k drops to 0.1% top-1, showing the normalization is critical.
- Searching a randomly initialized ResNet-152 supernet beats searching a pretrained ResNet-50 supernet (74.6 vs 73.8) at only ~3x vs ~10x cost, so larger supernets are more efficient than pretraining.
Limitations
- Llama experiment uses prune-and-finetune on Alpaca rather than training an LLM from scratch due to resource constraints.
- Main NAS comparisons rely on search-cost estimates taken from other papers, some of which are rough (e.g., 'Low'/'High' levels).
Experimental setup
Datasets: ImageNet COCO Alpaca WikiText2 PTB BoolQ WinoGrande ARC-e ARC-c
Benchmarks: ImageNet COCO WikiText2 PTB BoolQ WinoGrande ARC-e ARC-c
Metrics: top-1 accuracy mAP perplexity zero-shot accuracy MACs search cost (GPU days)
Baselines: ZiCo Zen-score JointPruning ScaleNet ModelAmplification EfficientNet LLM-Pruner LFPC GReg-2 CC TPP evolutionary algorithm
Builds on: Taylor importance OFA EfficientNet MMRazor
Closest papers by meaning
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (ICML 2019 · similarity 0.86)
- Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasets (NeurIPS 2025 · similarity 0.85)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.84)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.84)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.84)
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.