AI Paper Atlas

Research map / Architectures and efficiency / Neural architecture search

Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces

Bert Moons, Parham Noorzad, Andrii Skliar, Giovanni Mariani, Dushyant Mehta, Chris Lott, Tijmen Blankevoort

ICCV 2021 · Architectures and efficiency › Neural architecture search · approach: Efficient NAS search strategies · 33 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use the DONNA linear accuracy predictor instead of the DNA ranking modelDNA ranking · DONNA search spaceKendall-Tau: 0.75 -> 0.91Improved
initialize finetuning from blockwise distillation weights instead of training from scratchDONNA architectures · ImageNettraining epochs: 450 -> 50Improved
increase architecture library from 10 to 30 targets (ablation)DONNA predictor · DONNA search spaceKendall-Tau: 0.87 -> 0.9Improved
increase architecture library from 10 to 40 targets (ablation)DONNA predictor · MobileNetV3 (1.2x) search spaceKendall-Tau: 0.79 -> 0.82Improved
use NSR quality metric instead of network-level top-1 accuracy (ablation)DONNA predictor · DONNA search spaceMSE (top-1 %): 0.44 -> 0.19Improved
use Ridge regression instead of LARS as predictor (ablation)DONNA predictor · DONNA search spaceKendall-Tau: 0.16 -> 0.91Improved
use Ridge regression instead of SVM as predictor (ablation)DONNA predictor · DONNA search spaceMSE (top-1 %): 1.84 -> 0.15Improved
allow optimal placement of SE/Swish attention in the search space (E vs D) (ablation)DONNA search space · V100 latency explorationaccuracy: +1% at 26msImproved
add SE-attention optimally (C vs D) in the search space (ablation)DONNA search space · V100 latency explorationlatency: 25% speedup at 76% accuracyImproved
apply FP-trained predictor to INT8-quantized networks (ablation)DONNA predictor · MobileNetV3 (1.2x)Kendall-Tau: 0.85 -> 0.86No clear effect
use OFA progressive-shrinking weights instead of from-scratch weights for the detection backbone (ablation)OFA backbone · COCO EfficientDet-D0mAP: +1.4% top-1 -> +0.1% mAPNo clear effect
search DONNA backbones directly on COCO instead of transferring from ImageNetDONNA backbones · COCO EfficientDet-D0mAPImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy