AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Moonshine: Distilling with Cheap Convolutions

Elliot J. Crowley, Gavin Gray, Amos J. Storkey

NeurIPS 2018 · Architectures and efficiency › RNNs and normalization · approach: Efficient network compression and training · 95 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace standard blocks with G(N/8) blocks and use attention transferWRN-40-2 · CIFAR-10error: 5.06 vs. 5.66Improved
Replace standard blocks with BG(2,2) blocks and use attention transferWRN-40-2 · CIFAR-10error: 5.57% vs. 5.66%Improved
Use attention transfer rather than train the G(2) student from scratch (ablation)WRN-40-2 G(2) · CIFAR-10error: 5.30 -> 4.87Improved
Use attention transfer rather than knowledge distillation (ablation)WRN-40-2 G(2) · CIFAR-10error: 5.37 -> 4.87Improved
Increase grouped-convolution groups from N/4 to N (ablation)WRN-40-2 G blocks · CIFAR-10error: 5.31 -> 6.57Hurt
Replace standard blocks with G(4) blocks and use attention transferResNet-34 · ImageNet validationtop-5 error: 8.62 -> 8.43Improved
Replace standard blocks with G(N) blocks and use attention transfer rather than train from scratch (ablation)ResNet-34 · ImageNet validationtop-5 error: 12.26 -> 10.66Improved
Apply attention transfer to the reduced-width ResNet-18-0.5 (ablation)ResNet-18-0.5 · ImageNet validationtop-5 error: 15.01 -> 15.02Hurt
Replace standard blocks with G(N) blocks and use attention transfer rather than train from scratch (ablation)ERFNet · Cityscapes validationIoU accuracy: 65.29 -> 68.11Improved
Replace ResNet-34 blocks with G(N) instead of shrinking to ResNet-18-0.5 at a similar parameter budgetImageNet student models · ImageNet validationtop-5 error: 10.66% vs. 15.02%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy