Research map / Architectures and efficiency / RNNs and normalization
Moonshine: Distilling with Cheap Convolutions
Elliot J. Crowley, Gavin Gray, Amos J. Storkey
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard blocks with G(N/8) blocks and use attention transfer | WRN-40-2 · CIFAR-10 | error: 5.06 vs. 5.66 | Improved |
| Replace standard blocks with BG(2,2) blocks and use attention transfer | WRN-40-2 · CIFAR-10 | error: 5.57% vs. 5.66% | Improved |
| Use attention transfer rather than train the G(2) student from scratch (ablation) | WRN-40-2 G(2) · CIFAR-10 | error: 5.30 -> 4.87 | Improved |
| Use attention transfer rather than knowledge distillation (ablation) | WRN-40-2 G(2) · CIFAR-10 | error: 5.37 -> 4.87 | Improved |
| Increase grouped-convolution groups from N/4 to N (ablation) | WRN-40-2 G blocks · CIFAR-10 | error: 5.31 -> 6.57 | Hurt |
| Replace standard blocks with G(4) blocks and use attention transfer | ResNet-34 · ImageNet validation | top-5 error: 8.62 -> 8.43 | Improved |
| Replace standard blocks with G(N) blocks and use attention transfer rather than train from scratch (ablation) | ResNet-34 · ImageNet validation | top-5 error: 12.26 -> 10.66 | Improved |
| Apply attention transfer to the reduced-width ResNet-18-0.5 (ablation) | ResNet-18-0.5 · ImageNet validation | top-5 error: 15.01 -> 15.02 | Hurt |
| Replace standard blocks with G(N) blocks and use attention transfer rather than train from scratch (ablation) | ERFNet · Cityscapes validation | IoU accuracy: 65.29 -> 68.11 | Improved |
| Replace ResNet-34 blocks with G(N) instead of shrinking to ResNet-18-0.5 at a similar parameter budget | ImageNet student models · ImageNet validation | top-5 error: 10.66% vs. 15.02% | Improved |
Closest papers by meaning
- Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks (NeurIPS 2022 · similarity 0.88)
- Learning Student Networks in the Wild (CVPR 2021 · similarity 0.88)
- Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts (NeurIPS 2020 · similarity 0.88)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.87)
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.87)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
