Research map / Architectures and efficiency / CNN design
Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Abdelrahman Mohamed, Matthai Philipose, Matt Richardson, Rich Caruana
This paper tried a common change: Add dropout
Add dropout during student mimic training · CIFAR-10 · accuracy Hurt
What 795 accepted papers report for this change
- 549 improved
- 87 no clear effect
- 12 mixed
- 147 hurt
- Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference ICML 2026 · apply layer dropout to whole transformer blocks instead of attention and FFN sub-blocks separately · 20 TPP val loss, 271M · val loss: 2.377 -> 2.367Improved
- Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles ICML 2026 · use MC Dropout · vision and SST-2 · accuracy: 86.3 -> 76.6 on Flowers102Hurt
- Motion Planning in Compressed Representation Spaces ICML 2026 · add nested dropout during training · WOMD guided generation · goal FDE: 4.145 -> 1.104Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a student on ensemble logits rather than hard labels | 10M-parameter CNNs with 1–4 convolutional layers · CIFAR-10 | accuracy: 84.6% -> 87.3%; 88.9% -> 90.0%; 91.2% -> 91.9%; 91.75% -> 92.6% | Improved |
| Add convolution instead of using a fully connected MLP | 31.6M-parameter CIFAR-10 students · CIFAR-10 | accuracy: 70.2% -> 87.7% | Improved |
| Increase convolutional depth from 1 to 4 layers | 31.6M-parameter student CNN · CIFAR-10 | accuracy: 87.7% -> 92.6% | Improved |
| Increase student CNN parameter budget from 1M to 31.6M | Student CNNs · CIFAR-10 | accuracy: 84.5% -> 87.7%; 87.9% -> 90.3%; 90.7% -> 92.3%; 91.3% -> 92.6% | Improved |
| Increase MLP depth beyond 2–3 hidden layers without adding convolution | Student MLPs · CIFAR-10 | accuracy: 74.3% -> 71.5% | Hurt |
| Add dropout during student mimic training (ablation) | Student mimic networks · CIFAR-10 | accuracy | Hurt |
| Add weight decay during student mimic training (ablation) | Student mimic networks · CIFAR-10 | accuracy | Hurt |
| Use distillation rather than hard labels for shallow CNN students (ablation) | Student CNNs with 1–4 convolutional layers · CIFAR-10 | accuracy: 2.7%; 0.8% | Improved |
Closest papers by meaning
- Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (ICCV 2019 · similarity 0.87)
- The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes (NeurIPS 2022 · similarity 0.86)
- Greedy Layerwise Learning Can Scale To ImageNet (ICML 2019 · similarity 0.86)
- Deep Convolutional Networks as shallow Gaussian Processes (ICLR 2019 · similarity 0.85)
- How to Start Training: The Effect of Initialization and Architecture (NeurIPS 2018 · similarity 0.85)
- Layer Compression of Deep Networks with Straight Flows (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
