Research map / Architectures and efficiency / RNNs and normalization
Dropout distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Dropout distillation (training same-architecture student to match MC-averaged dropout predictor) instead of standard dropout inference | CIFAR10 Quick, CIFAR10 NiN, CIFAR10 AllConv, CIFAR100 AllConv, CIFAR100 NiN · CIFAR/MNIST test error | test error %: e.g. Quick 18.15 -> 17.20; NiN CIFAR10 11.16 -> 11.14; AllConv CIFAR10 11.20 -> 10.80; CIFAR100 AllConv 32.46 -> 32.07; | Mixed |
| Dropout distillation instead of MC dropout (100 samples) | All networks in Table 1 · CIFAR/MNIST test error | test error %: MC better in most cases, e.g. Quick 16.78 vs 17.20; AllConv CIFAR10 10.60 vs 10.80 | Hurt |
| Distillation target set S drawn from original training samples, perturbed training samples, or test samples (ablation) | Dropout distillation on CIFAR networks · CIFAR10/CIFAR100 test error | test error %: gap of at most ≈0.3% across strategies | No clear effect |
| Compress distilled student by shrinking only the penultimate FC layer to ~40% of parameters (ablation) | CIFAR10 Quick · CIFAR10 test error | test error %: ≈18.02% vs standard dropout 18.15% with all parameters | Helped |
| Shrinking all layers of the Quick network proportionally (compression) (ablation) | CIFAR10 Quick distilled student · CIFAR10 test error | test error %: distilled beats baseline up to ~40% compression; retrained baseline better at higher compression | Mixed |
Closest papers by meaning
- Demystifying Dropout (ICML 2019 · similarity 0.90)
- Dropout with Expectation-linear Regularization (ICLR 2017 · similarity 0.90)
- Dropout Reduces Underfitting (ICML 2023 · similarity 0.89)
- On the Regularization Properties of Structured Dropout (CVPR 2020 · similarity 0.89)
- R-Drop: Regularized Dropout for Neural Networks (NeurIPS 2021 · similarity 0.89)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.