Research map / Architectures and efficiency / RNNs and normalization
Demystifying Dropout
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use different forward and backward dropout ratios (augmented dropout) instead of a single standard dropout ratio | LeNet on MNIST · MNIST | test accuracy: 0.9918 to 0.9925 at ratio 0.3 | Helped |
| Use different forward and backward dropout ratios (augmented dropout) | ConvNet-Quick · SVHN and CIFAR10 | test accuracy: e.g. SVHN 0.9231 to 0.9258; CIFAR10 0.7657 to 0.7674 | Helped |
| Use different forward and backward dropout ratios (augmented dropout) | ResNet-20 · SVHN, CIFAR10, CIFAR100 | test accuracy: CIFAR100 0.6762 to 0.6786 at ratio 0.1 | Helped |
| Use augmented dropout instead of baselines (Monte Carlo, distillation, EL dropout) | ALL-Conv and NiN · CIFAR10 | test error: ALL-Conv 10.04 vs best baseline 10.60; NiN 10.28 vs 11.04 | Helped |
| Apply dropout only in the forward pass (forward dropout) vs standard dropout on both passes (ablation) | ConvNet-Quick · CIFAR10 | test accuracy: 0.7617 vs 0.7617? see row; at p=0.3 0.6908 vs 0.7523 standard | Hurt |
| Apply dropout only in the backward pass (backward dropout) vs standard dropout on both passes (ablation) | ConvNet-Quick · CIFAR10 | test accuracy: e.g. 0.7585 vs 0.7617 at p=0.2 for standard 0.7617 | Hurt |
| Apply dropout only in the backward pass (noisy gradient) with very small ratio 0.005 (ablation) | ConvNet-Quick · CIFAR10 | test accuracy: 0.7599 vs plain 0.7579 | Helped |
| Apply dropout only in the backward pass with very small ratio 0.005 (ablation) | ResNet-20 · CIFAR10 | test accuracy: 0.9146 vs plain 0.9143 | Helped |
| Apply dropout only in the forward pass with small ratio 0.005 (forward dropout) (ablation) | ResNet-20 and ConvNet-Quick · CIFAR10 | test accuracy: ConvNet-Quick 0.7578 vs plain 0.7579; ResNet-20 0.9142 vs 0.9143 | No clear effect |
| Increase forward dropout ratio to large values (p=0.3) (ablation) | forward dropout on ConvNet-Quick · CIFAR10 | test accuracy: 0.6908 at p=0.3 vs plain 0.7579 | Hurt |
| Backward dropout with mild ratio vs standard dropout at p=0.05 (ablation) | forward/backward dropout on ConvNet-Quick · CIFAR10 | test accuracy: forward 0.7627 and backward 0.7593 vs standard 0.7647 at p=0.05 | Mixed |
| Standard dropout (both passes) vs plain network (ablation) | ConvNet-Quick and ResNet-20 · CIFAR10 | test accuracy: ConvNet-Quick best 0.7657 vs plain 0.7579; large p=0.3 ResNet 0.9163 vs plain 0.9143 | Mixed |
Closest papers by meaning
- Guided Dropout (AAAI 2019 · similarity 0.92)
- On the Implicit Bias of Dropout (ICML 2018 · similarity 0.91)
- Dropout with Expectation-linear Regularization (ICLR 2017 · similarity 0.91)
- Dropout Reduces Underfitting (ICML 2023 · similarity 0.91)
- Jumpout : Improved Dropout for Deep Neural Networks with ReLUs (ICML 2019 · similarity 0.90)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.