Research map / Architectures and efficiency / RNNs and normalization
Jumpout : Improved Dropout for Deep Neural Networks with ReLUs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace constant dropout with jumpout (monotone sampled rate, adaptive normalization, rescaling) | DNNs with ReLU across CIFAR10(s), CIFAR10, CIFAR100, Fashion-MNIST, STL10, SVHN, ImageNet · test accuracy, 10 trials | test accuracy (%): e.g. +3.75 on CIFAR10(s), +1.48 on CIFAR10, +2.88 on CIFAR100, +1.28 on Fashion, +2.78 on STL10, +0.21 on SVHN, +0.34 on | Helped |
| add monotone (half-Gaussian) sampled dropout rate (modification I) (ablation) | dropout on small CNN and WideResNet · CIFAR10(s), CIFAR10, CIFAR100, STL10 | test accuracy (%): +1.06 CIFAR10(s), +0.85 CIFAR10, +0.48 CIFAR100, +0.39 STL10 | Helped |
| add adaptive dropout rate normalization by fraction of active neurons (modification II) (ablation) | dropout on small CNN and WideResNet · CIFAR10(s), CIFAR10, CIFAR100, STL10 | test accuracy (%): +0.71 CIFAR10(s), +0.46 CIFAR10, +0.90 CIFAR100, +0.57 STL10 | Helped |
| add rescaling with (1-p)^-0.75 to work with BN (modification III) (ablation) | dropout on small CNN and WideResNet · CIFAR10(s), CIFAR10, CIFAR100, STL10 | test accuracy (%): +1.27 CIFAR10(s), +0.97 CIFAR10, +1.20 CIFAR100, +0.32 STL10 | Helped |
| combine modifications I and II vs. dropout (ablation) | dropout · CIFAR10(s) | test accuracy (%): +3.56 over dropout 86.50 | Helped |
| add modification III (rescaling) on top of II (ablation) | dropout+II · CIFAR10(s) | test accuracy (%): 89.64 vs 87.14 without III | Helped |
| add modification I on top of III (ablation) | dropout+III · CIFAR10(s) | test accuracy (%): 87.36 vs 87.70 without I | No clear effect |
| all three modifications together vs. any pair (ablation) | dropout · CIFAR100 (WideResNet-28-10) | test accuracy (%): 82.48 vs 81.70 for best pair (I+II) | Helped |
Closest papers by meaning
- Dropout with Expectation-linear Regularization (ICLR 2017 · similarity 0.91)
- Demystifying Dropout (ICML 2019 · similarity 0.90)
- Guided Dropout (AAAI 2019 · similarity 0.90)
- Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training (NeurIPS 2021 · similarity 0.89)
- Dropout Reduces Underfitting (ICML 2023 · similarity 0.89)
- Adaptive Dropout with Rademacher Complexity Regularization (ICLR 2018 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.