Research map / Optimization / Implicit bias and generalization
Dropout Reduces Underfitting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use early dropout instead of no dropout | ViT-T · ImageNet-1K | top-1 accuracy: 76.3 -> 76.7 | Helped |
| Apply standard dropout throughout training | ViT-T · ImageNet-1K | top-1 accuracy: 76.3 -> 71.5 | Hurt |
| Start stochastic depth after the first 50 epochs rather than using it throughout | ViT-B · ImageNet-1K | top-1 accuracy: 81.6 -> 82.3 | Helped |
| Pre-train with early dropout | Swin-F · COCO | box AP: 36.4 -> 37.1 | Helped |
| Pre-train with early dropout | ViT-T · ADE20K | mIoU: 39.2 -> 40.0 | Helped |
| Use early dropout for 5 to 300 epochs instead of the default schedule (ablation) | ViT-T · ImageNet-1K | top-1 accuracy | Helped |
| Replace the linear decreasing early-dropout schedule with a constant schedule (ablation) | ViT-T · ImageNet-1K | top-1 accuracy: 76.7 vs. 76.7 | No clear effect |
| Use increasing dropout rather than early dropout (ablation) | ViT-T · ImageNet-1K | top-1 accuracy: 76.7 -> 75.2 | Hurt |
| Use decreasing dropout rather than early dropout (ablation) | ViT-T · ImageNet-1K | top-1 accuracy: 76.7 -> 74.7 | Hurt |
| Use curriculum dropout rather than early dropout (ablation) | ViT-T · ImageNet-1K | top-1 accuracy: 76.7 -> 70.4 | Hurt |
| Apply early dropout to a larger ViT-B model (ablation) | ViT-B · ImageNet-1K | top-1 accuracy: 78.7 -> 78.7 | No clear effect |
| Increase batch size to 8192 (ablation) | ViT-T with early dropout · ImageNet-1K | top-1 accuracy | No clear effect |
Closest papers by meaning
- The Implicit and Explicit Regularization Effects of Dropout (ICML 2020 · similarity 0.93)
- Dropout with Expectation-linear Regularization (ICLR 2017 · similarity 0.91)
- Demystifying Dropout (ICML 2019 · similarity 0.91)
- On the Regularization Properties of Structured Dropout (CVPR 2020 · similarity 0.90)
- Adversarial Dropout for Supervised and Semi-Supervised Learning (AAAI 2018 · similarity 0.90)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.