Research map / Optimization / Implicit bias and generalization
The Implicit and Explicit Regularization Effects of Dropout
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine the Hessian-based explicit regularizer with derived implicit noise instead of using dropout | LSTM · Penn Treebank | validation perplexity: 73.76 -> 72.99 | Helped |
| Add the Hessian-based explicit regularizer without implicit noise | LSTM · Penn Treebank | validation perplexity: 112.04 -> 84.52 | Helped |
| Replace dropout with the explicit regularizer | Transformer-XL · WikiText-103, full dataset | best validation perplexity: 23.39 -> 24.12 | Hurt |
| Average dropout loss over more noise samples per update (ablation) | Transformer-XL with dropout · WikiText-103, full dataset | best validation perplexity: 23.39 -> 23.13; 23.39 -> 23.14 | No clear effect |
| Average dropout loss over more noise samples per update (ablation) | Transformer-XL with dropout · WikiText-103, 0.2× dataset | best validation perplexity: 46.05 -> 46.07; 46.05 -> 47.40 | Hurt |
| Replace the loss Hessian with the identity matrix in the explicit regularizer (ablation) | LSTM · Penn Treebank | best validation perplexity: 108.76 | Hurt |
| Use the loss-Jacobian-based explicit regularizer instead of the Hessian-based regularizer, while adding implicit noise (ablation) | LSTM · Penn Treebank | validation perplexity: 72.99 -> 79.54 | Hurt |
| Use the loss-Jacobian-based explicit regularizer instead of the Hessian-based regularizer, without implicit noise (ablation) | LSTM · Penn Treebank | validation perplexity: 84.52 -> 84.06 | No clear effect |
| Add the derived implicit noise to multi-sample dropout updates | LSTM · Penn Treebank and WikiText-2 | validation perplexity | Helped |
| Increase the number of dropout noise samples used per update (ablation) | LSTM · Penn Treebank and WikiText-2 | validation perplexity | Hurt |
| Use only the explicit regularizer, omitting the implicit regularization effect (ablation) | LSTM · Penn Treebank and WikiText-2 | validation perplexity | Hurt |
Closest papers by meaning
- Dropout Reduces Underfitting (ICML 2023 · similarity 0.93)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.91)
- On the Regularization Properties of Structured Dropout (CVPR 2020 · similarity 0.91)
- Demystifying Dropout (ICML 2019 · similarity 0.90)
- On the Implicit Bias of Dropout (ICML 2018 · similarity 0.90)
- Estimating Implicit Regularization in Deep Learning (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.