Research map / Optimization / Implicit bias and generalization
Stochastic Modified Equations and Dynamics of Dropout Algorithm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives stochastic modified equations (SMEs) that weakly approximate the dropout iteration on two-layer networks, and empirically shows inverse variance-flatness and Hessian-variance alignment for dropout noise.
Problem
Understanding the mechanism by which dropout regularization leads to flatter minima and better generalization, given that its discrete iterative dynamics are hard to analyze.
Approach
Approximates the discrete dropout iteration in the weak sense by an SDE (stochastic modified equation) whose modified loss adds a dropout-induced regularizer and whose noise covariance is derived explicitly; then uses Hessian-covariance structural similarity to empirically test inverse variance-flatness and Hessian-variance alignment during dropout training.
Key results
- Derives SME for dropout showing order-1 weak approximation with drift -grad L_S and order-2 approximation with corrected drift.
- Empirically Tr(H Sigma) significantly exceeds Tr(H Sigma_bar) with isotropic noise throughout training on MNIST FNN across learning rates and dropout rates.
- Power-law relationships observed between noise variance and interval flatness (inverse variance-flatness) and between Hessian eigenvalues and projected variance on MNIST, ResNet-20 and transformer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dropout-induced anisotropic noise instead of isotropic noise | FNN trained with dropout · MNIST | Tr(HΣ) alignment | Helped |
| Train with dropout-induced noise | FNN trained with dropout · MNIST | noise variance versus interval flatness | Mixed |
| Train with dropout-induced noise | FNN trained with dropout · MNIST | projected variance versus Hessian eigenvalue | Helped |
| Train with dropout-induced noise | ResNet-20 · CIFAR-100 | noise variance versus flatness | Mixed |
| Train with dropout-induced noise | Transformer · Multi30k | noise variance versus flatness | Mixed |
Limitations
- Theory restricted to two-layer networks with quadratic loss and weak approximation; Hessian-covariance similarity relies on intuitive approximations.
- Hessian-variance experiments only compute traces over a subset of parameters due to computational limits.
- ResNet and Transformer experiments are only partially done, with Hessian computed at specific settings.
Experimental setup
Datasets: MNIST CIFAR-100 Multi30k
Benchmarks: MNIST CIFAR-100 Multi30k
Metrics: weak approximation order Tr(H Sigma) alignment interval flatness projected variance Hessian eigenvalues
Baselines: SGD GD
Builds on: stochastic modified equations dropout SGD noise structure analysis
Closest papers by meaning
- The Implicit and Explicit Regularization Effects of Dropout (ICML 2020 · similarity 0.91)
- On the Regularization Properties of Structured Dropout (CVPR 2020 · similarity 0.90)
- Demystifying Dropout (ICML 2019 · similarity 0.89)
- Dropout Reduces Underfitting (ICML 2023 · similarity 0.89)
- Continuum Dropout for Neural Differential Equations (AAAI 2026 · similarity 0.88)
- Dropout as a Structured Shrinkage Prior (ICML 2019 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.