Research map / Architectures and efficiency / RNNs and normalization
Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
Lucas Fernandez-Sarmiento
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use front-loaded (step) dropout schedule instead of constant dropout at fixed budget | ReLU MLP · ReLU h-sweep, mean h=0.1 | cross-entropy loss: +35.4% loss reduction, +2.04 pp accuracy | Improved |
| use front-loaded step dropout instead of constant dropout | GELU MLP · GELU h-sweep, mean h=0.1 | cross-entropy loss: +29.8% loss reduction, +0.62 pp accuracy | Improved |
| use early step dropout in an overfitting MLP setup | MLP · MLP overfitting | cross-entropy loss: +17.9% loss reduction, +0.83 pp accuracy | Improved |
| use linearly decreasing dropout schedule instead of constant | ViT · CIFAR-100 | test accuracy: 48.69% -> 49.38% | Improved |
| use linearly decreasing dropout schedule instead of constant | ViT · CIFAR-100 | cross-entropy loss: +4.2% loss reduction | Improved |
| apply early step dropout to both blocks of ViT (ablation) | ViT · CIFAR-10 | cross-entropy loss: +6.3% loss reduction, +0.52 pp accuracy | Improved |
| switch from constant to saturated (step-like) dropout profile at fixed budget | mean-field objective · correlation length | effective correlation length: xi_step/xi_uniform = sqrt(hmax / h_bar) >= 1 | Improved |
| place dropout in early layers rather than late layers (same budget) (ablation) | saturated dropout schedule · downstream regularization exposure | cumulative downstream exposure | Improved |
| use independent dropout masks per input (vs shared mask) (ablation) | mean-field correlation map · perfect-alignment fixed point | c*=1 fixed point | Hurt |
| swap smooth (tanh) for kinked (ReLU) activation under dropout (ablation) | critical scaling · order parameter exponent beta | critical exponent: 0.99 (tanh) vs 1.96 (ReLU) | Mixed |
| increase dropout to high levels or use overly narrow networks (ablation) | front-loaded schedules · MLP/ViT CIFAR | advantage over constant dropout | Hurt |
Closest papers by meaning
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (ICML 2025 · similarity 0.87)
- On Dropout and Nuclear Norm Regularization (ICML 2019 · similarity 0.86)
- Surprising properties of dropout in deep networks (JMLR 2018 · similarity 0.86)
- Stochastic Modified Equations and Dynamics of Dropout Algorithm (ICLR 2024 · similarity 0.86)
- Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos (ICLR 2025 · similarity 0.85)
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
