AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos

Lucas Fernandez-Sarmiento

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use front-loaded (step) dropout schedule instead of constant dropout at fixed budgetReLU MLP · ReLU h-sweep, mean h=0.1cross-entropy loss: +35.4% loss reduction, +2.04 pp accuracyImproved
use front-loaded step dropout instead of constant dropoutGELU MLP · GELU h-sweep, mean h=0.1cross-entropy loss: +29.8% loss reduction, +0.62 pp accuracyImproved
use early step dropout in an overfitting MLP setupMLP · MLP overfittingcross-entropy loss: +17.9% loss reduction, +0.83 pp accuracyImproved
use linearly decreasing dropout schedule instead of constantViT · CIFAR-100test accuracy: 48.69% -> 49.38%Improved
use linearly decreasing dropout schedule instead of constantViT · CIFAR-100cross-entropy loss: +4.2% loss reductionImproved
apply early step dropout to both blocks of ViT (ablation)ViT · CIFAR-10cross-entropy loss: +6.3% loss reduction, +0.52 pp accuracyImproved
switch from constant to saturated (step-like) dropout profile at fixed budgetmean-field objective · correlation lengtheffective correlation length: xi_step/xi_uniform = sqrt(hmax / h_bar) >= 1Improved
place dropout in early layers rather than late layers (same budget) (ablation)saturated dropout schedule · downstream regularization exposurecumulative downstream exposureImproved
use independent dropout masks per input (vs shared mask) (ablation)mean-field correlation map · perfect-alignment fixed pointc*=1 fixed pointHurt
swap smooth (tanh) for kinked (ReLU) activation under dropout (ablation)critical scaling · order parameter exponent betacritical exponent: 0.99 (tanh) vs 1.96 (ReLU)Mixed
increase dropout to high levels or use overly narrow networks (ablation)front-loaded schedules · MLP/ViT CIFARadvantage over constant dropoutHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy