AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

Mostafa Elhoushi, Alexander D. Pretko, Nolan Simran Dey, Bin Claire Zhang, Gavia Gray, Gurpreet Gosal, Abdulrahman Mahmoud, Shane Bergsma, Joel Hestness

ICML 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
apply layer dropout to whole transformer blocks instead of attention and FFN sub-blocks separately (ablation)layer dropout, rate 0.1 · 20 TPP val loss, 271Mval loss: 2.377 -> 2.367Improved
sample dropout mask per sequence instead of per batch (ablation)layer dropout · 20 TPP val loss, 271M to 906Mval loss increaseImproved
scale residual by 1/density during training (r_train = 1/rho) instead of 1 (ablation)layer dropout with muP/CompleteP · hyperparameter transfer, 271Moptimal LR and batch size transferImproved
use increasing layer distribution instead of uniform at equal FLOPs savings (ablation)layer dropout · 20 TPP val loss, 906M, 20% savingsval loss: 2.008 -> 1.998Improved
use alternating distribution instead of increasing at equal FLOPs savings (ablation)layer dropout · 20 TPP val loss across 271M, 503M, 906Mval loss: 271M 20%: 2.357 vs 2.363; 906M 20%: 2.004 vs 1.998Mixed
use decreasing time schedule instead of increasing schedule at equal FLOPs savings (ablation)ILD layer dropout · 20 TPP val loss, 906M, 20% savingsval loss: 2.090 -> 1.983Improved
use increasing time schedule instead of constant (ablation)ILD layer dropout · 20 TPP val loss, 906M, 10% savingsval loss: 1.972 -> 2.014Hurt
add ILD plus decreasing schedule (max 0.2) to dense training at 5% FLOPs savingsdense baseline · 20 TPP val loss, 906Mval loss: 1.953 -> 1.951Improved
train with layer dropout instead of dense for early exitdense baseline · early exit validation loss, 20 TPPval lossImproved
use decreasing schedule (ends dropout-free) instead of constant dropout for early-exit robustness (ablation)ILD layer dropout · early exit, 20% FLOPs savingsval lossNo clear effect
use alternating instead of increasing distribution for layer skipping (ablation)layer dropout · intermediate layer skipping, 20% FLOPs savingsval lossImproved
pre-train with layer dropout before adding Balcony exit adaptersdense pre-training · early exit CE loss, 270M/503M/906MCE lossImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy