Research map / Architectures and efficiency / Pruning and efficient tuning
Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
Mostafa Elhoushi, Alexander D. Pretko, Nolan Simran Dey, Bin Claire Zhang, Gavia Gray, Gurpreet Gosal, Abdulrahman Mahmoud, Shane Bergsma, Joel Hestness
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply layer dropout to whole transformer blocks instead of attention and FFN sub-blocks separately (ablation) | layer dropout, rate 0.1 · 20 TPP val loss, 271M | val loss: 2.377 -> 2.367 | Improved |
| sample dropout mask per sequence instead of per batch (ablation) | layer dropout · 20 TPP val loss, 271M to 906M | val loss increase | Improved |
| scale residual by 1/density during training (r_train = 1/rho) instead of 1 (ablation) | layer dropout with muP/CompleteP · hyperparameter transfer, 271M | optimal LR and batch size transfer | Improved |
| use increasing layer distribution instead of uniform at equal FLOPs savings (ablation) | layer dropout · 20 TPP val loss, 906M, 20% savings | val loss: 2.008 -> 1.998 | Improved |
| use alternating distribution instead of increasing at equal FLOPs savings (ablation) | layer dropout · 20 TPP val loss across 271M, 503M, 906M | val loss: 271M 20%: 2.357 vs 2.363; 906M 20%: 2.004 vs 1.998 | Mixed |
| use decreasing time schedule instead of increasing schedule at equal FLOPs savings (ablation) | ILD layer dropout · 20 TPP val loss, 906M, 20% savings | val loss: 2.090 -> 1.983 | Improved |
| use increasing time schedule instead of constant (ablation) | ILD layer dropout · 20 TPP val loss, 906M, 10% savings | val loss: 1.972 -> 2.014 | Hurt |
| add ILD plus decreasing schedule (max 0.2) to dense training at 5% FLOPs savings | dense baseline · 20 TPP val loss, 906M | val loss: 1.953 -> 1.951 | Improved |
| train with layer dropout instead of dense for early exit | dense baseline · early exit validation loss, 20 TPP | val loss | Improved |
| use decreasing schedule (ends dropout-free) instead of constant dropout for early-exit robustness (ablation) | ILD layer dropout · early exit, 20% FLOPs savings | val loss | No clear effect |
| use alternating instead of increasing distribution for layer skipping (ablation) | layer dropout · intermediate layer skipping, 20% FLOPs savings | val loss | Improved |
| pre-train with layer dropout before adding Balcony exit adapters | dense pre-training · early exit CE loss, 270M/503M/906M | CE loss | Improved |
Closest papers by meaning
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.88)
- Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICML 2024 · similarity 0.88)
- Streamlining Redundant Layers to Compress Large Language Models (ICLR 2025 · similarity 0.87)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.87)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.87)
- The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
