Research map / Architectures and efficiency / Attention and transformers
Reducing Transformer Depth on Demand with Structured Dropout
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LayerDrop to the 16-layer Adaptive Inputs model | Adaptive Inputs · WikiText-103 test | perplexity: 18.7 -> 18.3 | Helped |
| Increase depth to 40 layers and train with LayerDrop | Adaptive Inputs · WikiText-103 test | perplexity: 18.3 -> 17.7 | Helped |
| Add LayerDrop to Transformer Big | Transformer Big · WMT14 en-de newstest2014 | BLEU: 29.3 -> 29.6 | Helped |
| Add LayerDrop and increase encoder depth to 12 layers | Transformer Big · WMT14 en-de newstest2014 | BLEU: 30.2 BLEU | Helped |
| Add LayerDrop | Transformer · CNN-Dailymail test | ROUGE-1: 40.1 -> 40.5 | Helped |
| Add LayerDrop | Transformer Multitask · ELI5 test | ROUGE-1: 28.9 -> 29.4 | Helped |
| Add LayerDrop to the 24-layer RoBERTa | RoBERTa · GLUE dev, Books + Wiki | MNLI-m accuracy: 89.0 -> 89.2 | Helped |
| Add LayerDrop | Transformer · IWSLT de-en test | BLEU: 34.4 -> 34.5 | No clear effect |
| Prune a LayerDrop-trained model to a desired depth without finetuning | Transformer generation models · Generation tasks at varied depths | task performance | Helped |
| Drop heads instead of whole layers (ablation) | Structured Dropout Transformer · WikiText-103 valid | perplexity | Hurt |
| Drop sub-layers instead of whole layers (ablation) | Structured Dropout Transformer · WikiText-103 valid | perplexity | No clear effect |
| Combine head dropping with layer or sub-layer dropping (ablation) | Structured Dropout Transformer · WikiText-103 valid | perplexity | Helped |
Closest papers by meaning
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.87)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.86)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.86)
- The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction (ICLR 2024 · similarity 0.86)
- SlimLLM: Accurate Structured Pruning for Large Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.