Research map / Architectures and efficiency / Attention and transformers
Faster Depth-Adaptive Transformers
Yijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen, Jinan Xu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Estimate per-word depth from mutual information instead of using fixed depth | BERT-initialized Transformer · Larger classification benchmarks, average | accuracy: +0.77% | Improved |
| Estimate per-word depth from mutual information instead of using a halting unit | BERT-initialized Transformer · Larger classification benchmarks, average | accuracy: +0.76% | Improved |
| Estimate per-word depth from reconstruction loss instead of using fixed depth | BERT-initialized Transformer · Larger classification benchmarks, average | accuracy: +0.24% | Improved |
| Estimate per-word depth from reconstruction loss instead of using a halting unit | BERT-initialized Transformer · Larger classification benchmarks, average | accuracy: +0.23% | Improved |
| Add a halting unit for adaptive depth selection | BERT-initialized Transformer · Larger classification benchmarks | accuracy | No clear effect |
| Use word-level MI-based depth adaptation rather than fixed layers | Transformer · IMDB | speed: about 7x | Improved |
| Increase the reconstruction depth penalty lambda from 0 to 0.20 (ablation) | Reconstruction estimation · IMDB | speed: 23 -> 58 | Improved |
| Increase the reconstruction depth penalty lambda from 0 to 0.20 (ablation) | Reconstruction estimation · IMDB | accuracy: 96.54 -> 96.29 | Mixed |
| Increase the reconstruction depth penalty lambda from 0 to 0.20 (ablation) | Reconstruction estimation · IMDB | average depth: 9.5 -> 3.6 | Hurt |
Closest papers by meaning
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.86)
- Adaptive Depth Networks with Skippable Sub-Paths (NeurIPS 2024 · similarity 0.85)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.84)
- Prompt-based Depth Pruning of Large Language Models (ICML 2025 · similarity 0.83)
- Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers (NeurIPS 2025 · similarity 0.83)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
