Research map / Architectures and efficiency / Attention and transformers
Deep Transformers with Latent Depth
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned latent depth instead of static depth at the same effective depth | Transformer on WMT'16 En-De · WMT'16 En-De | BLEU_test: 26.5 -> 27.88±0.04 | Helped |
| Train with latent layer selection | 24-layer encoder on crosslingual MLM · Crosslingual MLM held-out data | Perplexity: 2.91 -> 2.82 | Helped |
| Train with latent layer selection | 48-layer encoder on crosslingual MLM · Crosslingual MLM held-out data | Perplexity: 2.60 -> 2.71 | Hurt |
| Train a 96-layer encoder with latent layer selection rather than static depth | Transformer on crosslingual MLM · Crosslingual MLM held-out data | Perplexity: Diverged -> 2.66 | Helped |
| Use latent layer selection in the decoder for increased depth | Multilingual O2M-Related translation · O2M-Related | Avg. BLEU: 20.48 -> 20.83 | Helped |
| Use latent layer selection rather than a 12/12 static model | Multilingual O2M-Diverse translation · O2M-Diverse | Avg. BLEU: 23.34 -> 24.46 | Helped |
| Use latent layer selection in the encoder and decoder | Multilingual M2O-Diverse translation · M2O-Diverse | Avg. BLEU: 27.73 -> 28.56 | Helped |
| Add target-depth loss L_K (ablation) | LL-D, 24/24 on O2M-Diverse · O2M-Diverse | Avg. BLEU: 24.28 -> 24.46 | Helped |
| Add the KL term (ablation) | LL-D, 24/24 on O2M-Diverse · O2M-Diverse | Avg. BLEU: 23.89 -> 24.46 | Helped |
| Add both L_K and KL losses (ablation) | LL-D, 24/24 on O2M-Diverse · O2M-Diverse | Avg. BLEU: 23.75 -> 24.46 | Helped |
| Increase KL coefficient β from 1 to 10 (ablation) | Latent layers on M2O-Diverse · M2O-Diverse | Avg. valid BLEU: 28.53 -> 28.23 | Hurt |
| Vary inner-loop frequency from I=1 to I=10 (ablation) | Latent layers on M2O-Diverse · M2O-Diverse | Avg. validation BLEU: 28.28; 28.49; 28.24; 28.25 | No clear effect |
Closest papers by meaning
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.88)
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.88)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.87)
- HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization (NeurIPS 2025 · similarity 0.87)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.