Research map / Architectures and efficiency / Attention and transformers
Deep Residual Output Layers for Neural Language Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a deep residual output label mapping | AWD-LSTM · Penn Treebank | perplexity: 57.3 -> 55.7 | Helped |
| Add a deep residual output label mapping | AWD-LSTM · WikiText-2 | perplexity: 65.8 -> 61.9 | Helped |
| Add a deep residual output label mapping | Transformer (base) · WMT 2014 English-German | BLEU: 27.3 -> 28.1 | Helped |
| Increase label encoder depth from 1 to 2 layers (ablation) | DRILL · PennTreebank | perplexity: 56.2 -> 56.0 | Helped |
| Increase label encoder depth from 2 to 3 layers (ablation) | DRILL · PennTreebank | perplexity: 56.0 -> 55.9 | Helped |
| Increase label encoder depth from 3 to 4 layers (ablation) | DRILL · PennTreebank | perplexity: 55.9 -> 55.7 | Helped |
| Remove residual connections between label encoder layers (ablation) | DRILL · PennTreebank | perplexity: 57.5 -> 55.7 | Helped |
| Add variational dropout between label encoder layers (ablation) | DRILL · PennTreebank | perplexity: 60.7 -> 55.7 | Helped |
| Replace the full softmax output layer with DRILL (ablation) | AWD-LSTM · PennTreebank | perplexity: 66.8 -> 55.7 | Helped |
Closest papers by meaning
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.84)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.84)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.84)
- Do Language Models Use Their Depth Efficiently? (NeurIPS 2025 · similarity 0.83)
- Self Iterative Label Refinement via Robust Unlabeled Learning (NeurIPS 2025 · similarity 0.83)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.