Research map / Architectures and efficiency / Attention and transformers
Adaptive Input Representations for Neural Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive input embeddings instead of character-CNN inputs | Self-attentional language model · Billion Word | test perplexity: 25.88 -> 25.22 | Helped |
| Tie adaptive input and output weights (ablation) | Adaptive input/softmax model · WikiText-103 | test perplexity: 21.74 -> 20.51 | Helped |
| Tie adaptive input and output weights (ablation) | Adaptive input/softmax model · Billion Word | test perplexity: 26.49 -> 25.58 | Helped |
| Add dropout 0.2 to adaptive softmax tail projections (ablation) | ADP · WikiText-103 | validation perplexity: 24.74 -> 21.23 | Helped |
| Add dropout to adaptive softmax tail projections (ablation) | Adaptive softmax · Billion Word | perplexity | No clear effect |
| Increase training block size from 512 to 3072 (ablation) | Adaptive inputs · WikiText-103 | test perplexity: 20.51 -> 19.34 | Helped |
| Increase inference context from 0 to 2560 tokens (ablation) | Adaptive inputs with 3072-token training blocks · WikiText-103 | test perplexity: 19.34 -> 18.70 | Helped |
| Reduce fixed input embedding dimension from 256 to 64 (ablation) | Embedding + Adaptive output · WikiText-103 | validation perplexity: 23.39 -> 21.23 | Helped |
| Tie the adaptive softmax head projection as well as embeddings and projections (ablation) | ADP-T · WikiText-103 | validation perplexity: 19.79 -> 20.06 | Hurt |
| Use equal-sized word representations instead of dimension-reduced adaptive softmax representations (ablation) | Tied adaptive input/output model · WikiText-103 | validation perplexity: 22.63 -> 25.48 | Hurt |
| Remove input/output weight tying (ablation) | Adaptive input/softmax model · WikiText-103 | test perplexity: 20.51 -> 21.74 | Hurt |
| Use adaptive inputs rather than fixed-size word embeddings | Self-attentional language model · WikiText-103 | test perplexity: 24.92 -> 21.74 | Helped |
Closest papers by meaning
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.87)
- Dual Dimensionality for Local and Global Attention (NeurIPS 2026 · similarity 0.87)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.87)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.86)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.86)
- Layer by Layer: Uncovering Hidden Representations in Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.