Research map / Learning theory and algorithms / Matrix and tensor methods
On the Softmax Bottleneck of Recurrent Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace softmax with GSS across settings that yield diverse log P ranks | AWD-LSTM · PTB and WikiText-2 | test perplexity: 57.08±0.09 -> 57.00±0.14; 64.63±0.08 -> 64.50±0.13 | No clear effect |
| Increase log P rank using GSS parameter choices | AWD-LSTM with GSS · PTB and WikiText-2 | test perplexity: 0.48; 0.53; 0.24; 0.21 | Mixed |
| Replace softmax with SigSoftmax | AWD-LSTM · PTB | test perplexity: 57.08±0.09 -> 57.05±0.15 | No clear effect |
| Replace softmax with SigSoftmax | AWD-LSTM · WikiText-2 | test perplexity: 64.63±0.08 -> 64.35±0.17 | Helped |
| Add the PLIF layer before softmax | AWD-LSTM · PTB and WikiText-2 | test perplexity: 57.08±0.09 -> 56.80±0.11; 64.63±0.08 -> 64.16±0.17 | Helped |
| Freeze the PLIF layer rather than learning its slopes (ablation) | LMS-PLIF · PTB and WikiText-2 | test perplexity: 56.81 ± 0.11; 56.82 ± 0.09; 64.15 ± 0.17; 64.30 ± 0.17 | No clear effect |
| Add a fixed PLIF transformation to the logits (ablation) | Softmax model · PTB and WikiText-2 | test perplexity: 57.08±0.09 -> 56.83±0.10; 64.63±0.08 -> 64.28±0.17 | Helped |
| Replace softmax with Mixture of Softmaxes | AWD-LSTM · PTB and WikiText-2 | test perplexity: 57.08±0.09 -> 54.91±0.26; 64.63±0.08 -> 61.96±0.40 | Helped |
| Increase MoS dropout to increase log P rank while keeping K = 15 | MoS · WikiText-2 | test perplexity: 0.290 0.145 0.000; 63 66 69 | Hurt |
| Increase the number of MoS mixtures | MoS · PTB and WikiText-2 | test perplexity | Mixed |
| Match MoS and Softmax capacity and tune regularization hyperparameters | MoS† versus Softmax‡ · PTB | test perplexity: 56.39±0.19; 56.45±0.10; 0.62 | No clear effect |
Closest papers by meaning
- Universal One-third Time Scaling in Learning Peaked Distributions (ICML 2026 · similarity 0.85)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.85)
- The Sparsity Whisperer (NeurIPS 2026 · similarity 0.85)
- DLP: Dynamic Layerwise Pruning in Large Language Models (ICML 2025 · similarity 0.85)
- Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (NeurIPS 2025 · similarity 0.85)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.