Research map / Large language models / Language, safety and interpretability
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a single Softmax with a probability mixture of Softmax components | AWD-LSTM · Penn Treebank test | perplexity: 51.1 -> 47.69 | Helped |
| Replace a single Softmax with a probability mixture of Softmax components | AWD-LSTM · WikiText-2 test | perplexity: 44.3 -> 40.68 | Helped |
| Replace Softmax with a probability mixture of Softmax components | 2-layer LSTM · 1B Word test | perplexity: 42.77 -> 37.10 | Helped |
| Replace Softmax with a probability mixture of Softmax components | Seq2Seq · Switchboard | perplexity: 34.657 -> 32.727 | Helped |
| Mix probabilities after each component Softmax rather than mixing context vectors before Softmax (ablation) | AWD-LSTM-MoC · Penn Treebank test | perplexity: 57.55 -> 55.97 | Helped |
| Mix probabilities after each component Softmax rather than mixing context vectors before Softmax (ablation) | AWD-LSTM-MoC · WikiText-2 test | perplexity: 65.98 -> 63.33 | Helped |
| Use MoS hyper-parameters without the MoS architecture (ablation) | AWD-LSTM · Penn Treebank test | perplexity: 58.95 -> 74.86 | Hurt |
| Use MoS hyper-parameters without the MoS architecture (ablation) | AWD-LSTM · WikiText-2 test | perplexity: 65.40 -> 69.18 | Hurt |
| Increase mixture components from 3 to 15 (ablation) | MoS · Penn Treebank test | perplexity: 58.62 -> 55.97 | Helped |
| Increase mixture components from 15 to 20 (ablation) | MoS · Penn Treebank test | perplexity: 55.97 -> 56.17 | Hurt |
| Use MoS instead of Softmax on character-level language modeling (ablation) | LSTM language model · text8 test | BPC: 1.49 -> 1.49 | No clear effect |
| Use dropout (ablation) | 1B Word language model · 1B Word | perplexity | No clear effect |
Closest papers by meaning
- On the Softmax Bottleneck of Recurrent Language Models (AAAI 2021 · similarity 0.91)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.87)
- SparseLLM: Towards Global Pruning of Pre-trained Language Models (NeurIPS 2024 · similarity 0.87)
- SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.87)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.87)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.