Research map / Architectures and efficiency / RNNs and normalization
Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Linear-Softmax with LMS-PLIF | AWD-LSTM · Penn Treebank | test perplexity: 58.37 -> 57.25 | Helped |
| Replace Linear-Softmax with LMS-PLIF | AWD-LSTM · WikiText-2 | test perplexity: 65.22 -> 64.86 | Helped |
| Replace LMS-PLIF with MoS | AWD-LSTM · Penn Treebank | test perplexity: 57.25 -> 56.43 | Helped |
| Replace LMS-PLIF with MoS | AWD-LSTM · WikiText-2 | test perplexity: 64.86 -> 63.33 | Helped |
| Add PLIF to MoS | AWD-LSTM · Penn Treebank | test perplexity: 56.43 -> 56.02 | Helped |
| Use LMS-PLIF instead of Linear-Softmax | AWD-LSTM · Penn Treebank | training time per epoch: ~60 -> ~70 | Hurt |
| Use LMS-PLIF instead of Linear-Softmax | AWD-LSTM · WikiText-2 | training time per epoch: ~120 -> ~150 | Hurt |
| Use LMS-PLIF rather than MoS | AWD-LSTM · Penn Treebank | training time per epoch: ~150 -> ~70 | Helped |
| Use LMS-PLIF rather than MoS | AWD-LSTM · WikiText-2 | training time per epoch: ~550 -> ~150 | Helped |
| Use LMS rather than Linear-Softmax or MoS | Synthetic distributions · Synthetic Dirichlet distributions | mode matching | Helped |
| Use LMS rather than Linear-Softmax or MoS | Synthetic distributions · Synthetic Dirichlet distributions | KL divergence | Helped |
| Use low-dimensional embeddings with a large vocabulary | Synthetic distributions · Low-D, large-M synthetic setting | mode matching | Hurt |
Closest papers by meaning
- On the Softmax Bottleneck of Recurrent Language Models (AAAI 2021 · similarity 0.89)
- Universal One-third Time Scaling in Learning Peaked Distributions (ICML 2026 · similarity 0.87)
- MultiMax: Sparse and Multi-Modal Attention Learning (ICML 2024 · similarity 0.86)
- Smooth Min-Max Monotonic Networks (ICML 2024 · similarity 0.86)
- Binary Hypothesis Testing for Softmax Models and Leverage Score Models (ICML 2025 · similarity 0.86)
- Loss Functions and Operators Generated by f-Divergences (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.