Research map / Large language models / Language, safety and interpretability
The Softmax Bottleneck Does Not Limit the Probabilities of the Most Likely Tokens
Ronen Basri, David Jacobs
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the output projection matrix instead of random init | GPT-2/TinyLlama OPM · top-m token specification | max m | No clear effect |
| use LP-optimized embedding instead of row-space embedding (adding orthogonal component) | GPT-2-sized random matrix · top-m token specification | max m with probability near 1: 26 -> 95 | Improved |
| use LP-optimized embedding instead of row-space bound | TinyLlama · top-m token specification | max m: 71 -> 400 | Improved |
| use LP-optimized embedding instead of row-space bound | Llama2-7B · top-m token specification | max m: 143 -> 1070 | Improved |
| prescribe exact probabilities instead of only top-m ordering | GPT-2 · top-m token specification | max m: 95 -> 85 | Hurt |
| draw target probabilities from GPT-2 on NYT text instead of uniform | GPT-2 · prescribed top-m probabilities | max m | No clear effect |
| increase embedding size d | random matrices (N=32000) · top-m token specification | max m: m ≈d/5 | Improved |
Closest papers by meaning
- Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models (NeurIPS 2024 · similarity 0.88)
- Transformers need glasses! Information over-squashing in language tasks (NeurIPS 2024 · similarity 0.87)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.87)
- Compute-Optimal LLMs Provably Generalize Better with Scale (ICLR 2025 · similarity 0.87)
- On the Generalization Ability of Next-Token-Prediction Pretraining (ICML 2025 · similarity 0.87)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
