AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

The Softmax Bottleneck Does Not Limit the Probabilities of the Most Likely Tokens

Ronen Basri, David Jacobs

ICLR 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train the output projection matrix instead of random initGPT-2/TinyLlama OPM · top-m token specificationmax mNo clear effect
use LP-optimized embedding instead of row-space embedding (adding orthogonal component)GPT-2-sized random matrix · top-m token specificationmax m with probability near 1: 26 -> 95Improved
use LP-optimized embedding instead of row-space boundTinyLlama · top-m token specificationmax m: 71 -> 400Improved
use LP-optimized embedding instead of row-space boundLlama2-7B · top-m token specificationmax m: 143 -> 1070Improved
prescribe exact probabilities instead of only top-m orderingGPT-2 · top-m token specificationmax m: 95 -> 85Hurt
draw target probabilities from GPT-2 on NYT text instead of uniformGPT-2 · prescribed top-m probabilitiesmax mNo clear effect
increase embedding size drandom matrices (N=32000) · top-m token specificationmax m: m ≈d/5Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy