Research map / Large language models / Language, safety and interpretability

Large language models suffer from a curse of ambiguity

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies a "curse of ambiguity" showing that next-token distributions with larger support are harder to learn, with theoretical roots in capacity, embedding dimension, optimization, and sampling noise, validated on synthetic 3-gram transformers and OLMo-2 models.

Problem

Neural networks producing discrete next-token distributions, including LLMs, may struggle to accurately learn high-entropy (ambiguous) next-token distributions, which matters for sampling-based training, diversity, and calibration.

Approach

Formalizes ambiguity as support size (effective support via exponential entropy) and shows via linear-softmax analysis that capacity scales as hd/k, embedding dimension must grow linearly with k (h >= 2k), gradient flow slows for large k, and label sampling noise adds a k-dependent floor (k-1)/(4k) to the KL.

Key results

• Capacity of the Hebbian linear head scales as hd/k (up to log factors), with a matching sign-pattern upper bound.
• Embedding dimension h >= 2k suffices to exactly represent all size-k uniform distributions in the idealized setting.
• Label sampling noise adds an asymptotic KL floor of (k-1)/(4k), approaching 1/4 as k grows; OLMo-2 1B shows higher per-context KL at higher effective support size at fixed semantic context frequency.

Surprising findings

• Transformers leak probability mass outside the support for ambiguous contexts, making the sequence-level distribution higher-entropy than ground truth.
• Some low-frequency contexts with OLMo-2 1B agree closely with the larger model, attributed to in-context copying mechanisms.

Limitations

• Theory uses uniform-on-support distributions and random/i.i.d. embeddings; real next-token distributions are non-uniform.
• Real-data analysis relies on proxies: OLMo-2 13B as ground truth and kNN embedding density as context frequency, both biased.
• Ideal-backbone assumption in embedding analysis ignores how the backbone learns embeddings.

Experimental setup

Datasets: TinyStories Dolma v1.7

Benchmarks: synthetic 3-gram task (TinyStories-derived) Dolma v1.7 per-context KL evaluation

Metrics: KL divergence probability mass on support entropy gap effective support size semantic context frequency

Baselines: OLMo-2 1B

Builds on: Hopfield networks softmax bottleneck compressed sensing gradient flow Hebbian associative memory

Closest papers by meaning

• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.90)
• Arrows of Time for Large Language Models (ICML 2024 · similarity 0.89)
• An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.89)
• Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.89)
• Generating in the Limit with Infinitely Many Hallucinations (NeurIPS 2026 · similarity 0.88)
• Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation (NeurIPS 2025 · similarity 0.88)