Research map / Large language models / Language, safety and interpretability

Superposition Yields Robust Neural Scaling

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses a weight-decay-controlled Anthropic-style toy model of superposition to show that under strong superposition loss scales ~1/model dimension for flat-to-moderate feature frequencies, and that open LLM output-head geometry and losses match this 1/m scaling (alpha_m ≈ 0.91).

Problem

Why neural scaling laws (loss as a power law in model size) arise, and when loss vs. model width follows a power law and with what exponent, given that LLM representations must pack far more features than dimensions.

Approach

Treat representation superposition as the mechanism behind width scaling: a weight decay term tunes the degree of superposition in a toy autoencoder; under weak superposition loss is the summed frequency of unlearned features (power law only if frequencies are power law), while under strong superposition the geometry of near-equiangular representation vectors gives squared overlaps ~1/m and hence

Key results

• Under strong superposition, toy-model loss scales with model dimension with fitted exponents about 1.0 (exponential), 1.0 (power-law) and 0.89 (linear) decay of feature frequency.
• Open LLMs (OPT, GPT-2, Qwen2.5, Pythia; ~100M to 70B) have language-model-head mean squared overlaps roughly following 1/m and a fitted model exponent alpha_m = 0.91 ± 0.04.
• Using Chinchilla scaling (N ∝ m^2.52) the paper infers alpha_m = 0.88 ± 0.06 for Chinchilla models, consistent with ~1/m width scaling.

Measured effects reported in the paper

Surprising findings

• Under strong superposition, loss is approximately a power law in width across all tested feature-frequency distributions, including exponential and linear, whereas weak superposition yields power-law loss only for power-law frequencies.

Limitations

• The analysis is largely heuristic; the toy model is not rigorously solved and the strong-superposition exponent for skewed frequencies is conjectural.
• The LLM mapping treats tokens as atomic features and omits transformer layers, so parsing-limited loss is not separated from representation-limited loss.
• LLM fits rely on a loss decomposition with a nonzero intercept and a limited set of model families and datasets.

Experimental setup

Datasets: Synthetic feature-frequency data (power-law, exponential, linear) Wikitext-103 Pile-10k C4 BookCorpus

Benchmarks: Wikitext-103 Pile-10k C4 BookCorpus

Metrics: test loss fitted model exponent alpha_m R-squared of power-law fit mean squared overlap of normalized rows

Baselines: weak superposition toy model Chinchilla scaling laws variance-limited 1/m width scaling

Builds on: Toy Models of Superposition AdamW Chinchilla scaling laws

Closest papers by meaning

• On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.90)
• Universal One-third Time Scaling in Learning Peaked Distributions (ICML 2026 · similarity 0.88)
• A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.87)
• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.87)
• Neural Neural Scaling Laws (NeurIPS 2026 · similarity 0.87)
• Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation (NeurIPS 2026 · similarity 0.87)