Research map / Large language models / Language, safety and interpretability
Linguistic Collapse: Neural Collapse in (Large) Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies neural collapse in 230 causal language models trained on TinyStories, finding NC properties emerge with scale and training and correlate with validation generalization.
Problem
Neural collapse is known for balanced, few-class, noise-free, zero-loss classification; language modelling via next-token prediction violates all these conditions (large imbalanced vocabulary, ambiguous contexts, undertraining). It is unclear whether NC emerges in causal LMs and how it relates to ge
Approach
Applies neural collapse analysis to next-token prediction using the full vocabulary as the class space, introduces a modified CDNV metric (extra mean-distance power), uniform duality (UNC3) as CoV of classifier-mean alignment, and tests NC-generalization correlations across scale and with fixed-architecture seed variation.
Key results
- Permutation test over 21 identically configured 2-layer 768-wide models: hyperspherical uniformity R^2=0.488 (p=0.0002) and classifier agreement R^2=0.490 (p=0.0001) correlate with validation loss.
- Uniform duality (UNC3) showed no correlation with validation loss in the permutation test (R^2=0.000036, p=0.978), while self-duality NC3 did (R^2=0.322, p=0.006).
- Trained 230 CLMs across widths 64-1024 and depths 1-12 for 1, 3, 10 epochs; NC properties such as within-class variability and hyperspherical uniformity improve with scale and training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model width | TinyStories causal language models · TinyStories | Within-class variability (CDNV) | Helped |
| Increase training epochs | TinyStories causal language models · TinyStories | Within-class variability (CDNV) | Helped |
| Increase weight decay from β=0.0005 to β=0.1 (ablation) | TinyStories causal language models · TinyStories | Neural collapse properties | Helped |
| Increase model scale | TinyStories causal language models · TinyStories | Equinormness | Helped |
| Increase model dimensions | TinyStories causal language models · TinyStories | Average interference | Helped |
| Increase model scale | TinyStories causal language models · TinyStories | Interference variation (equiangularity) | Hurt |
| Increase model scale | TinyStories causal language models · TinyStories | Hyperspherical uniformity | Helped |
| Increase model width | TinyStories causal language models · TinyStories | Uniform duality | Helped |
| Increase model depth | TinyStories causal language models · TinyStories | Uniform duality | Hurt |
| Increase model width or depth | TinyStories causal language models · TinyStories | Classifier agreement | Helped |
| Increase training for some wider models | TinyStories causal language models · TinyStories | Classifier agreement | Helped |
| Relate NC1 measurements to validation loss | Two-layer 768-wide CLM · TinyStories validation | R² correlation: 0.192011 | Helped |
Surprising findings
- Self-duality (NC3) does not develop with scale, and the classifier-mean alignment shows no meaningful trend with width.
- Uniform duality (UNC3) does not correlate with validation loss in the scale-independent permutation test despite correlating with scale.
Limitations
- Only small CLMs (3.4M to 205M parameters) on synthetic TinyStories; no evaluation on LLMs or real corpora.
- Confounder analysis limited to a single small architecture trained one epoch with weak weight decay.
- Only validation cross-entropy used as performance measure; no downstream task evaluation.
Experimental setup
Datasets: TinyStories
Benchmarks: TinyStories validation set
Metrics: validation cross-entropy loss class-distance normalized variance interference hyperspherical uniformity log-distance classifier-mean cosine similarity classifier agreement coefficient of determination R^2 permutation test p-value
Builds on: neural collapse generalized neural collapse GPT-Neo TinyStories
Closest papers by meaning
- Strong Model Collapse (ICLR 2025 · similarity 0.90)
- Generalized Neural Collapse for a Large Number of Classes (ICML 2024 · similarity 0.88)
- Neural Collapse Beyond the Unconstrained Features Model: Landscape, Dynamics, and Generalization in the Mean-Field Regime (ICML 2025 · similarity 0.87)
- Escaping Collapse: The Strength of Weak Data for Large Language Model Training (NeurIPS 2025 · similarity 0.87)
- Collapsed Language Models Promote Fairness (ICLR 2025 · similarity 0.86)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.