Research map / Large language models / Language, safety and interpretability
Demystifying Singular Defects in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the singular defect theory from ViTs to LLMs to explain the full life cycle of high-norm tokens (explosion, trigger, decay, development) and applies it to tensor-wise quantization and LLM model signatures.
Problem
Why LLMs produce abrupt high-norm hidden-state tokens (massive activations) that behave differently from the gradual high-norm defects in ViTs, and what mechanism drives their emergence and decay.
Approach
Applies a linearized single-token approximation of each transformer layer (L = I + R) to LLMs: the layer-wise leading left singular vector predicts the high-norm direction, the explosion layer's FFN leading right singular vector acts as a trigger subspace, and negative eigenvalues of R in decay layers explain norm suppression.
Key results
- Layer-wise singular defect directions for LLaMA2-7B match the empirical high-norm direction at layer 2 (6.05 degrees) and layer 31 (2.93 degrees).
- Skipping the down_proj in explosion and decay layers reduces RTN perplexity on LLaMA3-8B from 59.38 to 8.80 (FP16 baseline 6.14).
- Pairwise angles between LLaMA2-7B fine-tunes' singular defect directions are very small, while LLaMA3-8B vs LLaMA2-7B are near 90 degrees, supporting use as a model signature.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Preserve higher precision for F2 in layers 2 and 31 | LLaMA2-7B with tensor-wise RTN · WikiText2-v1 validation | perplexity: 10.18 -> 6.51 | Helped |
| Preserve higher precision for F2 in layers 2 and 31 | LLaMA2-7B with tensor-wise SmoothQuant · WikiText2-v1 validation | perplexity: 13.87 -> 6.78 | Helped |
| Preserve higher precision for F2 in layers 2 and 32 | LLaMA3-8B with tensor-wise RTN · WikiText2-v1 validation | perplexity: 59.38 -> 8.80 | Helped |
| Preserve higher precision for F2 in layers 2 and 32 | LLaMA3-8B with tensor-wise SmoothQuant · WikiText2-v1 validation | perplexity: 54.99 -> 9.14 | Helped |
| Remove self-attention layers (ablation) | LLaMA2-7B noninitial tokens · Token norms after explosion layer 2 | high-norm token retention | No clear effect |
| Remove self-attention layers (ablation) | LLaMA2-7B initial token · Token norm after explosion layer 2 | high-norm token emergence | Hurt |
| Remove the component in the explosion subspace before layer 2 FFN (ablation) | LLaMA2-7B · Intermediate token norms and generated text | generation quality | Hurt |
| Use sliding-window attention instead of the reported causal attention pattern | Mistral-7B · Initial-token high-norm emergence | high-norm token emergence | Hurt |
| Use bidirectional attention rather than causal attention | BERT, DistilBERT, and RoBERTa · High-norm token emergence | high-norm token emergence | Hurt |
Surprising findings
- Removing the high-norm component in the explosion subspace eliminates intermediate high-norm tokens but makes the model generate random text, showing high-norm tokens are functionally important.
- Random, non-learned token embeddings can also have higher norms than normal learned tokens after the explosion layer.
Limitations
- Root cause of high-norm token emergence is left as an open question; causal attention is only conjectured to be a factor.
- Analysis relies on a single-token linear approximation and uses real parts of eigenvalues/eigenvectors.
- Quantization evaluation is limited to perplexity on WikiText2 with RTN and SmoothQuant on two models.
Experimental setup
Datasets: WikiText2-v1
Benchmarks: WikiText2-v1
Metrics: perplexity mean pairwise angle (degrees) acute angle between singular defect and high-norm direction
Baselines: RTN SmoothQuant
Builds on: singular defects theory for ViTs (DINOv2) massive activations SmoothQuant LLM.int8()
Closest papers by meaning
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.90)
- Transformers need glasses! Information over-squashing in language tasks (NeurIPS 2024 · similarity 0.87)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.87)
- Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.87)
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.87)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.