Research map / Large language models / Language, safety and interpretability

Systematic Outliers in Large Language Models

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes activation, weight, and attention outliers in LLMs, attributes their formation to softmax attention, and shows that an explicit context-aware scaling factor in attention removes them and improves convergence and compression robustness on GPT-2.

Problem

Outliers in LLM activations, weights, and attention scores hurt performance and compression, but their formation mechanisms and functional roles are poorly understood, and prior work mostly mitigates them algorithmically.

Approach

Defines three outlier types, maps where they occur and how they interconnect (weight outliers -> activation outliers -> attention outliers), and argues via hypotheses, controlled GPT-2 attention variants, and theory that softmax's sum-to-one constraint forces large dynamic ranges that create outliers; they act as implicit context-aware scaling factors, and an explicit learnable scaling factor remo

Key results

• Context-aware scaling reduces GPT-2 perplexity under 8-bit AbsMax quantization from 93.44 to 29.22 on WikiText2.
• With 50% unstructured magnitude pruning, GPT-2 default reaches PPL 7235.68 while the context-aware scaling variant reaches 39.47.
• Activation and attention outliers overlap in 95% of cases in sequence dimensions, and weight and activation outliers align 100% in feature dimensions; the scaling variant adds under 0.1% parameters (123.59M to 123.70M).

Measured effects reported in the paper

Surprising findings

• Fixed or context-aware bias alone does not prevent outliers; only attention bias with a scaling term or an explicit scaling factor does.
• The context-aware scaling variant converges faster early but reaches final validation loss comparable to default attention.

Limitations

• Mechanism is argued via hypotheses and derivations rather than fully proven; the paper acknowledges that capturing full training dynamics is beyond its analysis.
• Scaling-factor intervention is validated mainly on small GPT-2 (124M) models, with a TinyLLaMA-120M check in the appendix.
• Final validation loss gains are not observed; benefits are mainly faster early convergence and compression robustness.

Experimental setup

Datasets: RedPajama WikiText2

Benchmarks: WikiText2

Metrics: perplexity outlier magnitude extremal ratio overlap of activation and attention outliers training loss

Baselines: Default attention (GPT-2) Explicit fixed bias Explicit context-aware bias Attention bias Sigmoid attention

Builds on: Massive Activations Attention Sink sigmoid attention NanoGPT TinyLlama

Closest papers by meaning

• Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models (ICLR 2024 · similarity 0.90)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.89)
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs (NeurIPS 2024 · similarity 0.89)
• Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.89)
• From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.89)
• Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity (ICML 2024 · similarity 0.89)