Research map / Optimization / Implicit bias and generalization

Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: In-context learning as gradient descent · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that attention sinks induce concentrated backward gradients ("gradient sinks") in pre-norm Transformers, interprets massive activations as RMSNorm-mediated regulators of that pressure, and proposes V-scale, a value-path gradient valve that suppresses massive activations while preserving attention sinks.

Problem

Existing explanations of attention sinks and massive activations focus on the forward pass, leaving unexplained why standard pre-norm Transformers learn extreme activations at sink tokens during training.

Approach

Under causal masking, sink tokens aggregate value- and key-path gradients from many later tokens (gradient sinks); because RMSNorm's Jacobian scales inversely with input norm, massive activations act as adaptive local attenuators of this gradient pressure. Attenuating value-path gradients via V-scale should therefore reduce massive activations while keeping attention sinks.

Key results

• In the 0.3B V-scale model, median gradient-sink ratio at token 0 drops from about 40.3 (baseline) to 5.6 while attention sinks are preserved.
• On 1B multi-key NIAH with bfloat16 at 8192 tokens under YaRN, V-scale reaches 62.00% vs 48.32% for the baseline.
• On 1B standard evaluation, V-scale matches baseline perplexity (WikiText 22.83 vs 22.84) and gains on ARC-C (27.05 vs 24.91) and BoolQ (56.27 vs 51.38) under bfloat16.

Surprising findings

• Gains in long-context multi-key retrieval appear most strongly under YaRN extension, where the baseline degrades most.
• Single-needle variant results are mixed: some YaRN-extended settings degrade under V-scale, e.g. AWQ at 4096 tokens.

Limitations

• Study restricted to Llama-like dense language models; transfer to MoE or multimodal Transformers untested.
• Does not exclude other factors such as data, optimizer, or numerical precision shaping AS and MA.
• Pretrained-model gradient measurements are post-hoc diagnostics rather than the original training gradients.

Experimental setup

Datasets: C4 GPT-2 tokenizer

Benchmarks: WikiText perplexity LAMBADA ARC-C ARC-E BoolQ COPA HellaSwag OpenBookQA PIQA SciQ WinoGrande multi-key Needle-in-a-Haystack

Metrics: perplexity accuracy retrieval accuracy sink rate gradient sink ratio activation norm

Baselines: Baseline Llama-like model (matched scratch-trained) SmoothQuant BitsAndBytes GPTQ AWQ

Builds on: StreamingLLM Llama RMSNorm YaRN

Closest papers by meaning

• On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
• Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.88)
• Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · similarity 0.87)
• In-context Convergence of Transformers (ICML 2024 · similarity 0.87)
• How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression (NeurIPS 2024 · similarity 0.87)
• Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention (NeurIPS 2026 · similarity 0.86)