Research map / Large language models / Retrieval and knowledge

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding

ICML 2025 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

The paper shows that massive values in attention queries and keys (not values) concentrate in specific dimensions due to RoPE, and that disrupting them mainly harms contextual knowledge understanding while sparing parametric knowledge retrieval.

Problem

Understanding why massive activation values concentrate in attention queries and keys of RoPE-based LLMs, and what functional role they play in contextual versus parametric knowledge processing and quantization robustness.

Approach

Defines massive values in Q and K via a norm-based threshold, shows they concentrate in low-frequency RoPE dimensions consistently across heads and layers, and tests their role by replacing them with mean/zero/min values during prefilling, comparing contextual versus parametric tasks.

Key results

• Disrupting massive values in Llama3-8B drops GSM8K accuracy from 76.90% to 4.00% and passkey (128,6) from 100% to 9%, while non-massive disruption keeps GSM8K at 77.40%.
• Disrupting massive values keeps parametric retrieval relatively high, e.g., Cities 88.20% for Llama3-8B versus 99.40% vanilla.
• Llama3-8B IMDB perplexity rises from 8.38 to 112.61 when massive values are disrupted, versus 8.94 for non-massive disruption.

Measured effects reported in the paper

Surprising findings

• With massive values disrupted, models on the conflicting-context Cities task score above chance, suggesting they fall back on parametric knowledge and ignore misleading context.
• Massive values appear already in the first layers and are present in Q and K before RoPE is applied to them, implying they are formed through training rather than solely by RoPE.

Limitations

• Causal claim about RoPE rests on comparisons across architectures and hypotheses rather than controlled ablation.
• Synthetic factual QA dataset is small (200 per category) and verified partly by LLMs and a manual pass rate of 89-92%.

Experimental setup

Datasets: GSM8K AQUA IMDB Cities Passkey retrieval (synthetic) Synthetic factual QA (Sports, Arts, Technology, Celebrity)

Benchmarks: GSM8K AQUA IMDB Passkey Retrieval Cities Sports Art Technology Celebrity

Metrics: accuracy perplexity 2-gram diversity

Baselines: AWQ SmoothQuant GPTQ OPT GPT-2

Builds on: RoPE AWQ SmoothQuant GPTQ Round and Round we go

Closest papers by meaning

• Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.88)
• Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
• Mixture of In-Context Experts Enhance LLMs' Long Context Awareness (NeurIPS 2024 · similarity 0.88)
• Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.88)
• Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.88)
• Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.88)