Research map / Large language models / Language, safety and interpretability

Interpreting the Repeated Token Phenomenon in Large Language Models

ICML 2025 · Large language models › Language, safety and interpretability · approach: Task vectors and steering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Mechanistically explains why LLMs fail to repeat a single token by linking the failure to attention-sink neurons triggered by repeated tokens, introduces a "cluster attack" and a targeted neuron patch that mitigates it.

Problem

LLMs often fail to accurately repeat a single token when asked, outputting unrelated text (repeated token divergence), an unexplained vulnerability that can leak training data.

Approach

Shows repeated tokens acquire attention-sink-like high norms via the same sink neurons that mark the BoS token, because the first attention layer marks any single-token sequence as "first"; this yields a cluster attack using non-repeated same-cluster tokens and a simple neuron-zeroing patch.

Key results

• Patching a single sink neuron (LLaMA-2 neuron 7890, layer 1) changes benchmark accuracy by at most about 1 point (e.g., MMLU +0.97, TruthfulQA -1.83 for LLaMA-1), indicating negligible side effects.
• Repeated-token sink neurons appear after roughly 450-4000 repeats across LLaMA-1, LLaMA-2, Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.1 (Table of sink neurons).
• Theory shows the last-token representation of n repeats converges to that of a single token at rate O(1/n) under bounded positional encodings such as RoPE.

Measured effects reported in the paper

Surprising findings

• Some tokens (e.g., 'Another', 'bit', 'dust') do not induce attention sinks even when repeated.
• Simple surface-level filters for repeated-token prompts are bypassed by the cluster attack using non-repeated tokens from the same cluster.

Limitations

• Cluster attack success was not quantified because training-data leakage is hard to assess.
• The first-attention-layer mechanism was found to be unique to LLaMA-2, and sink induction varies across tokens and models.
• The mitigation does not fully address the inherent vulnerability of causally masked transformers to repeated tokens.

Experimental setup

Datasets: MMLU HellaSwag TruthfulQA WinoGrande AI2-ARC The Pile Tiny Shakespeare

Benchmarks: MMLU HellaSwag TruthfulQA WinoGrande AI2-ARC

Metrics: accuracy difference before and after patch L2 norm of hidden states

Baselines: original unpatched models

Builds on: attention sinks repeated token divergence (Nasr et al. 2023) massive activations RoPE

Closest papers by meaning

• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.89)
• When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.88)
• Finding Interpretable Prompt-Specific Circuits in Language Models (NeurIPS 2026 · similarity 0.87)
• The emergence of sparse attention: impact of data distribution and benefits of repetition (NeurIPS 2025 · similarity 0.87)
• On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.87)
• Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.87)