Research map / Large language models / Language, safety and interpretability

Transformers need glasses! Information over-squashing in language tasks

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically and empirically that decoder-only Transformers suffer representational collapse of last-token representations (exacerbated by low precision) and over-squashing of early tokens, explaining failures on counting and copying.

Problem

Decoder-only Transformer LLMs fail on simple copying and counting tasks; the paper asks why, by analysing what information about the input survives in the final-layer last-token representation used for next-token prediction under finite precision.

Approach

Uses signal-propagation analysis of the last token's final representation to prove representational collapse (distinct sequences, e.g. with a repeated final token, yield arbitrarily close representations as length grows) and links causal attention path counting to GNN over-squashing, with bounds on sensitivity to early tokens.

Key results

• Gemini 1.5 begins to output incorrect 'one' when copying the last element of a 0-then-ones sequence already at a sequence length of about 300 elements.
• On Gemini 1.5 counting tasks, the model most frequently outputs 100 once sequences reach around 100 elements, suggesting crude subitising rather than counting.
• In Gemma 7B, last-layer representation differences for repeated-digit sequences approach machine precision at sequence length around 50.

Measured effects reported in the paper

Surprising findings

• Gemini 1.5 copying the last token of a long 1...10 sequence fails at around 300 tokens while the first token is copied correctly, and hints or interleaving help.
• Counting errors concentrate on the output value 100, suggesting subitising-like behaviour.

Limitations

• Theoretical results rely on simplifying assumptions such as attention independent of inputs and constant layer-norm effects.
• Representational collapse proof requires repeated final token and positional encodings decaying with distance; tokenisation complicates experiments.
• Frontier-model experiments are limited to Gemini 1.5 via API and a single open model.

Experimental setup

Metrics: accuracy absolute error of counts L-infinity distance between last-layer representations total variation between softmax distributions

Builds on: over-squashing in GNNs Pre-LN Transformer Chomsky hierarchy analysis of Transformers

Closest papers by meaning

• Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.90)
• Looking Beyond the Top-1: Transformers Determine Top Tokens in Order (ICML 2025 · similarity 0.89)
• Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.89)
• Distilling Sequential Computation in Transformer Language Models (NeurIPS 2026 · similarity 0.89)
• Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.89)
• Transformers Struggle to Learn to Search (ICLR 2025 · similarity 0.88)