Research map / Architectures and efficiency / Attention and transformers

Round and Round We Go! What makes Rotary Positional Encodings useful?

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how a trained Gemma 7B uses RoPE frequencies, showing high frequencies build positional heads and low frequencies act as semantic channels, and proposes p-RoPE, which truncates the lowest RoPE frequencies and improves perplexity for Gemma 2B models.

Problem

Why RoPE is useful in LLMs is poorly understood; the common belief that it helps decay attention with relative distance relies on unrealistic constant queries and keys.

Approach

Mechanistic analysis of Gemma 7B's RoPE frequency usage shows the highest frequencies create robust positional attention, while the lowest frequencies carry semantic information but cannot be robust over long contexts; truncating the lowest frequencies (p-RoPE) gives robust semantic channels.

Key results

• p-RoPE with 0.75 kept frequencies reaches 4.4414 Wiki and 6.4422 FlanV2 validation perplexity, versus 4.4627 and 6.4429 for standard RoPE (theta=10k) at Gemma 2B scale.
• NoPE is clearly worse, with 4.8594 Wiki and 6.6429 FlanV2 perplexity.
• Gemma 7B's key/query norm is concentrated on the lowest RoPE frequencies at most layers, while the first and last layers also use high frequencies.

Measured effects reported in the paper

Surprising findings

• Removing a fraction of the lowest RoPE frequencies maintains or slightly improves validation perplexity.
• The common claim that RoPE decays attention with distance does not hold for Gaussian random queries and keys.

Limitations

• Experiments with p-RoPE use only an 8k context and small Gemma 2B models; long-context benefits are untested.
• Analysis focuses mainly on Gemma 7B, with Llama 3.1 8B results only in the appendix.
• Theoretical results cover only the single-frequency (d=2) case for the instability claim.

Experimental setup

Datasets: Wiki FlanV2 Shakespeare quotes

Benchmarks: Wiki validation perplexity FlanV2 validation perplexity

Metrics: validation perplexity mean 2-norm per RoPE frequency chunk attention coefficients

Baselines: NoPE RoPE RoPE theta=500k 0.75-RoPE reversed RoPE partial

Builds on: RoPE NoPE ALiBi Llama 3 long-context base wavelength increase

Closest papers by meaning

• PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.92)
• What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.91)
• Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.90)
• Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.90)
• Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.88)
• YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.88)