Research map / Architectures and efficiency / Attention and transformers
Exploring Transformer Extrapolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a sufficient condition for RPE-based length extrapolation (convergence of the series of exp(RPE)), proposes a training-free Theoretical Receptive Field estimate, and validates with small decoder transformers trained from scratch on four corpora.
Problem
Length extrapolation in transformers with relative positional encodings: which RPEs allow models tested on sequences longer than training length to keep stable perplexity, and how to estimate receptive fields without training.
Approach
Shows that an RPE yields length extrapolation whenever the series of exp(RPE) converges (a sliding-window-like hypothesis), derives a training-free Theoretical Receptive Field (TRF) from the RPE formula alone, and builds two new RPEs (1/n^2 and exp(-ln^2 n)) that satisfy the condition.
Key results
- On Wikitext-103 the Type 1 (1/n^2) RPE keeps PPL near 21.9-24.3 from length 512 to 9216, matching ALiBi (22.12 at 9216) while Sinusoidal grows to 423.92.
- The necessity examples 1/n and 1/(n ln n) see PPL rise steadily with length; on Wikitext-103 1/n reaches 58.46 at 9216 versus 24.67 at 512.
- TRF curves computed from the RPE formula alone follow the trend of empirically measured ERF across Alibi, Kerple, Sandwich and the two new RPEs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace sinusoidal positional encoding with Type1 RPE | Transformer language model · Wikitext-103, inference length 512 to 9216 | PPL: 24.73 -> 21.90 | Helped |
| Replace sinusoidal positional encoding with Type1 RPE | Transformer language model · Books, inference length 512 to 9216 | PPL: 7.49 -> 6.99 | Helped |
| Replace sinusoidal positional encoding with Type1 RPE | Transformer language model · Github, inference length 512 to 9216 | PPL: 2.29 -> 2.02 | Helped |
| Replace sinusoidal positional encoding with Type1 RPE | Transformer language model · WikiBook, inference length 512 to 9216 | PPL: 17.98 -> 16.28 | Helped |
| Use nonconvergent 1/n RPE | Transformer language model · Wikitext-103, inference length 512 to 9216 | PPL: 24.67 -> 58.46 | Hurt |
| Use nonconvergent 1/(n ln n) RPE | Transformer language model · Wikitext-103, inference length 512 to 9216 | PPL: 24.64 -> 27.63 | Hurt |
| Use nonconvergent 1/n RPE | Transformer language model · Books, inference length 512 to 9216 | PPL: 7.40 -> 26.34 | Hurt |
| Use nonconvergent 1/(n ln n) RPE | Transformer language model · Books, inference length 512 to 9216 | PPL: 7.35 -> 10.65 | Hurt |
| Use nonconvergent 1/n RPE | Transformer language model · Github, inference length 512 to 9216 | PPL: 2.28 -> 76.78 | Hurt |
| Use nonconvergent 1/(n ln n) RPE | Transformer language model · Github, inference length 512 to 9216 | PPL: 2.27 -> 16.35 | Hurt |
| Use nonconvergent 1/n RPE | Transformer language model · WikiBook, inference length 512 to 9216 | PPL: 17.91 -> 204.10 | Hurt |
| Use nonconvergent 1/(n ln n) RPE | Transformer language model · WikiBook, inference length 512 to 9216 | PPL: 17.90 -> 133.36 | Hurt |
Surprising findings
- Concentrating attention weight on neighboring tokens (e.g., 1/n and 1/(n ln n)) is not sufficient for extrapolation; the series convergence condition is what matters.
- Sliding window inference is more than 44 times slower than nonoverlapping inference, though it has lower perplexity.
Limitations
- The sufficiency proof assumes bounded queries and keys (norm at most l), and the necessity is only checked empirically.
- Models are small (6 or 12 layers) and trained only up to 512 tokens; results on large-scale models are not shown.
- Stabilized PPL levels differ across RPEs due to factors the paper does not study.
Experimental setup
Datasets: Wikitext-103 Books Github WikiBook
Benchmarks: Wikitext-103 Books Github WikiBook
Metrics: perplexity effective receptive field
Baselines: Sinusoidal ALiBi Kerple-Log Kerple-Power Sandwich
Builds on: ALiBi Kerple Sandwich
Closest papers by meaning
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.89)
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025 · similarity 0.89)
- Extrapolation by Association: Length Generalization Transfer In Transformers (NeurIPS 2025 · similarity 0.89)
- Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR 2024 · similarity 0.88)
- DAPE: Data-Adaptive Positional Encoding for Length Extrapolation (NeurIPS 2024 · similarity 0.88)
- Length Generalization for Transformers via Compression (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.