Research map / Architectures and efficiency / Attention and transformers
DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DAPE, a data-adaptive additive positional encoding that feeds attention logits and a fixed positional bias (Alibi, Kerple, FIRE) through a small two-layer MLP to produce input-dependent position biases, improving length extrapolation on Arxiv, Books3 and CHE.
Problem
Static positional encodings (APE, RPE, RoPE, Alibi, Kerple, FIRE) are fixed after training regardless of input, limiting adaptability and length generalization of transformers.
Approach
Adds a learned MLP f(QK^T, B) over the pre-softmax attention logits and the positional bias matrix B, so the positional bias becomes semantically dependent on input context; compatible with any additive RPE and implementable per head with a small hidden width (e.g., 32).
Key results
- On Arxiv with training length 128, DAPE-Kerple reaches perplexity 5.00 at evaluation length 8192 versus Kerple's 31.93.
- On Books3 with training length 512, DAPE-Kerple reaches 17.88 perplexity at length 8192 versus Kerple's 39.31.
- On CHE, DAPE-Kerple and DAPE-FIRE gave the best results on 10 of 11 permutation-variant tasks, and DAPE-Alibi scored 50.30 on Parity Check versus a best of 50.97.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt Kerple bias using DAPE context (attention and position) | Kerple · Arxiv, train 512, eval 8192 | perplexity: 13.3524 -> 3.8642 | Helped |
| adapt Kerple bias using DAPE context (attention and position) | Kerple · Books3, train 512, eval 8192 | perplexity: 39.31 -> 17.88 | Helped |
| adapt attention and positional bias jointly instead of using f(B) alone (ablation) | DAPE bias ablation · Books, train 512, eval 8192 | perplexity: 23.49 -> 17.88 | Helped |
| add a learned bias correction f(B) (ablation) | static bias QK^T+B · Books, train 512, eval 8192 | perplexity: 39.31 -> 23.49 | Helped |
| use concatenation rather than additive residual variant (ablation) | DAPE-Kerple · train length 128, eval 8192 | perplexity: 5.17 -> 5.03 | Helped |
| use concatenation rather than additive residual variant (ablation) | DAPE-Kerple · train length 512, eval 8192 | perplexity: 3.75 -> 3.69 | Helped |
| add residual connection to concatenation variant (ablation) | DAPE-Kerple · train length 128, eval 8192 | perplexity: 5.03 -> 5.00 | No clear effect |
| add residual connection to concatenation variant (ablation) | DAPE-Kerple · train length 512, eval 8192 | perplexity: 3.69 -> 3.70 | No clear effect |
| set DAPE hidden dimension to 4 (ablation) | DAPE-Alibi · Arxiv, train 128, eval 128 | perplexity: 8.31 -> 8.25 | Helped |
| set DAPE hidden dimension to 4 (ablation) | DAPE-Alibi · Arxiv, train 128, eval 8192 | perplexity: 5.85 -> 5.67 | Helped |
Surprising findings
- Adding a learned MLP over the attention logits and bias only (without query-key context, i.e., f(B)) gives less gain than the context-dependent version, but still improves over the plain bias baseline.
- Standard RoPE and T5 bias degrade sharply beyond training length (e.g., RoPE at 265 perplexity at length 8192 when trained on 512 in the three-seed study), while DAPE variants stay stable.
Limitations
- Extra O(N^2) cost from the MLP over the full attention matrix, increasing time and memory with sequence length.
- Evaluation protocol (last-256-token perplexity over full-sequence processing) differs from prior work, making direct comparison with reported baseline numbers difficult.
- Reported results rely on models of 125M to 6.7B parameters and a limited set of datasets; no large-scale pretraining evaluation.
Experimental setup
Datasets: Arxiv Books3 Chomsky Hierarchy Evaluation benchmark (CHE)
Benchmarks: Arxiv perplexity Books3 perplexity CHE
Metrics: perplexity accuracy statistical significance (p-value) time cost (ms)
Baselines: NoPE RoPE YaRN Randomized RoPE T5 bias Alibi Kerple FIRE
Builds on: Alibi Kerple FIRE RoPE randomized positional encoding
Closest papers by meaning
- Exploring Transformer Extrapolation (AAAI 2024 · similarity 0.88)
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.88)
- Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR 2024 · similarity 0.86)
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025 · similarity 0.86)
- Algebraic Positional Encodings (NeurIPS 2024 · similarity 0.86)
- Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.