Research map / Architectures and efficiency / Attention and transformers
Functional Interpolation for Relative Positions improves Long Context Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FIRE, a learnable functional relative position encoding that normalizes query-key distances by a learnable-threshold progressive interpolation, enabling zero-shot length generalization for decoder-only Transformers.
Problem
Transformer language models degrade on inputs longer than their training context; existing relative position encodings either saturate, are biased toward locality, are slow on accelerators, or require knowing the target length in advance (interpolation methods).
Approach
Learn the relative position bias with a tiny MLP applied to the query-key distance normalized by the (log-transformed, thresholded) query position, so the MLP input stays in [0,1] for any sequence length; the paper proves this can represent T5 RPE, Alibi, Kerple and Sandwich biases.
Key results
- On C4 base models trained at length 2048, FIRE reaches 21.24 perplexity at length 8192 versus 23.52 for the best baseline (Kerple).
- On SCROLLS, FIRE base averages 25.64 vs 24.14 for T5's RPE and 23.23 for RoPE-PI; large averages 27.05.
- FIRE on GLUE/SuperGLUE (length 1024) averages 71.14, on par with RoPE, Kerple and T5's RPE, showing no short-context sacrifice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FIRE instead of Kerple | 125M language model · C4 validation, sequence length 8192 | perplexity: 21.24 v.s. 23.52 perplexity | Helped |
| use FIRE instead of Kerple | base model · SCROLLS | Average: 24.11 -> 25.64 | Helped |
| use FIRE instead of RoPE-PI | large model · SCROLLS | Average: 26.04 -> 27.05 | Helped |
| add the log transform to relative distances (ablation) | FIRE · C4 validation, sequence length 8192 | log perplexity: 3.334 -> 3.085 | Helped |
| add thresholding to the normalizer (ablation) | log-transformed FIRE · C4 validation, sequence length 8192 | log perplexity: 3.085 -> 3.056 | Helped |
| add thresholding to the normalizer (ablation) | log-transformed FIRE · GLUE/SuperGLUE | Average accuracy: 70.42 -> 71.14 | Helped |
| use a two-hidden-layer MLP instead of a linear layer (ablation) | FIRE · C4 validation, sequence length 8192 | log perplexity: 3.32 -> 3.06 | Helped |
| increase the function approximator from one to two hidden layers (ablation) | FIRE · C4 validation, sequence length 8192 | log perplexity: 3.08 -> 3.06 | Helped |
| use GeLU instead of ReLU activation (ablation) | FIRE · C4 validation, sequence length 8192 | log perplexity: 3.06 -> 3.06 | No clear effect |
| add an activation to the final MLP output (ablation) | FIRE · C4 validation, sequence length 8192 | log perplexity: 3.08 -> 3.19 | Hurt |
| share the position encoding function across layers (ablation) | FIRE · C4 validation, sequence length 8192 | log perplexity: 3.06 -> 3.10 | Hurt |
| share the position encoding function across layers (ablation) | FIRE · GLUE/SuperGLUE | Average accuracy: 71.14 -> 71.04 | Hurt |
Surprising findings
- Linear-layer FIRE still uses non-linearity benefits: a one-hidden-layer MLP gives large gains over linear f_theta on long sequences.
- FIRE without the learnable threshold already outperforms all baselines on C4 at all lengths.
Limitations
- Only decoder-only models are studied.
- Does not analyze the role of other Transformer components, data, or optimizer in length generalization.
- YaRN and RoPE-PI require target length knowledge or finetuning, while FIRE was only evaluated zero-shot on perplexity.
Experimental setup
Datasets: C4 arXiv GitHub SCROLLS NarrativeQA GLUE SuperGLUE
Benchmarks: C4 arXiv GitHub SCROLLS NarrativeQA GLUE SuperGLUE
Metrics: log perplexity F1 EM Rgm average accuracy
Baselines: NoPE RoPE RoPE-PI Alibi Kerple T5's RPE YaRN Sandwich
Builds on: T5 relative position bias ALiBi Kerple RoPE position interpolation YaRN
Closest papers by meaning
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.89)
- Exploring Transformer Extrapolation (AAAI 2024 · similarity 0.88)
- Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding (NeurIPS 2025 · similarity 0.88)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.87)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.87)
- How Do Position Encodings Affect Length Generalization? Case Studies On In-Context Function Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.