Research map / Architectures and efficiency / Attention and transformers
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
Ta-Chung Chi, Ting-Han Fan, Peter Ramadge, Alexander Rudnicky
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use logarithmic relative-position kernel rather than ALiBi | GPT-NeoX 162M · OpenWebText2, GitHub, and ArXiv at lengths 4096–16384 | perplexity: 21.4 $\pm$ 0.6 vs. 22.5 $\pm$ 0.5; 2.75 $\pm$ 0.16 vs. 3.02 $\pm$ 0.13; 5.07 $\pm$ 0.16 vs. 5.78 $\pm$ 0.15 | Improved |
| Use the logarithmic kernel rather than the power kernel (ablation) | KERPLE · OpenWebText2, GitHub, and ArXiv, all tested extrapolation lengths | perplexity: 21.4 $\pm$ 0.6 vs. 21.6 $\pm$ 0.6; 2.75 $\pm$ 0.16 vs. 2.76 $\pm$ 0.13; 5.07 $\pm$ 0.16 vs. 5.07 $\pm$ 0.19 | Improved |
| Use the logarithmic kernel rather than T5 relative bias | GPT-NeoX 162M · OpenWebText2, GitHub, and ArXiv at lengths longer than 2048 | perplexity: OpenWebText2 21.2 vs. 22.5; GitHub 2.74 vs. 2.78; ArXiv 5.20 vs. 5.32 | Improved |
| Use the logarithmic kernel rather than T5 relative bias | GPT-NeoX 162M · OpenWebText2, GitHub, and ArXiv at length 512 | perplexity: 23.9 $\pm$ 0.6 vs. 23.7 $\pmb{\pm}$ 0.6; 3.40 $\pm$ 0.20 vs. 3.38 $\pm$ 0.21; 6.07 $\pm$ 0.26 vs. 6.03 $\pmb{\pm}$ 0.26 | Hurt |
| Use the logarithmic kernel rather than T5 relative bias | GPT-NeoX 162M · GitHub training | sec/step: 0.307 vs. 0.340 | Improved |
| Add a third learnable parameter to the logarithmic kernel (ablation) | KERPLE · OpenWebText2, GitHub, and ArXiv at lengths 512–8192 | perplexity: 21.3 vs. 21.3; 2.68 vs. 2.71; 5.00 vs. 5.01 at length 8192 | Mixed |
| Add a distance-dependent weight kernel alongside the power bias (ablation) | KERPLE · OpenWebText2, GitHub, and ArXiv, lengths 512–8192 | perplexity: 24.1 vs. 23.9; 3.44 vs. 3.40; 6.11 vs. 6.07 at length 512 | Hurt |
| Use Gaussian-like kernels instead of the logarithmic kernel (ablation) | KERPLE · OpenWebText2, GitHub, and ArXiv at extrapolation lengths | perplexity: At length 8192: OpenWebText2 23.2 vs. 21.3; GitHub 3.05 vs. 2.71; ArXiv 5.50 vs. 5.01 | Hurt |
Closest papers by meaning
- Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs (NeurIPS 2024 · similarity 0.87)
- Exploring Transformer Extrapolation (AAAI 2024 · similarity 0.86)
- The Role of Sparsity for Length Generalization in LLMs (ICML 2025 · similarity 0.86)
- Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation (ICML 2024 · similarity 0.86)
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional Encoding (ICML 2025 · similarity 0.86)
- DAPE: Data-Adaptive Positional Encoding for Length Extrapolation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
