Research map / Architectures and efficiency / Attention and transformers
Relative Positional Encoding for Transformers with Linear Complexity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace APE with sineSPE | Linear Transformer-ReLU · Long-Range Arena: Retrieval | accuracy: 58.78 ± 0.93 -> 62.39 ± 0.59 | Helped |
| Replace APE with sineSPE | Linear Transformer-ReLU · Long-Range Arena: ListOps, Text, Retrieval, and Image | accuracy | Mixed |
| Replace APE with convSPE | Linear Transformer-ReLU · Long-Range Arena: ListOps, Text, Retrieval, and Image | accuracy | Hurt |
| Use SPE instead of APE | Performer music generation · Pop piano music generation, positions beyond 2,048 | validation cross-entropy | Helped |
| Use SPE instead of APE | Performer groove continuation · Groove continuation | time-pitch style similarity | Helped |
| Use SPE instead of APE | Performer groove continuation · Groove continuation | onset-duration style similarity | Helped |
| Use non-vanishing sinusoidal relative attention rather than vanishing convolutional relative attention | SPE on Long-Range Arena · Long-Range Arena | accuracy | Mixed |
Closest papers by meaning
- Algebraic Positional Encodings (NeurIPS 2024 · similarity 0.88)
- DAPE: Data-Adaptive Positional Encoding for Length Extrapolation (NeurIPS 2024 · similarity 0.86)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.86)
- Toward Relative Positional Encoding in Spiking Transformers (NeurIPS 2025 · similarity 0.86)
- RelFlexformer: Efficient Attention Transformers for Integrable Relative Positional Encodings (NeurIPS 2026 · similarity 0.85)
- A Formal Framework for Understanding Length Generalization in Transformers (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.