Research map / Architectures and efficiency / Attention and transformers
Selective Rotary Position Embedding
Sajad Movahedi, Timur Carstensen, Arshia Afzal, Frank Hutter, Antonio Orvieto, Volkan Cevher
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace RoPE with Selective RoPE | FoX 370M · lm-eval-harness (zero-shot) | average accuracy: 45.1 -> 46.1 | Improved |
| replace RoPE with Selective RoPE | FoX 370M · Wikitext | perplexity: 25.29 -> 33.87 | Hurt |
| replace RoPE with Selective RoPE | GLA 370M · lm-eval-harness (zero-shot) | average accuracy: 46.5 -> 47.1 | Improved |
| replace RoPE with Selective RoPE | GLA 370M · Wikitext | perplexity: 26.51 -> 25.98 | Improved |
| replace NoPE with Selective RoPE | GLA 370M · Wikitext | perplexity: 25.72 -> 25.98 | No clear effect |
| replace RoPE with Selective RoPE | Gated DeltaNet ~400M · lm-eval-harness (zero-shot) | average accuracy: 46.7 -> 47.5 | Improved |
| replace NoPE with Selective RoPE | GLA 1.3B · lm-eval-harness (zero-shot) | average accuracy: 55.2 -> 54.6 | Hurt |
| replace RoPE with Selective RoPE | GLA 1.3B · Wikitext | perplexity: 18.50 -> 17.87 | Improved |
| replace RoPE with Selective RoPE | GLA 1.3B · LAMBADA | perplexity: 8.88 -> 8.50 | Improved |
| add SiLU activation after the convolution following the omega projection (ablation) | Selective RoPE GLA · language modeling | performance | Improved |
| add a gate on the rotation (phase) (ablation) | Selective RoPE GLA · language modeling | performance | Improved |
| add a learnable bias term to rotation angles (ablation) | Selective RoPE GLA · language modeling | performance | No clear effect |
Closest papers by meaning
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.91)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.90)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.90)
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.88)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.87)
- Revisiting Multimodal Positional Encoding in Vision–Language Models (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
