Research map / Architectures and efficiency / Attention and transformers
Frayed RoPE and Long Inputs: A Geometric Perspective
Davis Wertheimer, Aozhong Zhang, Derrick Liu, Penghang Yin, Naigang Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RoPE-ID (high-frequency RoPE on half the channels) with length-based temperature scaling | RoPE-ID on Llama-1B · RULER average, length 16k | accuracy: 0.03 -> 30.83 | Improved |
| add RoPE-ID (scaling) versus tuning-free YaRN at 1B | RoPE-ID on Llama-1B · RULER average, length 16k | accuracy: 30.25 -> 30.83 | Improved |
| RoPE-ID (scaling) versus YaRN at 3B, 8k | RoPE-ID on Llama-3B · RULER average, length 8k | accuracy: 45.09 -> 43.39 | Hurt |
| add input-length temperature scaling to RoPE-ID (ablation) | RoPE-ID on Llama-1B · RULER average, length 16k | accuracy: 14.29 -> 30.83 | Improved |
| vary temperature scaling exponent from 2.0 (ablation) | RoPE-ID on Llama-1B · RULER average, length 16k | accuracy: 30.83 -> 31.15 (2.5) | No clear effect |
| apply RoPE to half the channels without raising frequency (HalfRoPE) | RoPE-ID on Llama-1B · RULER average, length 8k | accuracy: 43.07 (4k) -> 0.14 (8k) | Hurt |
| raise RoPE frequency on all channels (High frequency) | RoPE-ID on Llama-1B · RULER average, length 4k | accuracy: 39.72 -> 16.04 | Hurt |
| raise lowest frequency so wavelength is 2048 instead of 4096 (ablation) | RoPE-ID on Llama-1B · RULER average, length 16k | accuracy: 16.07 -> 30.83 | Improved |
| reduce fraction of RoPE channels from 1/2 to 1/8 (ablation) | RoPE-ID on Llama-1B · RULER average, length 16k | accuracy: 30.83 -> 2.18 | Hurt |
| reduce fraction of RoPE channels from 1/2 to 1/8 at short context (ablation) | RoPE-ID on Llama-1B · RULER average, length 4k | accuracy: 39.15 -> 46.30 | Improved |
| highest-frequency wavelength 64 instead of 32 (ablation) | RoPE-ID on Llama-1B · RULER average, length 16k | accuracy: 30.84 -> 9.46 | Hurt |
| RoPE-ID versus RoPE on short commonsense tasks | RoPE-ID on Llama-3B · commonsense average | accuracy: 51.96 -> 52.08 | No clear effect |
Closest papers by meaning
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.93)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.91)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.90)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.90)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.89)
- LinearARD: Linear-Memory Attention Distillation for RoPE Restoration (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
