Research map / Large language models / Language, safety and interpretability
How Base Frequency Shapes RoPE: An Analytical Study of Frequency-Band Formation
Yui Oka, Itsumi Saito, Kyosuke Nishida, Kuniko Saito
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace lowest-frequency RoPE dimensions with NoPE (r=0.50) (ablation) | Llama-3 · Wikitext-103 L=4096 | perplexity: 2.29 -> 84.50 | Hurt |
| replace lowest 10% frequency dimensions with NoPE (r=0.9) (ablation) | Llama-3 · Wikitext-103 L=4096 | perplexity: 2.29 -> 2.29 | No clear effect |
| replace low-frequency dimensions with NoPE (r=0.9) (ablation) | Phi-3 · Wikitext-103 L=4096 | perplexity: 2.84 -> 46.11 | Hurt |
| replace low-frequency dimensions with NoPE (r=0.9) (ablation) | Gemma · Wikitext-103 L=4096 | perplexity: 2.52 -> 2.70 | Hurt |
| apply YaRN position interpolation | Llama-2 · Wikitext-103 L=4096 | band index: 53.53 -> 51.93 | No clear effect |
| increase base theta from 10000 to 1000000 | RoPE from scratch Ltrain=512 · WikiText-103 | band index: 30.12 -> 15.37 | Improved |
| increase training length from 512 to 2048 | RoPE theta=10000 · WikiText-103 | band index: 30.12 -> 52.12 | Improved |
| set theta equal to training length (FMRoPE) instead of 10000 | RoPE from scratch Ltrain=512 · WikiText-103 inference L=3512 | perplexity: > 100 -> 23.00 | Improved |
| set theta equal to training length (FMRoPE) instead of 10000 | RoPE from scratch Ltrain=512 · WikiText-103 inference L=512 | perplexity: 19.39 -> 19.58 | Hurt |
| add FMRoPE theta=1512 during YaRN fine-tuning instead of theta=10000 | YaRN · WikiText-103 inference L=15512 | perplexity: 52.59 -> 20.51 | Improved |
| add FMRoPE theta=1512 during YaRN fine-tuning instead of theta=10000 | YaRN · WikiText-103 inference L=512 | perplexity: 19.10 -> 19.62 | Hurt |
| enlarge inference base to 2048 while training at theta=1024 | FMRoPE 1B-scale LM · L=2048 | perplexity: 42.36 -> 19.33 | Improved |
Closest papers by meaning
- Scaling Laws of RoPE-based Extrapolation (ICLR 2024 · similarity 0.89)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.88)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.86)
- Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization (ICML 2025 · similarity 0.86)
- A Circular Argument: Does RoPE need to be Equivariant for Vision? (NeurIPS 2025 · similarity 0.85)
- Wavelet-based Positional Representation for Long Context (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
