Research map / Large language models / Language, safety and interpretability
Base of RoPE Bounds Context Length
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that RoPE's base sets an absolute lower bound on the context length a model can truly use, derived from a long-term decay property of similarity-vs-random attention, and validates it via fine-tuning and pre-training experiments.
Problem
Long-context extension via RoPE base adjustment is usually explained by avoiding out-of-distribution rotation angles, but this can yield superficial long-context ability that keeps perplexity low while failing retrieval. The paper asks what determines the usable context length as a function of RoPE
Approach
Derives a long-term decay property: the advantage of attending to similar keys over random keys, B_{m,θ}=Σcos(mθ_i), must stay non-negative for context length L, which gives an absolute lower bound on RoPE's base for each target context length (e.g., 4.3e3 for 1k, 6.4e5 for 32k).
Key results
- Fine-tuning Llama2-7B on 32k with base below the derived lower bound causes Long-eval accuracy to drop sharply while perplexity stays low.
- A 2B model pre-trained from scratch with RoPE base=100 retrieves information only from roughly the most recent 500 tokens despite 4,096 training length.
- Method 2 (no OOD angles) gets Long-eval 0.00 at 30k with 2554 positions of B_m≥0 versus Method 1's 0.27 at 30k, showing OOD theory alone is insufficient.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a small RoPE base (500) instead of lower bound-satisfying base | Llama2-7B fine-tuned to 32k · Perplexity (PG19) and Long-eval / NIH retrieval | retrieval accuracy | Hurt |
| Increase RoPE base above lower bound (e.g., 2e5 vs 1e4) when fine-tuning on 32k | Llama2-7B-Base fine-tuned on 32k · Long-eval 32k | Long-eval accuracy | Helped |
| Pre-train from scratch with RoPE base 100 instead of larger base | 2B model pre-trained at 4k · Effective context length (NIH) | effective context length | Hurt |
| Increase RoPE base from 1e4 to 1e6 when fine-tuning 2B model on 32k | 2B pre-trained model · Effective context length (NIH) | effective context length | Helped |
| Ensure B_m,theta >= 0 across relative distances (large enough base) versus violating it (Method 2) with OOD avoided in both (ablation) | Llama2-7B extended 4k to 32k · Long-eval 30k | Long-eval accuracy: 0.27 vs 0.00 | Helped |
Surprising findings
- Lowest-perplexity configuration (base=2e5) had very low Long-eval accuracy, showing perplexity is a poor proxy for long-context ability.
- Small-base RoPE can keep perplexity low even at 128k while failing to retrieve information at 1k.
Limitations
- The derived lower bound is not strict because stacked layers can extract information beyond single-layer range; bound is computed numerically rather than in closed form.
- Upper bound on RoPE base is left open.
- Lack of effective long-context benchmarks limits the scope of capability evaluated.
Experimental setup
Datasets: RedPajama subset PG19
Benchmarks: PG19 perplexity Long-eval Needle in Haystack
Metrics: perplexity retrieval accuracy effective context length
Baselines: Position Interpolation NTK-aware scaling YaRN Dynamic NTK Llama2-7B with base 500 (prior small-base work)
Builds on: RoPE Position Interpolation NTK-aware scaling YaRN
Closest papers by meaning
- Scaling Laws of RoPE-based Extrapolation (ICLR 2024 · similarity 0.91)
- YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.91)
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.91)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.90)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.90)
- What Rotary Position Embedding Can Tell Us: Identifying Query and Key Weights Corresponding to Basic Syntactic or High-level Semantic Information (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.