Research map / Large language models / Instruction and fine-tuning
Scaling Laws of RoPE-based Extrapolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that fine-tuning RoPE-based LLaMA2 with either smaller or larger rotary bases (vs. 10000) at the training length improves extrapolation, and explains this with a periodic "scaling law" and a critical dimension.
Problem
RoPE-based LLMs fail to extrapolate beyond their training context length; prior work adjusts the rotary base only upward, and the relationship between base, tuning length, and extrapolation limit is not understood.
Approach
Fine-tuning with both smaller and larger rotary bases improves extrapolation; a periodic view yields theorems linking the base to the extrapolation upper bound (larger bases) or to a monotone improvement (smaller bases), and a critical dimension where RoPE rotary periods exceed the training length explains the failure.
Key results
- Fine-tuning LLaMA2 7B with base 500 yields strong extrapolation over 48K context, outperforming Dynamic NTK.
- Fine-tuning with base 1000000 at 4K tuning length extends LLaMA2 7B and 13B beyond 100K context, with the upper bound matching the predicted T_extra.
- Base 500 with 16K tuning, log-scaled attention, reaches about 19.07 perplexity at 1M context for LLaMA2 7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with RoPE base=500 instead of base=1000000 | LLaMA2 7B · Books3, 1M | perplexity: 4914.81 -> 51.28 | Helped |
| add log-scaled attention to RoPE base=500 (ablation) | LLaMA2 7B with RoPE base=500 · Books3, 1M | perplexity: 51.28 -> 19.07 | Helped |
| increase RoPE base from 500 to 1000000 | LLaMA2 7B · Books3, 128K | perplexity: 9.49 -> 7.07 | Helped |
| increase RoPE base from 500 to 1000000 | LLaMA2 7B · Books3, 256K | perplexity: 12.41 -> 76.82 | Hurt |
| increase RoPE base from 500 to 1000000 | LLaMA2 7B · Books3, 512K | perplexity: 23.99 -> 1755.87 | Hurt |
| fine-tune with a smaller RoPE base, such as 500 | LLaMA2 7B · Books3, 1M | perplexity: 4914.81 -> 51.28 | Helped |
| trim the final 36 dimensions of query and key vectors before fine-tuning (ablation) | LLaMA2 7B · extrapolation beyond 20K | perplexity | Helped |
| set a max index of 4096 for the final 36 dimensions (ablation) | LLaMA2 7B query and key vectors · long-context extrapolation | attention score variation | Helped |
| fine-tune with RoPE base=500 instead of base=10000 | LLaMA2 · extrapolation beyond training context | perplexity | Helped |
| fine-tune with a larger RoPE base | LLaMA2 · within the extrapolation upper bound | perplexity | Helped |
| fine-tune with a larger RoPE base beyond its extrapolation upper bound | LLaMA2 · beyond the extrapolation upper bound | perplexity | Hurt |
Surprising findings
- Base 10000, the default, is the worst base for extrapolation after fine-tuning at the original training length; both smaller and larger bases improve it.
- Fine-tuning with base 1000000 at the original 4K length extrapolates beyond 100K, without longer tuning data.
Limitations
- Smaller bases degrade short-context performance on some tasks (e.g., base 500 drops on Lambada and TopicRetrieval).
- Extrapolation with smaller bases is not uniform and perplexity still worsens with length; no hard upper bound is shown for small bases, but practical limits exist.
- Experiments limited to LLaMA2 models and Books3 perplexity for main analysis.
Experimental setup
Datasets: The Pile Books3
Benchmarks: Books3 validation perplexity Lambada SuperGLUE Hellaswag PIQA Winogrande OBQA NQ TriviaQA TruthfulQA ARC MMLU
Metrics: perplexity accuracy
Baselines: Linear PI NTK fixed Dynamic NTK Code LLaMA
Builds on: RoPE NTK-aware scaling Dynamic NTK Linear Position Interpolation
Closest papers by meaning
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.91)
- Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs (NeurIPS 2024 · similarity 0.91)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.90)
- YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.89)
- LongRoPE2: Near-Lossless LLM Context Window Scaling (ICML 2025 · similarity 0.88)
- ParallelComp: Parallel Long-Context Compressor for Length Extrapolation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.