Research map / Large language models / Instruction and fine-tuning
YaRN: Efficient Context Window Extension of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes YaRN, a RoPE interpolation method combining NTK-by-parts frequency interpolation with attention temperature scaling (plus dynamic scaling at inference) to extend LLaMA and Llama 2 context windows to 64k-128k with little fine-tuning.
Problem
RoPE-based LLMs fail to generalize beyond their pretraining context length, and prior extension methods (Position Interpolation, NTK-aware) either degrade quality or need large amounts of fine-tuning compute.
Approach
Per-dimension RoPE interpolation chosen by wavelength-to-context ratio (no interpolation for high-frequency dims, full interpolation for low-frequency dims, ramp in between), combined with a constant attention logit temperature sqrt(1/t)=0.1 ln(s)+1 implemented as a rescaling of rotary embeddings, and optional dynamic scaling at inference.
Key results
- Llama 2 13B extended with YaRN (s=32) reaches 128k context with sliding-window perplexity 2.24 on 10 Proof-pile documents at 131072 tokens, and passkey accuracy of 99.4% at 128k.
- Fine-tuning on under ~0.1% of pretraining data (400 steps on 64k chunks) gives YaRN models with minimal Open LLM Leaderboard degradation (e.g., 7B ARC-c 52.3 vs 53.1 baseline).
- Without fine-tuning, Dynamic-YaRN achieves lower long-range perplexity than Dynamic-PI on Llama 2, and YaRN ablations on LLaMA 7B show it beats PI, NTK-aware, and NTK-by-parts under the same training budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use YaRN instead of NTK-by-parts (ablation) | LLaMA 7B · Proof-pile, 8192 context | perplexity: 3.62 -> 3.30 | Helped |
| Use YaRN instead of PI (ablation) | LLaMA 7B · Proof-pile, 8192 context | perplexity: 4.64 -> 3.30 | Helped |
| Fine-tune with YaRN for 400 steps rather than use no fine-tuning (ablation) | LLaMA 7B · Proof-pile, 32768 context | perplexity: 3.45 -> 2.77 | Helped |
| Use YaRN instead of PI with 400 fine-tuning steps (ablation) | LLaMA 7B · Proof-pile, 8192 context | perplexity: 4.64 -> 3.30 | Helped |
| Use YaRN instead of NTK-aware with 400 fine-tuning steps (ablation) | LLaMA 7B · Proof-pile, 8192 context | perplexity: 3.73 -> 3.30 | Helped |
| Use YaRN with 400 steps rather than PI with 1000 steps (ablation) | Llama 2 7B · Proof-pile, 8192 context | perplexity: 3.51 -> 3.50 | No clear effect |
| Extend Llama 2 7B to 128k with YaRN rather than 64k | Llama 2 7B · Passkey retrieval | accuracy: 96.3% -> 99.4% | Helped |
| Extend Llama 2 13B to 128k with YaRN rather than 64k | Llama 2 13B · Passkey retrieval | accuracy: 97.5% -> 99.4% | Helped |
| Extend context with YaRN instead of leaving the model unextended | Llama 2 7B · HellaSwag | score: 77.8 -> 78.8 | Helped |
| Extend context with YaRN instead of leaving the model unextended | Llama 2 13B · ARC-Challenge | score: 59.4 -> 58.1 | Hurt |
| Use YaRN instead of Dynamic-PI | pretrained Llama 2 without fine-tuning · GovReport long-range evaluation | perplexity | Helped |
| Use YaRN instead of competing interpolation techniques with the same training budget | LLaMA 7B · Passkey retrieval, prompts up to 32k | accuracy | Helped |
Surprising findings
- Perplexity alone is not a reliable indicator of effective context size: Code Llama 13B has rising perplexity above 100k but still retrieves passkeys at 128k.
- The best attention temperature was found to be consistent across samples and positions and to transfer from LLaMA to Llama 2 models, suggesting some universality.
Limitations
- NTK-aware and related methods require empirically tuned base or scale choices, and the theoretical scale does not match true extension.
- Perplexity-based evaluation does not fully capture effective context usage; passkey retrieval only tests a simple retrieval task.
- ReRoPE and LM-Infinite were not compared because they modify attention and are incompatible with FlashAttention 2.
Experimental setup
Datasets: PG19 GovReport Proof-pile RedPajama
Benchmarks: Proof-pile sliding-window perplexity GovReport perplexity passkey retrieval Hugging Face Open LLM Leaderboard (ARC-c, HellaSwag, MMLU, TruthfulQA)
Metrics: perplexity passkey retrieval accuracy accuracy
Baselines: Position Interpolation NTK-aware interpolation NTK-by-parts Code Llama Together.ai Llama-2-7B-32K LLongMA-2 7B
Builds on: RoPE Position Interpolation NTK-aware interpolation Dynamic NTK FlashAttention-2
Closest papers by meaning
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.91)
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.91)
- PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.90)
- Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.90)
- LongRoPE2: Near-Lossless LLM Context Window Scaling (ICML 2025 · similarity 0.89)
- An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.