Research map / Large language models / Instruction and fine-tuning

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

ICML 2024 · Large language models › Instruction and fine-tuning · approach: Calibration and knowledge in LLM tuning · 15 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends pre-trained LLaMA2-7B and Mistral-7B context windows to 2048k tokens via evolutionary search for non-uniform RoPE rescale factors, progressive 256k fine-tuning, and short-context readjustment.

Problem

Pre-trained LLMs with RoPE positional embeddings have limited context windows (around 128k even after extension); extending further causes catastrophic out-of-distribution position values, scarce long training data, high fine-tuning cost, and degraded short-context performance.

Approach

Searches non-uniform RoPE rescale factors (per dimension, plus a token-position threshold for initial tokens) with an evolutionary algorithm using a monotonic constraint, then applies a progressive strategy (search and fine-tune to 256k, then a second search to 2048k without fine-tuning) and readjusts factors for short windows.

Key results

• LongRoPE-2048k on LLaMA2-7B and Mistral-7B extends context to 2048k tokens with perplexity still near 7-8 (LLaMA2 ft=256k: 7.08 at 2048k on Books3 for Books3 metric) and 1.88 Proof-pile perplexity at 256k.
• Passkey retrieval accuracy stays at or above 90% from 4k to 2048k for LongRoPE-LLaMA2-2048k (ft=256k), while existing models drop to 0 beyond 128k.
• Without fine-tuning, the searched RoPE extends LLaMA2-7B's context 8x (4k to 32k), while PI, NTK and YaRN perplexity spikes after 2x.

Measured effects reported in the paper

Surprising findings

• YaRN underperforms PI and Dynamic-NTK on PG19 without fine-tuning because it does not reach the target length, with perplexity spiking after 7k in an 8k setting.
• Mistral's 2048k model exceeds perplexity 7 beyond 256k, while LLaMA2 perplexity decreases with longer context, which the authors attribute to using 16k training length for Mistral.

Limitations

• Perplexity evaluation at 2048k takes about 50 minutes, which constrains search iterations.
• Fine-tuning and inference beyond 512k are hard to serve with GPU memory and compute limits.
• Mistral fine-tuning used 16k sequences, which limited further extension; the initial-token non-uniformity did not help at 2048k.

Experimental setup

Datasets: RedPajama PG19 Proof-pile Pile-Books3 Together Long-Data Collections

Benchmarks: Proof-pile PG19 Books3 passkey retrieval Hugging Face Open LLM Leaderboard (ARC-Challenge, HellaSwag, MMLU, TruthfulQA)

Metrics: perplexity passkey retrieval accuracy accuracy

Baselines: Position Interpolation (Together-32k) Code LLaMA LongLoRA YaRN Dynamic-NTK MistralLite Mistral v0.1

Builds on: Position Interpolation NTK-aware interpolation YaRN Rotary Position Embedding

Closest papers by meaning

• LongRoPE2: Near-Lossless LLM Context Window Scaling (ICML 2025 · similarity 0.93)
• An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.92)
• Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.91)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.91)
• Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.91)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.91)