Research map / Large language models / Language, safety and interpretability

LongRoPE2: Near-Lossless LLM Context Window Scaling

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LongRoPE2, which extends LLaMA3-8B and Phi3-mini-3.8B to a 128K effective context using evolutionary search over RoPE rescaling factors guided by needle-driven perplexity, plus mixed context window training that preserves short-context performance, using 10B tokens.

Problem

Context window extension of pre-trained LLMs suffers from RoPE out-of-distribution issues, so existing rescaling methods fail to reach the effective target length and degrade short-context performance, and restoring it typically needs costly mid-training.

Approach

Hypothesizes that insufficient training in higher RoPE dimensions lengthens their empirical rotation periods, shifting the real critical dimension earlier; searches rescaling factors with evolutionary search scored by perplexity on only needle answer tokens in synthetic long data, and trains with mixed short (original RoPE, packed with attention masks) and long (rescaled RoPE) windows.

Key results

• LongRoPE2-extended LLaMA3-8B reaches 82.03 on RULER at 128k versus 73.40 for LongRoPE, while retaining 98.6% of short-context performance.
• With 10B training tokens, LongRoPE2 on LLaMA3-8B is reported as 80x fewer tokens than Meta's 800B-token LLaMA3.1-8B-128k extension.
• On Phi3-mini-3.8B, LongRoPE2 scores 58.81 on RULER at 128k versus 53.71 for LongRoPE and retains 97.6% of original short-context performance.

Measured effects reported in the paper

Surprising findings

• Naive PPL-guided search on generic PG19 128k text yields rescaling factors that fail to reach the target effective length, while needle-driven PPL succeeds.
• Mixed context window training improved long-context (8k-128k) RULER performance as well as short-context performance, not just preserving the latter.

Limitations

• Switching to rescaled RoPE at inference requires a one-time KV cache recomputation.
• Evaluated only on two base models (LLaMA3-8B and Phi3-mini-3.8B) and 128k target length; million-token scaling not explored.
• Results are reported without post-training, so absolute scores are lower than instruction-tuned models.

Experimental setup

Datasets: PG19 RedPajama-v1 RedPajama-v2 StarCoder FineWeb-Edu

Benchmarks: RULER Needle in a Haystack LOFT InfiniteBench LongBench MMLU MMLU-Pro HellaSwag TruthfulQA GSM8K

Metrics: accuracy average score perplexity

Baselines: YaRN NTK LongRoPE LLaMA3.1-8B-128k

Builds on: RoPE LongRoPE YaRN NTK-aware scaling

Closest papers by meaning

• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.93)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
• Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.90)
• YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.89)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.89)
• Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.89)