Research map / Large language models / Language, safety and interpretability

Why Does the Effective Context Length of LLMs Fall Short?

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes StRing, a training-free inference-time method that shifts well-trained relative positions onto the undertrained tail positions to improve effective context length within the training window, lifting Llama3.1 70B and Qwen2 72B on RULER and InfiniteBench.

Problem

Open-source LLMs' effective context length typically falls below half their training length; the authors attribute this to a left-skewed frequency distribution of relative positions seen during training, which leaves long-distance positions undertrained.

Approach

During inference, drop the infrequent tail of relative positions (beyond a threshold N) and shift the frequent positions from the main diagonal toward the bottom-left corner with offset S, plus a small local window W; implemented with FlashAttention as sliding-window attention plus shifted causal self-attention merged by log-sum-exp style weighting.

Key results

• On NIAH 4-needle across seven base models, StRing averages 85.7 versus 67.8 for RoPE and 73.1 for the next best method, DCA.
• On RULER at 128K, Llama3.1 70B with StRing reaches 81.7 average versus 66.6 for baseline, and Qwen2 72B reaches 84.6 versus 53.7 for baseline, surpassing GPT-4-1106-preview (81.2).
• On InfiniteBench, Llama3.1 70B with StRing averages 56.88 versus 45.25 with RoPE and outperforms GPT-4 (55.69), Claude2 (47.96) and Kimi-chat (43.91).

Measured effects reported in the paper

Surprising findings

• Extrapolation baselines (ReRoPE, Self-Extend, DCA, YaRN) largely fail to improve Llama3.1 8B on RULER, while StRing does.
• Frequency of position indices, not maximum training length alone, appears to determine effective context length in the TinyLlama probing experiments.

Limitations

• Pretraining experiments only cover context lengths below 4K tokens, so long-context training effects are not studied.
• Hyperparameters S and W were chosen by heuristics and ablations on NIAH; the method's benefits from a training-side fix are left to future work.
• Shifting positions may disrupt reasoning ability unless the training data distribution is similar to the original corpus.

Experimental setup

Datasets: SlimPajama-627B Paul Graham Essays (NIAH haystack) RULER InfiniteBench Needle-in-a-Haystack

Benchmarks: Needle-in-a-Haystack (4-needle) RULER InfiniteBench

Metrics: accuracy average score across RULER tasks InfiniteBench task scores

Baselines: RoPE NTK-Aware RoPE YaRN ReRoPE Self-Extend DCA GPT-4-128K Claude 2 Kimi-chat

Builds on: RoPE FlashAttention NTK-aware RoPE YaRN DCA

Closest papers by meaning

• An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.92)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.91)
• YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.91)
• Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.91)
• Exploring Context Window of Large Language Models via Decomposed Positional Vectors (NeurIPS 2024 · similarity 0.91)
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.90)