Research map / Large language models / Language, safety and interpretability

Exploring Context Window of Large Language Models via Decomposed Positional Vectors

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Decomposes LLM hidden states into semantic and mean-based positional vectors to analyze how positional information forms and drives context window extension, then proposes two training-free extension methods (positional vector replacement and attention window extension).

Problem

LLMs degrade in performance beyond their context window, and existing length extrapolation and context window extension methods lack an in-depth interpretation of their underlying mechanisms in hidden states.

Approach

Mean-based decomposition separates positional vectors from hidden states; analysis shows initial tokens act as anchors forming positional information, OOD positional vectors cause degradation beyond the window, and context window extension works by interpolating positional vectors via attention logit scaling. This yields two training-free extension methods that interpolate positional vectors.

Key results

• On PG-19, attention window extension (r=4, lambda=1.2) keeps PPL at 17.23 at 2K and 29.30 at 8K, while unmodified TL-Window reaches 660.51 at 8K.
• Positional vector replacement (r=5, alpha=1.3) reduces TL-NoPE PPL at 4K from over 1000 to 47.65, comparable to attention scaling (lambda=1.3 gives 43.84 at 4K).
• Dynamic NTK on TL-RoPE keeps PPL at 10.45 at 4K versus over 1000 without interpolation.

Measured effects reported in the paper

Surprising findings

• Positional vector replacement performs worse than attention scaling at 8K, which the authors attribute to a decreasing effective interpolation ratio across layers.
• Larger LLMs are less affected by removing positional vectors than smaller models, being more attuned to semantic information.

Limitations

• Analysis relies on small TinyLlama-based models trained by the authors, with mainstream LLMs all using RoPE.
• Proposed methods are validated only on the authors' own models, not on mainstream LLMs.
• Positional vector replacement degrades at 8K as the effective interpolation ratio decreases across layers.

Experimental setup

Datasets: RedPajama PG-19

Benchmarks: PG-19

Metrics: perplexity (PPL) cosine similarity of positional vectors

Baselines: Dynamic NTK Attention Scaling TL-RoPE TL-NoPE

Builds on: RoPE ALiBi NoPE Dynamic NTK attention scaling

Closest papers by meaning

• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.91)
• Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.91)
• Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.91)
• An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.90)
• YaRN: Efficient Context Window Extension of Large Language Models (ICLR 2024 · similarity 0.90)
• Long-Short Alignment for Effective Long-Context Modeling in LLMs (ICML 2025 · similarity 0.89)