Research map / Large language models / Instruction and fine-tuning

LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning

ICML 2024 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 13 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SelfExtend, an inference-only method that extends LLM context windows by mapping distant relative positions via floor-division grouped attention while keeping standard neighbor attention, requiring no fine-tuning.

Problem

LLMs with RoPE fail on inputs longer than their pretraining context window because unseen (out-of-distribution) relative positions appear at inference; existing extension methods require fine-tuning on long data.

Approach

Build two-level attention at inference: grouped attention that floor-divides positions for distant tokens (mapping unseen relative positions into the pretrained range) and standard neighbor attention for tokens within a local window; merge both before softmax. Only the attention computation changes, so no weights are updated.

Key results

• SelfExtend-Llama-2-7b-chat keeps PG-19 perplexity around 9 up to 16384 tokens while vanilla Llama-2-7b-chat exceeds 10^3 beyond 4096 tokens.
• SelfExtend achieves 100% passkey retrieval accuracy across all tested depths and context lengths for Mistral-7B-instruct-0.1 without fine-tuning.
• On LongBench, SE-Phi-2 (8k) improves NarrativeQA from 4.46 to 12.04 and Qasper from 7.01 to 12.10 over vanilla Phi-2 (2k); SelfExtend roughly maintains Open LLM Leaderboard short-context scores.

Measured effects reported in the paper

Surprising findings

• Low perplexity on long sequences does not imply long-context ability: Mistral with sliding window attention keeps low PPL but cannot retrieve passkeys outside its window.
• A purely inference-time attention modification can match or beat several fine-tuned long-context models on LongBench and L-Eval.

Limitations

• Performance degrades with large group sizes, so contexts cannot be extended indefinitely.
• Naive implementation adds computation across all query-key pairs and still processes the full sequence.
• Hyperparameters (group size, neighbor window) need tuning and the heuristic rule may not transfer across models, e.g., Llama-3 needs a much smaller neighbor window.

Experimental setup

Datasets: PG-19 LongBench L-Eval Hugging Face Open LLM Leaderboard tasks (ARC-c, HellaSwag, MMLU, TruthfulQA, GSM8K)

Benchmarks: PG-19 Passkey retrieval LongBench L-Eval Hugging Face Open LLM Leaderboard Varying-length passkey retrieval

Metrics: perplexity accuracy F1 / task score

Baselines: Llama-2-7b-chat LongChat1.5-7B-32k Vicuna1.5-7B-16k CLEX-7B-16k CodeLlama-7B-16k MistralLite-16k Yarn-Llama-2-7b-64k LongLora-7b-16k Llama2-13b-NTK GPT-3.5-Turbo-16k

Builds on: RoPE position interpolation ReRoPE LongBench L-Eval

Closest papers by meaning

• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.91)
• ReAttention: Training-Free Infinite Context with Finite Attention Scope (ICLR 2025 · similarity 0.91)
• SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.90)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.90)
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory (NeurIPS 2024 · similarity 0.90)
• Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.90)