Research map / Architectures and efficiency / Attention and transformers

ReAttention: Training-Free Infinite Context with Finite Attention Scope

ICLR 2025 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReAttention, a training-free method that performs position-agnostic top-k KV cache selection before ordinary RoPE self-attention, enabling LLMs to handle multi-million-token contexts within a finite attention window.

Problem

Length extrapolation: LLMs cannot process contexts beyond their pre-trained maximum because positional information goes out of distribution and attention entropy grows with length, limiting practical long-context use.

Approach

Select the most relevant middle KV-cache segments by dot product between the query and un-positioned keys, concatenate them with global and local caches, and only then apply positional embeddings, so the attention window and position indices stay within the pre-trained range while the total context is unbounded. A fused Triton kernel implements the top-k selection efficiently.

Key results

• On LongBench, ReAttention with LLaMA3-8B-8K averages 35.03 vs 35.48 for full attention and 32.49 for StreamingLLM.
• ReAttention extends LLaMA3.1-8B to at least 1M tokens and Qwen2-1B-32K to 4M tokens (128x) on NIAH without training.
• On InfiniteBench at 128K, ReAttention with LLaMA3.2-3B-Instruct reaches 15.36 average vs 10.52 for full attention; it fails on RULER, where Dynamic NTK succeeds.

Measured effects reported in the paper

Surprising findings

• ReAttention and InfLLM fail the RULER benchmark (e.g., MultiKey3), while full-attention Dynamic NTK succeeds, suggesting limits on chaotic or repetitive text.
• Attention without positional embeddings locates the needle more reliably than position-aware attention in the analyzed NIAH case.

Limitations

• Performs poorly on the RULER benchmark, especially multi-key retrieval over chaotic repetitive text.
• Requires sufficient memory for the full KV cache, so the infinite context is bounded by memory rather than by attention computation.
• Hyperparameters (span size, top-k, local size) are tuned on LLaMA3-8B-8K/LongBench and reused across models.

Experimental setup

Datasets: LongBench L-Eval InfiniteBench Needle-In-A-Haystack RULER

Benchmarks: LongBench L-Eval InfiniteBench Needle-In-A-Haystack Multi-NIAH RULER

Metrics: accuracy F1 rouge time to first token throughput GPU memory

Baselines: StreamingLLM InfLLM Dynamic NTK full attention (FlashAttention2)

Builds on: StreamingLLM InfLLM FlashAttention Dynamic NTK

Closest papers by meaning

• LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · similarity 0.91)
• An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding (NeurIPS 2024 · similarity 0.90)
• DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads (ICLR 2025 · similarity 0.90)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
• PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.90)
• Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.89)