Research map / Large language models / Language, safety and interpretability
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfLLM, a training-free block-level context memory with sliding-window attention that lets LLMs pretrained on a few thousand tokens process extremely long sequences (up to 1,024K).
Problem
LLMs pre-trained on short sequences cannot process much longer inputs due to out-of-domain positions and attention distraction from noisy distant context; existing fixes require costly continual training.
Approach
Stores evicted distant KV vectors in block-level memory units, represents each unit by its top-scoring representative tokens selected via attention-based scores, and at each step looks up only the most relevant units to attend alongside a local sliding window; uses CPU offloading with an LRU-style GPU cache. Entirely training-free.
Key results
- On ∞-Bench with Llama-3-8B, InfLLM (8K window) averages 45.0 vs 18.4 for the original Llama-3 and 38.0 for SelfExtend at 128K; Mistral-based InfLLM averages 57.7 vs 25.2 for base Mistral.
- Training-free InfLLM on Llama-3-8B-Instruct-Gradient-1048k is comparable to the continually trained model, using 26.3G VRAM and 26.7s vs 76.6G and 40.4s.
- InfLLM achieves 100% accuracy on Retrieve.PassKey up to 1024K tokens on a single GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add InfLLM context memory to Mistral-7B-Instruct-v0.2 | Mistral-7B-Instruct-v0.2 · ∞-Bench | Avg.: 44.8 -> 57.7 | Helped |
| Add InfLLM context memory to Llama-3-8B-Instruct | Llama-3-8B-Instruct · ∞-Bench | Avg.: 38.0 -> 45.0 | Helped |
| Use InfLLM instead of continual long-sequence fine-tuning | Llama-3-8B-Instruct · ∞-Bench | VRAM: 76.6G -> 26.3G | Helped |
| Use InfLLM instead of continual long-sequence fine-tuning | Llama-3-8B-Instruct · ∞-Bench | Time: 40.4s -> 26.7s | Helped |
| Apply InfLLM context memory to Llama-1M | Llama-1M · ∞-Bench | R.KV: 23.2 -> 55.8 | Helped |
| Replace InfLLM with RAG-E5 retrieval | Mistral-7B-Instruct-v0.2 · ∞-Bench retrieval | R.KV: 13.2 -> 96.8 | Hurt |
| Add InfLLM context memory to Mistral-7B-Instruct-v0.2 | Mistral-7B-Instruct-v0.2 · LongBench | Avg.: 43.78 -> 44.02 | Helped |
| Add InfLLM context memory to Llama-3-8B-Instruct | Llama-3-8B-Instruct · LongBench | Avg.: 44.73 -> 46.95 | Helped |
| Use memory lookup during both encoding and decoding instead of decoding only (ablation) | InfLLM · ∞-Bench Retrieve.KV | R.KV: 85.2 -> 96.8 | Helped |
| Add memory lookup (ablation) | InfLLM · ∞-Bench Retrieve.KV | R.KV: 0.4 -> 96.8 | Helped |
| Use selected representative tokens instead of mean key-vector representations (ablation) | InfLLM · ∞-Bench Retrieve.KV | R.KV: 84.6 -> 96.8 | Helped |
| Use memory lookup during both encoding and decoding instead of decoding only (ablation) | InfLLM · ∞-Bench Math.Find | Math.F: 26.3 -> 25.7 | Hurt |
Surprising findings
- Mean-of-keys unit representations perform competitively with the representative-token method, suggesting original attention vectors suffice for relevance scoring.
- Vicuna (4K) extended by InfLLM gains strongly on passkey and number retrieval but shows no gain on KV retrieval and Math.Find.
Limitations
- Stores large KV cache in CPU memory, increasing CPU memory usage.
- Limited inference speed-up; relies on heuristic fixed unit sizes that are suboptimal across tasks.
- Weak gains on models lacking robust noise filtering (e.g., Vicuna on KV retrieval and Math.Find).
Experimental setup
Datasets: ∞-Bench LongBench
Benchmarks: ∞-Bench LongBench Retrieve.PassKey (1024K scaling)
Metrics: accuracy score (average) VRAM usage time consumption cache miss rate
Baselines: Mistral-7B-Instruct-v0.2 Llama-3-8B-Instruct NTK SelfExtend LM-Infinite StreamingLLM H2O Llama-3-8B-Instruct-Gradient-1048k RAG-E5
Builds on: StreamingLLM LM-Infinite NTK-aware RoPE SelfExtend H2O
Closest papers by meaning
- LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · similarity 0.90)
- SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.90)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.90)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training (ICLR 2024 · similarity 0.90)
- Why Does the Effective Context Length of LLMs Fall Short? (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.