Research map / Large language models / Language, safety and interpretability
Efficient Streaming Language Models with Attention Sinks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that window attention collapses once initial tokens are evicted due to "attention sinks," and proposes StreamingLLM, which keeps a few initial-token KVs plus a rolling recent-window cache to stream LLMs to millions of tokens without fine-tuning, plus pre-training with a dedicated learnable sink token.
Problem
LLMs cannot be deployed on infinite-length streaming inputs: caching all KV states consumes unbounded memory, and window attention collapses once the sequence exceeds the cache size, while dense attention fails beyond the pre-training window.
Approach
Attention concentrates on initial tokens ("attention sinks") regardless of semantics because softmax must sum to one; keeping the KV of ~4 initial tokens alongside a rolling window of recent tokens (with positions assigned within the cache) stabilizes streaming without fine-tuning, and pre-training with a single learnable sink token lets one token serve as the dedicated sink.
Key results
- StreamingLLM keeps perplexity stable on texts up to 4 million tokens across Llama-2, MPT, Falcon, and Pythia, while window attention collapses once initial tokens are evicted (e.g., Llama-2-7B 0+4096 PPL 3359.95 vs 9.59 with 4+4092).
- StreamingLLM achieves up to 22.2x per-token decoding speedup over sliding window with re-computation on Llama-2-7B/13B on a single A6000.
- Pre-training with a single learnable sink token matches vanilla zero-shot accuracy on 7 benchmarks (e.g., ARC-e 45.6 vs 45.2) and gives stable streaming perplexity with just the sink plus recent tokens.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Keep four initial attention-sink tokens alongside the rolling window instead of evicting all initial tokens (ablation) | StreamingLLM on Llama-2-13B · PG19 first book, 65K tokens | perplexity: 5158.07 -> 5.40 | Helped |
| Replace the original initial tokens with four linebreak tokens (ablation) | StreamingLLM on Llama-2-13B · PG19 first book, 65K tokens | perplexity: 5.40 -> 5.60 | No clear effect |
| Add four initial attention-sink tokens rather than using none (ablation) | StreamingLLM on Falcon-7B, MPT-7B, Pythia-12B, and Llama-2-7B · PG19 concatenated test set, 400K tokens | perplexity: 17.90 -> 12.12; 460.29 -> 14.99; 21.62 -> 12.09; 3359.95 -> 9.59 | Helped |
| Pretrain with a learnable sink token instead of vanilla training (ablation) | Pythia-160M · PG19 first sample, streaming cache 1 + 1023 | perplexity: 18.49 -> 18.01 | Helped |
| Replace standard softmax attention with a zero sink / SoftMax-off-by-One (ablation) | Pythia-160M · PG19 first sample, streaming cache 1 + 1023 | perplexity: 18.49 -> 19.90 | Hurt |
| Increase recent-token cache size (ablation) | StreamingLLM on Falcon-7B and MPT-7B · PG19 concatenated test set, 400K tokens | perplexity: Falcon-7B: 13.61 (4 + 252), 12.34 (4 + 1020); MPT-7B: 14.12 (4 + 252), 14.99 (4 + 2044) | Mixed |
| Use StreamingLLM with attention sinks and a rolling KV cache instead of window attention | Llama-2-7B-Chat, Llama-2-13B-Chat, and Llama-2-70B-Chat · ARC-E and ARC-C streaming QA | accuracy: ARC-E: 3.58 -> 71.34, 0.25 -> 80.89, 0.12 -> 91.37; ARC-C: 1.39 -> 55.03, 0.34 -> 65.61, 0.32 -> 80.20 | Helped |
| Use StreamingLLM instead of one-shot sample-by-sample evaluation | Llama-2-7B/13B/70B-Chat · ARC-E and ARC-C streaming QA | accuracy: ARC-E: 71.25 -> 71.34, 78.16 -> 80.89, 91.29 -> 91.37; ARC-C: 53.16 -> 55.03, 63.31 -> 65.61, 78.50 -> 80.20 | Mixed |
Surprising findings
- Replacing the first four tokens with linebreak tokens yields nearly the same perplexity recovery, indicating absolute position rather than semantics drives the sink effect.
- Increasing the cache size does not consistently lower perplexity, suggesting models do not fully use the provided context.
Limitations
- Does not extend the context window or long-term memory; limited to content within the cache.
- Underperforms truncation baseline on LongBench with 4 sink tokens plus recent window.
Experimental setup
Datasets: PG19 deduplicated Pile ARC-Easy ARC-Challenge HellaSwag LAMBADA OpenbookQA PIQA Winogrande LongBench
Benchmarks: PG19 ARC-Easy ARC-Challenge StreamEval LongBench HellaSwag LAMBADA OpenbookQA PIQA Winogrande
Metrics: perplexity accuracy decoding latency memory usage speedup
Baselines: dense attention window attention sliding window with re-computation Zero Sink (SoftMax-off-by-One) truncation baseline (LongBench)
Builds on: RoPE ALiBi FlashAttention Pythia
Closest papers by meaning
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.90)
- Streaming Attention Approximation via Discrepancy Theory (NeurIPS 2025 · similarity 0.90)
- StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses (NeurIPS 2024 · similarity 0.90)
- InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory (NeurIPS 2024 · similarity 0.90)
- DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads (ICLR 2025 · similarity 0.89)
- SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.