Research map / Architectures and efficiency / Attention and transformers
Star Attention: Efficient LLM Inference over Long Sequences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Star Attention, a two-phase block-sparse inference scheme (blockwise-local context encoding with anchor blocks, then global-attention query encoding) that speeds up long-context LLM inference up to 11x (16.9x at 1M tokens) over Ring Attention while keeping 97-100% of accuracy, without fine-tuning.
Problem
Long-context LLM inference is costly because self-attention is quadratic and autoregressive decoding must attend to all prior tokens; existing chunked or sparse methods often need fine-tuning or architecture changes.
Approach
Shard the context into blocks, each prefixed with a shared anchor block (first block) so attention sinks are placed at anchor tokens; discard anchor KVs and keep only local KVs, then have query tokens attend globally across hosts using distributed softmax with log-sum-exp aggregation.
Key results
- Star Attention retains 97-100% of Ring/full-attention accuracy on RULER with up to 11x speedup for Llama-3.1-8B-Instruct and 4.7x for Llama-3.1-70B-Instruct at 64K.
- At 1M tokens on Llama-3-8B-Instruct-1048K, Star Attention reaches 16.9x speedup with a 5.32% accuracy drop versus Ring Attention.
- On InfiniteBench with Llama-3.1-8B-Instruct, Star Attention averages 46.51 vs 48.06 for full attention, outperforming StreamingLLM (14.72) and MInference (34.07).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Star Attention instead of full attention | Llama-3.1-8B-Instruct · RULER average | accuracy: 85.21 -> 84.44 | Hurt |
| use Star Attention instead of StreamingLLM | Llama-3.1-8B-Instruct · RULER average | accuracy: 45.07 -> 84.44 | Helped |
| use Star Attention instead of MInference | Llama-3.1-8B-Instruct · RULER average | accuracy: 80.71 -> 84.44 | Helped |
| use Star Attention instead of full attention | Llama-3.1-8B-Instruct · InfiniteBench average | accuracy: 48.06 -> 46.51 | Hurt |
| use Star Attention instead of StreamingLLM | Llama-3.1-8B-Instruct · InfiniteBench average | accuracy: 14.72 -> 46.51 | Helped |
| use Star Attention instead of MInference | Llama-3.1-8B-Instruct · InfiniteBench average | accuracy: 34.07 -> 46.51 | Helped |
| use Star Attention instead of Ring Attention | Llama-3.1-8B-Instruct · RULER 16K–128K | inference speedup: 1.1x; 1.2x; 1.8x; 2.7x | Helped |
| remove the anchor block (ablation) | Star Attention · RULER-NIAH 64K | accuracy: 97.61 -> 60.11 | Hurt |
| remove the anchor block (ablation) | Star Attention · RULER-NIAH 128K | accuracy: 94.94 -> 73.75 | Hurt |
| change anchor position IDs from first-block positions to random positions (ablation) | Star Attention · RULER-NIAH 64K and 128K | accuracy: 97.61 -> 96.79; 94.94 -> 97.16 | No clear effect |
| replace anchor content with repeated constant tokens (ablation) | Star Attention · RULER-NIAH 64K and 128K | accuracy: 97.61 -> 0.00; 94.94 -> 0 | Hurt |
| replace first-block anchor content with random tokens (ablation) | Star Attention · RULER-NIAH 64K and 128K | accuracy: 97.61 -> 90.55; 94.94 -> 82.63 | Hurt |
Surprising findings
- Using wrong position IDs for the anchor block has only a minor effect on accuracy, while anchor content is critical.
- Without anchor blocks, blockwise context encoding fails to produce correct outputs on NIAH tasks (e.g., 60.11% at 64K).
Limitations
- Multi-hop tracing degrades because blockwise encoding restricts cross-block token interaction.
- Anchor block size must match context block size for best accuracy, and accuracy degrades with smaller blocks on longer sequences.
- Evaluated mainly with Llama-3 family models; base-model anomalies on BABILong.
Experimental setup
Datasets: RULER BABILong InfiniteBench
Benchmarks: RULER BABILong InfiniteBench
Metrics: accuracy inference speedup time per sample
Baselines: Ring Attention StreamingLLM MInference Full attention (global)
Builds on: Ring Attention Flash Attention StreamingLLM MInference
Closest papers by meaning
- SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.93)
- FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.91)
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024 · similarity 0.91)
- LLM Maybe LongLM: SelfExtend LLM Context Window Without Tuning (ICML 2024 · similarity 0.90)
- CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration (ICML 2025 · similarity 0.89)
- Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.