Research map / Architectures and efficiency / Attention and transformers
Multipole Attention for Efficient Long Context Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MpAttn, a sparse decoding attention method that clusters keys, computes exact attention on top-scoring clusters, and approximates the rest with centroid attention, plus a fast online cluster update for generated reasoning tokens.
Problem
Long chain-of-thought reasoning creates a large KV cache; existing sparse attention methods lose accuracy under aggressive sparsity and struggle to index newly generated tokens online.
Approach
Cluster keys by semantic similarity, use centroid attention scores both to select important keys for exact attention and to approximate the attention contribution of unselected clusters using value centroids, with a hierarchical variant and a blockwise sliding-window online clustering update.
Key results
- On LongBench-V2 (short split) with Qwen3-8B at token budget 128, MpAttn reaches 38.1 vs 41.5 baseline and 36.3 for Squeezed Attention and 13.7 for QUEST.
- Attention kernel speedups of up to 4.5x over FlashDecoding on A6000 for batch size 16 at 95% sparsity.
- On GSM-Infinite 8K 1-op at budget 128, MpAttn achieves 0.72 vs 0.58 for Squeezed Attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Approximate attention to unselected keys with cluster centroids rather than omit those keys | sparse attention on Qwen3-8B · GSM-Infinite, 8K and 16K, budgets 128 and 512 | accuracy: 0.58 -> 0.72; 0.16 -> 0.19; 0.28 -> 0.40; 0.13 -> 0.17; 0.65 -> 0.82; 0.17 -> 0.22; 0.41 -> 0.61; 0.18 -> 0.30 | Helped |
| Use hierarchical multipole lookup and approximation | Multipole Attention on Qwen3-8B · GSM-Infinite, budget 128 | accuracy: 0.72 -> 0.71; 0.19 -> 0.23; 0.40 -> 0.41; 0.17 -> 0.21 | Mixed |
| Use 1 centroid per 128 KV tokens instead of 1 per 16 (ablation) | Multipole Attention on Qwen3-8B · LongBenchV2 Short (<32K), budget 128 | accuracy: 38.1 -> 29.3 | Hurt |
| Use 1 centroid per 64 KV tokens instead of 1 per 16 (ablation) | Multipole Attention on Qwen3-8B · LongBenchV2 Short (<32K), budget 128 | accuracy: 38.1 -> 35.4 | Hurt |
| Use 1 centroid per 32 KV tokens instead of 1 per 16 (ablation) | Multipole Attention on Qwen3-8B · LongBenchV2 Short (<32K), budget 128 | accuracy: 38.1 -> 38.5 | Hurt |
| Use 1 centroid per 8 KV tokens instead of 1 per 16 (ablation) | Multipole Attention on Qwen3-8B · LongBenchV2 Short (<32K), budget 128 | accuracy: 38.1 -> 37.0 | Hurt |
| Reduce clustering block size to 2K (ablation) | Multipole Attention on Qwen3-8B · LongBenchV2 Short (<32K), budget 128 | accuracy: 38.1 -> 35.0 | Hurt |
| Reduce clustering block size to 4K (ablation) | Multipole Attention on Qwen3-8B · LongBenchV2 Short (<32K), budget 128 | accuracy: 38.1 -> 38.0 | Hurt |
| Increase clustering block size to 16K (ablation) | Multipole Attention on Qwen3-8B · LongBenchV2 Short (<32K), budget 128 | accuracy: 38.1 -> 36.9 | Hurt |
| Use Multipole Attention instead of Squeezed Attention | attention decoding on A6000 · Qwen3-8B, batch sizes 1, 4, and 16 | speedup: 2.8$\times$ -> 2.4$\times$; 3.3$\times$ -> 3.3$\times$; 3.6$\times$ -> 3.7$\times$ | Mixed |
Surprising findings
- QUEST drops sharply at token budget 128 (13.7 vs 41.5 baseline on LongBench-V2 short split).
Limitations
- Only accelerates decoding; prefill attention is not accelerated.
- Requires engineering effort to integrate into existing LLM serving frameworks.
- Needs additional memory for centroids, though overhead is small.
Experimental setup
Datasets: LongBench-V2 GSM-Infinite LongBench-V1
Benchmarks: LongBench-V2 GSM-Infinite LongBench-V1
Metrics: accuracy attention speedup memory operations
Baselines: Squeezed Attention QUEST DuoAttention TOVA FlashDecoding
Builds on: Squeezed Attention QUEST FlashDecoding FlashAttention-2 YaRN
Closest papers by meaning
- Stop When Further Reasoning Won’t Help: Attention-State Adaptive Generation in Reasoning Models (ICML 2026 · similarity 0.87)
- BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference (ICML 2026 · similarity 0.87)
- Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.86)
- A State-Transition Framework for Efficient LLM Reasoning (ICLR 2026 · similarity 0.86)
- TriAttention: Efficient Long Reasoning with Trigonometric KV Compression (ICML 2026 · similarity 0.86)
- Fuel Gauge: Estimating Chain-of-Thought Length Ahead of Time in Large Multimodal Models (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.