Research map / Architectures and efficiency / Attention and transformers
AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AttentionPredictor, a tiny shared CNN that predicts next-step attention scores from block-compressed attention history for KV cache compression, paired with a cross-token prefetching system for CPU-offloaded KV caches.
Problem
Heuristic and learned KV cache compression methods fail to accurately identify critical tokens because they ignore temporal patterns in attention scores, degrading LLM performance at high compression ratios and slowing decoding under cache offloading.
Approach
Treat the stacked per-step attention maps as a 2D spatiotemporal series and predict next-step attention with a lightweight CNN shared across all layers and heads; add cross-token prefetching so that critical-token estimation and KV transfer overlap with the main LLM's compute for the current token.
Key results
- On LongBench with LLaMA-3.1-8B-Instruct at 1024 budget, AttentionPredictor averages 48.58 vs 48.17 full cache and 46.92 for Quest; average loss under 0.5% at all budgets.
- Achieves 5.6x decode speedup over FlashAttention2 at 32K context under cache offloading, and 1.6x over H2O.
- On AIME with QwQ-32B, reaches 76.7% accuracy at KV budget ratio 0.13, whereas Quest drops to 43.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AttentionPredictor instead of Quest with a 1K KV budget | LongChat-v1.5-7B-32K · LongBench | Average: 35.29 -> 35.62 | Helped |
| Use AttentionPredictor instead of Quest with a 1K KV budget | LLaMA-3.1-8B-Instruct · LongBench | Average: 46.92 -> 48.58 | Helped |
| Use AttentionPredictor instead of Quest with a 2K KV budget | LongChat-v1.5-7B-32K · LongBench | Average: 36.14 -> 36.15 | No clear effect |
| Use AttentionPredictor instead of Quest with a 2K KV budget | LLaMA-3.1-8B-Instruct · LongBench | Average: 48.00 -> 48.04 | No clear effect |
| Use AttentionPredictor instead of SnapKV with a 4K KV budget | LongChat-v1.5-7B-32K · LongBench | Average: 35.99 -> 36.10 | Helped |
| Use AttentionPredictor instead of SnapKV with a 4K KV budget | LLaMA-3.1-8B-Instruct · LongBench | Average: 47.84 -> 48.17 | Helped |
| Use AttentionPredictor instead of the strongest shown baseline at 1K KV budget | LLaMA-3.1-8B-Instruct · GSM8K | Accuracy: 53.78 -> 54.33 | Helped |
| Use AttentionPredictor with a 0.13 KV budget ratio instead of Quest | QwQ-32B · AIME2024 | Accuracy: 43.3% -> 76.7% | Helped |
| Use AttentionPredictor for cache offloading instead of full-cache FlashAttention2 | LLaMA-3.1-8B · 32K-token cache offloading | Speedup: 5.6× | Helped |
| Use AttentionPredictor with larger block sizes (ablation) | AttentionPredictor · LongBench (LongChat) | Average performance | Hurt |
Surprising findings
- On LongBench at 1K budget, AttentionPredictor slightly exceeds full-cache performance for LLaMA-3.1-8B-Instruct (48.58 vs 48.17).
- Quest drops 16.91% accuracy on long n-shot GSM8K at 16K, while AttentionPredictor loses only 0.91%.
Limitations
- Predictor is trained on LongBench attention data and evaluated on overlapping task families; generalization is shown only on a limited set of datasets.
- Requires periodic full-attention computation for distribution error calibration, and the first two layers are left uncompressed.
Experimental setup
Datasets: LongBench GSM8K
Benchmarks: LongBench InfiniteBench RULER QA AIME GSM8K MMLU GPQA Needle In A Haystack
Metrics: average task score attention recovery rate per-token decode latency needle retrieval accuracy
Baselines: StreamingLLM Cascade H2O SnapKV Quest SeerAttention FlashAttention2
Builds on: Quest H2O SeerAttention StreamingLLM SnapKV InfiniGen
Closest papers by meaning
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs (ICLR 2024 · similarity 0.91)
- SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference (NeurIPS 2025 · similarity 0.91)
- SALS: Sparse Attention in Latent Space for KV Cache Compression (NeurIPS 2025 · similarity 0.91)
- ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification (NeurIPS 2024 · similarity 0.90)
- CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences (ICLR 2025 · similarity 0.90)
- EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.