Research map / Architectures and efficiency / Attention and transformers
Coverage-Verified Sparse Attention: Closed-Loop Quality Control for Long-Context LLM Decoding
Yan-Cheng Li, Chun-Yi Lee
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes coverage-verified sparse attention, a closed-loop method that uses the softmax mass captured by selected KV tokens as a runtime signal to verify and recover sparse decoding quality.
Problem
Sparse attention for long-context LLM decoding reads a subset of the KV cache, but existing methods are open-loop: they give no signal on whether enough attention mass was preserved and no recovery when it was not.
Approach
Uses "coverage", the fraction of softmax attention mass on the selected tokens, which the kernel already computes and which satisfies an exact algebraic relation, as a closed-loop verification and recovery signal for sparse attention selection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add coverage verification and recovery to 5% sparse attention (ablation) | CVSA-c4 (5% base) · LongBench (3 7B avg) | score: 36.35 -> 40.70 | Improved |
| vary acceptance threshold tau from 0.80 to 0.97 (ablation) | CVSA-c4 · LongBench (3 7B avg) | score: 0.14 points spread | No clear effect |
| swap uniform page selector for adaptive allocator with verification on (ablation) | CVSA-c4 · LongBench (3 7B avg) | score: 0.06 points | No clear effect |
| add dense layer 0 (ablation) | CVSA-c4 · LongBench (3 7B avg) | score: +1.16 avg, +3.00 on Qwen | Improved |
| reduce scoring interval from M=8 to M=4 (ablation) | CVSA · LongBench (3 7B avg) | gap to dense: -2.04 -> -1.13 | Improved |
| reduce scoring interval from M=4 to M=2 at 7B (ablation) | CVSA · LongBench (3 7B avg) | gap to dense / speedup: gap -1.13 -> -0.43; speedup 1.98x -> 1.57x | Mixed |
| use M=2 instead of M=4 at 70B (ablation) | CVSA on Llama-3.1-70B · LongBench (7 tasks, 70B) | score: 49.76 -> 52.63 | Improved |
| approximate recovery (expanded pages) instead of exact complement recovery (ablation) | CVSA · LongBench (3 7B avg) | score: within 0.05 points | No clear effect |
| replace score-guided recovery with uniform-random page selection (ablation) | CVSA · LongBench (3 7B) | score | Hurt |
| raise baseline KV budgets from 10% to 20% (ablation) | open-loop baselines · LongBench (3 7B) | score: closes at most 1.0 point | Improved |
| CVSA-c4 versus uniform Fixed-12% at similar read (ablation) | CVSA-c4 · LongBench (3 7B avg, ~12% read) | score: 37.13 -> 40.70; aligned controls 41.19 (uniform), 41.03 (offline) | Mixed |
| run CVSA with nucleus sampling instead of greedy | CVSA-c4 on Llama-3.1-8B · LongBench (8 tasks) | gap to dense: -1.31 -> -0.63 | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- Quest (10%) — QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference (ICML 2024) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs (ICLR 2026 · similarity 0.91)
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.88)
- TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention (ICLR 2025 · similarity 0.88)
- SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference (NeurIPS 2026 · similarity 0.87)
- Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.87)
- MagicPIG: LSH Sampling for Efficient LLM Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
