AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Coverage-Verified Sparse Attention: Closed-Loop Quality Control for Long-Context LLM Decoding

Yan-Cheng Li, Chun-Yi Lee

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paper

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes coverage-verified sparse attention, a closed-loop method that uses the softmax mass captured by selected KV tokens as a runtime signal to verify and recover sparse decoding quality.

Problem

Sparse attention for long-context LLM decoding reads a subset of the KV cache, but existing methods are open-loop: they give no signal on whether enough attention mass was preserved and no recovery when it was not.

Approach

Uses "coverage", the fraction of softmax attention mass on the selected tokens, which the kernel already computes and which satisfies an exact algebraic relation, as a closed-loop verification and recovery signal for sparse attention selection.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add coverage verification and recovery to 5% sparse attention (ablation)CVSA-c4 (5% base) · LongBench (3 7B avg)score: 36.35 -> 40.70Improved
vary acceptance threshold tau from 0.80 to 0.97 (ablation)CVSA-c4 · LongBench (3 7B avg)score: 0.14 points spreadNo clear effect
swap uniform page selector for adaptive allocator with verification on (ablation)CVSA-c4 · LongBench (3 7B avg)score: 0.06 pointsNo clear effect
add dense layer 0 (ablation)CVSA-c4 · LongBench (3 7B avg)score: +1.16 avg, +3.00 on QwenImproved
reduce scoring interval from M=8 to M=4 (ablation)CVSA · LongBench (3 7B avg)gap to dense: -2.04 -> -1.13Improved
reduce scoring interval from M=4 to M=2 at 7B (ablation)CVSA · LongBench (3 7B avg)gap to dense / speedup: gap -1.13 -> -0.43; speedup 1.98x -> 1.57xMixed
use M=2 instead of M=4 at 70B (ablation)CVSA on Llama-3.1-70B · LongBench (7 tasks, 70B)score: 49.76 -> 52.63Improved
approximate recovery (expanded pages) instead of exact complement recovery (ablation)CVSA · LongBench (3 7B avg)score: within 0.05 pointsNo clear effect
replace score-guided recovery with uniform-random page selection (ablation)CVSA · LongBench (3 7B)scoreHurt
raise baseline KV budgets from 10% to 20% (ablation)open-loop baselines · LongBench (3 7B)score: closes at most 1.0 pointImproved
CVSA-c4 versus uniform Fixed-12% at similar read (ablation)CVSA-c4 · LongBench (3 7B avg, ~12% read)score: 37.13 -> 40.70; aligned controls 41.19 (uniform), 41.03 (offline)Mixed
run CVSA with nucleus sampling instead of greedyCVSA-c4 on Llama-3.1-8B · LongBench (8 tasks)gap to dense: -1.31 -> -0.63No clear effect

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy