Research map / Architectures and efficiency / Attention and transformers

CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CompactAttention, which turns 2D block-sparse masks into GQA-aware per-group KV block tables via Q-block and intra-group union, enabling zero-copy paged dense attention for chunked prefill.

Problem

Block-sparse attention is inefficient under chunked prefill (Q much smaller than KV) and pattern search repeats per chunk; query-subsampled token-level KV selection (QUOKA) misses query-specific KV entries and incurs gather-copy overhead.

Approach

Use block-sparse masks only as KV-selection signals: union across query blocks and across heads in a KV group to get minimal per-group KV block tables, then execute dense paged attention over the original KV cache in place (KV-head-major layout) with no KV copy.

Key results

• On LLaMA-3.1-8B-Instruct RULER avg, CompactAttention-FP scores 82.30 vs dense 82.78 (Avg over 32K/64K/128K).
• CompactAttention-FP reaches 2.72x attention speedup and 1.96x end-to-end speedup over dense at 128K on H200 (chunk size 1024).
• QUOKA underperforms dense on RULER for both models (LLaMA avg 77.98 vs 82.78), degrading on Multi-key NIAH-3 and CWE.

Surprising findings

• Block union reduces sparsity substantially (e.g., 89.8% to 70.2%), yet attention speedup still exceeds the block-sparse baseline because dense paged execution outweighs the lost sparsity.
• On Qwen3-30B-A3B at 128K/256K, FlashPrefill is slower than dense at 32K-64K-level or near parity (0.77-1.05x), while CompactAttention-FP reaches 1.56-1.64x.

Limitations

• Inherits quality of the underlying block-sparse pattern search; missed blocks cannot be recovered.
• Q-block and intra-group union trade sparsity for coverage; benefit may vary with architecture, chunk size, and group partitioning.
• Gains depend on large accumulated KV cache to amortize search and metadata overhead; short contexts can be slower than dense.

Experimental setup

Datasets: RULER LongBench V2

Benchmarks: RULER LongBench V2

Metrics: accuracy attention latency speedup end-to-end latency speedup

Baselines: QUOKA XAttention SeerAttention FlashPrefill FlashInfer dense attention

Builds on: FlashInfer SeerAttention FlashPrefill QUOKA FlashAttention-3

Closest papers by meaning

• Block Sparse Flash Attention (NeurIPS 2026 · similarity 0.88)
• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.87)
• Training-Free Long-Context Scaling of Large Language Models (ICML 2024 · similarity 0.87)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.87)
• AttentionPredictor: Temporal Patterns Matter for KV Cache Compression (NeurIPS 2025 · similarity 0.87)
• Virtual Head Attention (NeurIPS 2026 · similarity 0.87)