Research map / Architectures and efficiency / Attention and transformers

Block Sparse Flash Attention

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Block Sparse Flash Attention (BSFA), a training-free FlashAttention-2 extension that gates value-block computation using exact per-block max scores against offline-calibrated per-(layer, head, position) thresholds, yielding up to 1.13x end-to-end LongBench speedup on Llama-3.1-8B.

Problem

Attention's quadratic compute cost dominates long-context LLM prefill; existing sparse attention methods predict block importance before observing exact scores, which can hurt accuracy and slow down short contexts.

Approach

Compute exact QK scores within the FlashAttention tile, then skip the V load and PV multiplication for blocks whose max score falls below a calibrated per-layer/head/position threshold that retains roughly top-k blocks per query.

Key results

• On Llama-3.1-8B, BSFA k=64 reaches 1.13x end-to-end LongBench TTFT speedup at 1.1% accuracy drop (39.78% vs 40.24% dense).
• At RULER 64K with k=192, the attention kernel is 1.38x faster and end-to-end 1.13x with 2.2% accuracy drop.
• On 64K Needle-in-a-Haystack, BSFA k=32 retains 99% of baseline accuracy at 1.24x speedup.

Surprising findings

• Pre-attention selection methods (MInference, FlexPrefill, XAttention) regress to as low as 0.17x on contexts under 4K, while BSFA stays near dense on short contexts.

Limitations

• Targets prefill only; decode not addressed and FA-3 integration left to future work.
• Speedups are modest (about 1.1x end-to-end) and accuracy drops grow at higher sparsity (e.g., -4.0% at 128K k=512).
• Baseline comparisons on H100 and A100 are mixed, and some baselines (e.g., XAttention DP optimizer, SpargeAttention) were not fully reproduced as released.

Experimental setup

Datasets: RULER LongBench Needle-in-a-Haystack

Benchmarks: LongBench RULER Needle-in-a-Haystack

Metrics: accuracy end-to-end TTFT speedup attention-kernel speedup block density

Baselines: SpargeAttention MInference FlexPrefill XAttention BLASST Sliding Window

Builds on: FlashAttention-2 BLASST SpargeAttention MInference

Closest papers by meaning

• Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps (NeurIPS 2026 · similarity 0.90)
• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.89)
• Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.88)
• Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.88)
• MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.88)
• SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.88)