Research map / Architectures and efficiency / Attention and transformers

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

NeurIPS 2024 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 29 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MInference, a training-free dynamic sparse attention method that assigns each head a sparse pattern (A-shape, Vertical-Slash, Block-Sparse) offline and builds input-dependent sparse indices online to speed up long-context LLM pre-filling.

Problem

Pre-filling long prompts is dominated by quadratic attention cost, making time-to-first-token of long-context LLMs very slow (e.g., 30 minutes for 1M tokens on one A100), and existing sparse or approximate methods lose accuracy or add too much overhead when applied to pretrained long-context LLMs.

Approach

Identify three recurring sparse attention patterns (A-shape, Vertical-Slash, Block-Sparse), choose the best pattern and sparsity budget per head with a kernel-aware offline search that measures attention-output recall at real GPU FLOPs, then estimate dynamic indices cheaply online (last-64-query attention for vertical/slash lines, mean-pooled 64-block attention for block-sparse) and run custom Tri

Key results

• On InfiniteBench, MInference averages 38.8 vs 38.2 for full-attention LLaMA-3-8B-262K, outperforming StreamingLLM (23.8) and InfLLM (34.8).
• MInference reduces pre-filling latency for a 1M-token prompt on LLaMA-3-8B from 30 minutes to 3 minutes on a single A100 (up to 10x speedup).
• On RULER, MInference gives an effective context of 64K for GLM-4-9B-1M with 89.6 average vs 88.0 full attention.

Measured effects reported in the paper

Surprising findings

• Static sparse indices cause near-zero accuracy on KV retrieval, showing dynamic index building is essential.
• MInference slightly outperforms full attention on RULER beyond 32K lengths.

Limitations

• Index-building overhead dominates at short contexts (e.g., about 30% at 10k), so speedup vanishes there.
• Higher sparsity rates can noticeably degrade performance.
• Evaluated mainly on a small set of long-context models; relies on a single-sample validation set for pattern search.

Experimental setup

Datasets: KV retrieval synthetic data (30k tokens, one validation sample) for pattern search

Benchmarks: InfiniteBench RULER Needle In A Haystack PG-19

Metrics: accuracy perplexity latency speedup

Baselines: StreamingLLM StreamingLLM w/ dilated StreamingLLM w/ strided InfLLM Ours w/ static SnapKV

Builds on: StreamingLLM FlashAttention PIT Triton InfLLM

Closest papers by meaning

• Efficient Low Rank Attention for Long-Context Inference in Large Language Models (NeurIPS 2025 · similarity 0.91)
• FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference (ICLR 2025 · similarity 0.91)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.91)
• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.91)
• Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.91)
• SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference (NeurIPS 2026 · similarity 0.90)